Concussed, I'm not sure what you're struggling with. It seems the wife is okay with this, but you're here questioning it so, are you?
You start with 'I'm not a swinger', and then you had sex with another woman's husband, with her permission.
So...if you're comfortable with this, what's the issue? If you're uncomfortable with this lifestyle, stay away from this couple. Get new friends who aren't living this lifestyle.Take up your space (and do it well).
"That's the thing about pain, it demands to be felt."