Constant Learner,
Okay, there is something to work with.
You may not believe me, but the feelings you are harboring for the AP is a major part of where you are in your thinking. I didn't feel the whole "soul mate" this "person is perfect for me", but I felt like because he was so supportive of me emotionally I could never have that with my husband. And, the emotional connection is very important to women especially. I am also not a huge subscriber to the "fog" in it's exact terms that are spelled out, but you are in it. I wouldn't have believed me either when I was still in it. I could point to so many things that my husband wasn't, my marriage wasn't, and I was tired...so tired of carrying it all. Lonlier than I have ever been.
How long have you been NC? It's almost sounding like your AP was single? So, if he's single there is a high likelihood you've been able to communicate. I would not have been able to stop that on my own.
The reason NC is so important is that it allows us to come back to our rational thinking, and when you are back to that rational thinking I can't guarantee you will decide you want to stay married after all, but I can guarantee that you will have done what you needed to do in order to make logical decisions rather than emotional ones.
The AP is not a safe choice for you. I don't know if you will believe me about that, but he knew both you and your husband and he was willing to do this anyway. It means that he himself has boundary issues. The likelihood that he would have boundary issues in a relationship is high, or at the very least have trust issues with you. He was willing to sleep with another man's wife - justify that if you must but there is something very wrong there.
So, here is my advice: If you are not willing to give up the AP leave your husband. Save him from further pain.
If you are not willing to leave your husband then you have to make the hard choice to begin programming yourself to stay away from the AP- and try and see the ways he manipulated this situation for his own gain. Staying away from him doesn't just mean NC, it means staying away from thinking about him positively, looking in on him through social media...you have to give him up entirely.
You have to realize the reason I knew it was there is I could see the thought patterns behind it. I had these thought patterns. I commend you for admitting it, very hard to do around here.
I can tell you for me, once I realized I was really going to lose my husband, the life that we'd worked for, disappoint my daughters who would have to visit us separately on holidays and have to fret if it's okay to invite both of us to major events and have to take all the stress of navigating that.....well I came out of it.
What I came out of it to was still very bleak. Getting things back on the road has been hard and will continue to be hard. But, as I have grown and changed, I realized the ways I was blocking us from having that emotional connection. Yes, you read that right...it's true my husband is not very emotional...but I learned my own barriers that were not helping. He is still hurt, we are still working on it but we have more good days right now. I feel closer and more grateful for him than I have ever been. He is opening up more and responding to that. I also learned that his open handed way of loving was far more healthier in letting life flow than the firm grip the AP and I had on each other. Look up things Buddhist say about attachment for more thoughts on that.
Am I telling you this because I don't want you to get a divorce? No. I am telling you this
1. Because I want you to know that there are things you can't know unless you really put it all down to try. And, by doing that if you do decide to divorce it will be in clarity and not in all this mud that is in your head right now.
2. I am also telling you this because if you aren't willing to take that path, that's okay. Just don't put yourself and your husband through it any more. Yes, you will disappoint people. But in the end you are dragging your husband down a road for no reason. Love him enough to take the hit of disappointing the other people.
I hope you find this helpful. For me, I said once I was out of the "fog" that I would give it a solid six months, give it my honest to goodness all and then re-evaluate. I didn't need to re-evaluate by then. My path was clear. And for you it still might be clear that you still want to go but you could then do it in good conscience and will have done the honest work to grow healthier for the next chapter of your life.
I know right now you feel fear of really losing your AP for good... it adds a lot of pressure to the situation. Let that go if you choose the work on yourself and marriage path.
And, by the way, if you do choose to leave, the likelihood that you will work on yourself diminishes so much because you won't have the precipitance any longer. You will be back in happy land, until this relationship gets further down the road and these issues once again come up. Because some of these are your issues, you carry these with you.