In the words of my therapist: you are just feeding the disease, whatever disease that is plaguing your relationship with this man (sex addiction, narcissism, substance abuse, immaturity, etc.)
You are trying to make sense of nonsense (that is what Dr. Phil says...and he's right).
Stand up for yourself, draw a line in the sand, and follow through. This is what I finally did after 7+ years of marital misery of one type or another (SAWH had an affair I was suspicious of but he denied for 2+ years). I finally decided that whatever was going on, I could not live like this any longer. The timing of mustering the courage to confront interestingly coincided with info that fell into my lap about the relationship I suspected. The universe has a way of telling you things (if you are listening).
None of this has been easy or sunshine and lollipops, but when I look back to where I was 4 months ago, I am grateful I know what I know now and that the situation is being addressed - whatever the ultimate outcome.