It sounds like maybe he hasn't been completely honest about the nature of the A. Not unusual but extremely hurtful. Try to impress upon him how devastating any new information is after your mind has come to terms with what you already know to be the truth.
Around here we call it Trickle Truth. When you find out more information it sets you back, sometimes to square one. It's no fun and not fair.
I hope he can come to you and give you the truth you need to heal and to R the marriage.