The good news is, the two of you are having romantic nights and intimacy. He shared his feelings with you. He's doubting your story and he's devastated over your cheating. He spoke his true feelings to you. His words are going to hurt. He's in pain, you're at the beginning of the very long process of picking up the pieces...these tough conversations are going to happen.
If he's glad you're in pain take it for what it is. He's probably relieved to see that he hurt your feelings because that's telling him that you still care about the marriage. Or he's just happy to see you in pain because of the pain you caused him. Either way, those are his honest feelings at the moment.
Keep talking and communicating. You're right, this is a very long road.