While I agree with ladiesfirst that a marriage needs to have its positive components, you also have some major rug sweeping going on.
Of course we all want the people around us to be happy, but when one of a team has been stabbed in the back by the other, it isn't out of the question for there to be some down moments. If your spouse hasn't understood that yet, and doesn't actually embrace the opportunity to help you in the down times, then you know you do not have remorse.
Read the books. Set time to discuss these things. The wayward has to face these issues or you are looking at more problems down the road. I have seen over and over how conflict avoidant most waywards are. This is part of that conflict avoidance and the patterns have to change.
It's not acceptable for him to make you feel bad for feeling bad, when he is the reason you are feeling bad! Triggers are hard for most waywards to understand. They fight against their reality. You have to really work through them.
By doing those things, your concerns in your last paragraph will be addresses. When a wayward is making consistent healing actions, your fears will be lessened, and rightly so.