I don't feel it's strange that you aren't going to talk to him. You said you feel anxious, which means you feel threatened by the idea of trying to discuss. You already said you didn't feel cared for. Does anyone want to run up to a person they don't think cares and try to get comfort and love from them?
My wayward withdraws too and says he wants left alone. Amazingly he also says the reason is he is tired of everyone blaming him, so he wants left alone and not to see anyone.
In your case, possible your husband needs recovery time after painful discussion. In my case, we never get around to discussing because he pulls the poor me card and nothing ever gets accomplished.
I feel the same threat you may be feeling. I want the comforting and validation but how dare I imply I may need it because he just may not care enough to give it.
In your case, you have a chance. Atm in my case, not a snowballs chance. it hurts pretty badly.
I would say just give him some space between discussions. And I don't know if I am wrong, but I feel anxious at the thought of talking, so I am erring on the side of maybe being too cautious.