So sorry you are hurting. I had a friend that was in a similar situation with losing weight and immediate marital problems. They ended getting a divorce, her husband could not handle the thought that she was beautiful and was fearful that she would also leave him. It so didn't make sense to her or to me, but it happened. Maybe what he says makes sense to him....but remember ultimately he is BROKEN. Regardless, this is not about you...this is about him. He broke your marriage not you.
It is ok that this is your deal breaker. Everyone responds differently to this. There is no right or wrong way to respond for everyone. Only the right way for you. If this is a deal breaker for you, that is ok. Fortunately, you do not have to make a decision now. As you continue to go through your emotions, you may change your mind many times through the process. That is normal too.
Support being sent to you.