You are not ruining his life. He did that all on his own.
He is acting sad because he is a petulant emotional infant and you are not letting him have his cake and eat it too.
You are not breaking up the family. You are changing the living arrangements to reflect the reality of what your WS has done to the family. HE left the family emotionally a long time ago. Ideally, he would physically leave the home as well, but since he isn't, you are establishing a new home for the remaining family (you and the kids). The family was broken by his infidelity, you are just taking care of what is left in the best way possible.
Maybe I wasn't there for him as I should have been. Maybe I was more needy. And he also said I rarely initiated sex anymore
Do NOT accept any blame for his cheating. So you were needy. It was his job as your husband to be a support to you. By that reasoning, you should have been able to cheat on him because he failed you in your time of need. (insert sarcasm)
And don't get me started on the issue of who initiates sex. My STBXH didn't initiate sex anymore because I "turned him down in the past." Sorry I didn't feel like having sex when sick or exhausted. I felt rejected, even though he wouldn't turn me down when I initiated sex. But I did not use it as an excuse to cheat. WS's will twist anything into an excuse. Your WS didn't like that you rarely initiated. My WS didn't like that I did initiate. You can't win for losing with these people.
Personally, I feel offended that your WS would use your childhood trauma as an excuse against you. My WS used my depression as an excuse against me. It is all just them trying to cover their asses. You need to lose the guilt and gain some anger. As women, we are culturally conditioned to not get angry, so it turns inward as guilt. This is something to work on in IC, or to just tell yourself over and over.
Edited to correct stupid typos.
Double Betrayal D-Day 7/26/2013
Two steps forward and one step backwards, is still progress.