You did not do anything wrong! He's crazy-making, where you are coming off as the bad guy even though you're just having a normal reaction to the things he's doing. My ex-husband (of SIX MONTHS!!!?!) used to grab my wrists, push me, scream at me, verbally assault me, insult me, refuse to have sex with me, smash his food onto the side of my face (usually meat, since I'm a vegetarian), hid our marriage from his friends, tried to grab my laptop and throw it 10-12 times, tried to break our (shared, obv) cell phone because I asked him about 3 unknown numbers texting him (and I called back and they were women, of course), spent a ton of my money, and used me for rent/food/utilities.
Throughout the beginning of our short marriage, I used to cater to him, tell him I'm so sorry, he's so right, I shouldn't have done _____. I thought it must be me because he used to be so great to me (things changed after marriage). I almost left so many times, but his mom (a majorrrrrr enabler, but nicest lady in the world) kept persuading me to stay ("he's just angry, he's stressed about work....") After the 3rd or 4th time of him smashing his food into my face while I was watching my online class on the TV, I slapped that asshole smack across his face right in front of my MIL...and it felt amazing. Six months of abuse came out in that slap and the look of shock on his horrid, pig face was so worth it.
I'll be damned if I'm going to be made out to be abusive because I finally reacted to the dick things this guy was doing. I have an amazing tolerance for things, but once things cross a certain line, I'm f*cking DONE. I'd never hit someone before, and I'm not abusive...I just believe that people have a breaking point and, eventually, your reaction is going to be a compilation of everything that built up inside of you that wasn't important enough to him to address.
From that moment on, he couldn't get to me anymore. I'd finally stood up for myself and no matter how much worse his actions got, I wasn't interested. I mourned the loss of my marriage before I ever even got out of there. I knew that I had a big weakness for MIL and seeing her sad, so I knew that whatever excuse I was going to use to leave her son had to be good enough that I couldn't be talked out of it. I needed to bide my time and wait for him to do something bad enough that I could get out, guilt-free. It didn't take him very long to go online and solicit Craigslist hookers from two states away and be too stupid to clear his internet history.
Well, long story short, I'm not abusive; I just finally had an explosive reaction to months and months of torment...and it sounds like you did, too. The abuser ALWAYS tries to make the abused think they're crazy. That's part of what makes them so dangerous!!