Sorry hubby came in as I was typing. I think you have justified concerns regardless of his issues - and it is a manipulative ploy on his part to claim that you are making it all about you during those moments. My WH does this sometimes still, and used to do it a lot. Don't let it get you off course, stay true to your own instincts.
That being said, sometimes you need to spend time with others who don't have a history of damaging you in order to get some perspective on a destructive relationship. It will hopefully give him a reality check that you are taking care of yourself and not at your own expense (instead of taking care of him at your expense). But the important thing is to feel what it is like to be nurtured by someone in a healthy way (obviously not cheating) but just interact in a sane way that reminds you what your own value is.
Hope this helps I speak from experience.