Last night in bed talk in & he mentioned something & then said he didn't want to make me mad so he wasn't going to talk about it. I pushed him to tell me. It has to do with appearances (mine). Yes I got upset & well I think he will never get it.
I AM MORE THAN A MY OUTWARD LOOKS!
You are very much more than your outward looks.
In this situation, bear in mind that you pushed for him to open up with his opinion. Whether or not you agree with them, pushing for him to express them and then immediately responding with the very behavior he was afraid to open up to is only going to make him feel like you are looking to trap him in an argument rather than listen to what he has to say and hear his opinion on something.
I feel like I'm not being true to myself by continuing in our marriage the way we are. Not much lovey stuff outside the bedroom & he never says he loves me. I stopped saying it to him because he'd respond with I love you back but it was almost under his breath.
This is something you should try to talk to him about. If he won't get on board with trying to at least meet halfway somehow then the answer to your question of how long do you let it continue is: as long as you choose to.