I'm sorry he hurt your feelings. But he's the one that called you so his anger is definitely misplaced.
Is it possible that you can talk to him when you're both home together? Let him know how this makes you feel and as long as he continues like this, you're never going to feel secure and start your healing.
He really needs to understand how this affects you."Don't give up, the beginning is always the hardest." My Mom:)
My tolerance for stupid shit is getting less and less.