My H ruined several special occassions due to his selfish behavior (case in point - he shared the news with me that his "lingering doubts" weren't going away on my 32nd birthday). On my first Mother's Day, he ignored me all morning and then told his friend (on the phone) to tell his wife Happy Mother's Day from him (he hadn't told me yet). So yeah, I kind of want a fuss made over me now on my birthday, and Mother's Day, and our anniversary. Does that make me selfish. Possibly. Do I give a shit? Not really.
I don't think you did anything wrong. He was being defensive and throwing it back on your lap.