I went through some similar things with SO. For example, sometimes he works until 10pm. I expect him to be home by 10:15-10:20, 10:30 at the latest! There are times where he strolls in at 10:45 or 11...
It's always something innocent - he stopped by his parents, work took longer to close, stopped to get food, whatever. Sometimes he'll text me when he's leaving work so I'll have a better idea when to expect him. Anyway, he's still almost always late, but my mind has completely stopped "going there."
I'm working on lowering my timing expectations and he's working on communicating more often.
I'm glad you guys talked about it right away. Keep talking about things and working through triggers together. He won't know if you don't tell him.