Trust your instincts.
The goal in recovery is to be very honest with each other. You need to be able to be honest with him when you start seeing red flags and he needs to give honest answers in return, otherwise real recovery won't happen.
With many waywards, anger is used as a deflection tool to shut the other person down and at the same time level a degree of punishment toward them so they will stop asking questions....
D-Days April - Oct. 2007 Recovery started Nov. 2007
"Found Myself", I was right there in my shoes all along!
Search for self called off!
Why Repentance Is Necessary? Because Undeserved Mercy Empowers Entitlement/Sin
I am sorry you are going through this. IMHO, he is acting suspicious. He sounds like he is gaslighting you. He is clearly in her area, multiple times, during time periods/ on days they could be meeting. You have told him " countless times" that you need to know from him, when he is near her. Yet he persists in being near her without telling you, and furthermore, gets mad about it and wants to write it off to coincidence?
Doesn't sound right to me. From a remorseful person with nothing to hide, I would expect him to tell you, as you have asked. And to apologize for causing you worry, and make darn sure he drives out of his way to make sure he doesn't worry you again.
He knows if you ask he acts out so therefore you will stop asking. He needs to grow the f up!
If it was me I wouldn't give a flip where he is and not be home when he thinks you should be home. I am sorry but honey he is just being mean. He should be carrying about you and bending over backwards to show you he loves you. Please don't take his abuse. Yes he is abusing you..
Really look at this marriage is it one that you will be proud of? Is he acting like a loving caring remorseful husband?
I hope you find the strength to move forward in a positive life sounds like it will be without him if it is to be positive.