After he spoke to her those two other times AFTER I had already told him not to, I spoke to him about it again and he said that he had misunderstood and will certainly stay away from her.
Kristin, I'm so sorry you find yourself here.
I have to ask, how does one 'misunderstand' the phrase, "don't talk to her anymore!" I find his lame excuse for talking to her AFTER you told him not to - that he 'misunderstood,' - a complete insult to your intelligence after all he's done. He understood your request perfectly and he knows it. Unfortunately, most affair partners don't just cut contact with each other after a D-DAY; they just find another way to be sneakier about keeping in touch while trying to fly lower under the radar and waiting for the dust to settle.
I think you also need to consider getting STD testing. Cheaters never admit to anything more than they absolutely have to, so if these two had the opportunity to spend any time together in the same room at all, the chances are extremely high that it was more than just a flirtatious friendship or EA. Especially since your husband was basically living alone and pretty much free to come and go as he pleased without having to answer to anyone.
Right now, I know you're trying to trust him but quite honestly, he's not worthy of any trust at ALL. If anything, you need to step up your awareness because it's highly likely he's still in contact with her. I also think down the road you'll have to prepare yourself for more 'truth' as you find out more information about this affair (maybe you'll finally wear him down interrogating him one day). Most of us do have to deal with 'trickle truth' (TT) after the affair. But it's pretty much a certainty that just about no one gets the whole story on D-Day. Some eventually find out the EA was actually a PA, some who were told it lasted a year find out it was actually 5 years; some who were told sex only happend 10 times find out it happend 100 times. The point is that there's always more 'truth' to be learned, so guard your heart and try to stay prepared for that eventuality.
Claiming his impatience with you is "really because he's so mad at himself," sounds a bit contrived. I think he's just trying to tell you what he thinks you want to hear. It's incredibly selfish of him to be irritated with you because you're not just giving him a free pass and pretending all is fine with the world. That speaks volumes about his lack of remorse about this whole thing.
For now Kristen, I wouldn't trust him as far as you can throw him. The clear majority of us here, like you, wanted to start trusting our cheating spouses again too - and pretty much ALL of us had the rug yanked out from underneath us when we discovered they were still in touch with their affair partners. They were just using new secret email accounts that they'd just opened, hidden pay-as-you-go cell phones, untraceable apps on their smartphones/iPhones for chatting/emailing/texting/video chatting, etc. etc. which don't leave a trail on the cell phone bill, and the list goes on and on. Giving us access to their email accounts means nothing when they've opened a new secret one we know nothing about. Letting us read their texts means nothing when everything they're doing is now contained within an app that doesn't leave a trail on the phone or on your cell phone bill. Ugh.
You take as long as you need to heal. He needs to find a way to deal with his 'irritation' because he doesn't GET it, Kristen. It's still about HIM.
Good luck to you.