Hi,
I am so sorry that you are in this situation, and that your husband doesn't realize that you are at breaking point, ready to jack in your marriage.
I want to say that even though the "lady" might have a gorgeous body, she has no shame of meeting men for sex. So, hum, I would not call her gorgeous. Something totally different.
For the self-esteem issue, yeah, it's hard. I personally took it very bad, this infidelity crisis. He liked talking to her, spending time with her, felt connected to her, yada yada. It made me question myself: am I that boring that even my husband would turn to somebody else to talk to? Am I that shallow of a person that he would need a connection with somebody else? My husband was everything to me, the good guy, the one who could do no wrong. So yeah. Was he entitled to have an affair?
Well, one day the bulb went off in my head. I was the one who got us out of financial troubles. I was the one who learned to cook properly. I was the one reading serious books. I was the one organizing playdates and weekends activities. I was the one homeschooling. And HE WAS THE JERK FOR NOT APPRECIATING ALL THIS. Seriously, the cheek. To my standards I was doing a great job at everything that I was taking under my wings. Why would it matter if it was not up to HIS standard? Because he sure didn't do anything himself, apart from critizising or kindly giving me his seal of approval. Which is easy. Talk is easy. Doing the work is not. And on a positive note, he's the one who has a boat-load of FOO issues, conflict avoidance, people pleaser, passive aggressive issues. Mr Perfect Guy no more. Ha!
IMHO, if you don't put up with your husband's shit anymore, you will feel in control and feel better about yourself. I tell you this only because my husband had the bright idea of taking his A underground for a week. I flipped, and chucked him out. Sure, he could have left me and the kids and not come back. It was a gamble. But the alternative would have been him carrying on his A, me not being able to do anything about it, and me falling to pieces bit by bit. So yeah, non. After, each time he would lie he would end up spending the night at the hotel. This summer I made him move out to a rented room because I wanted space. No, he didn't stop lying, but at least he knows the deal now. Which made me feel much better.
As long as your husband doesn't get real consequences for his actions, he will carry on his game. Hence him driving to meet the "lady" even though he knows it's destroying you. And apparently doesn't give a squeak about it.
Big hugs.