I am sorry this is happening to you. I will say that this conversation and any similar to this are his way of deflecting. You said "hope things work out with us". He can't say "Me too..." or "I am trying to figure things out". Because phrases like that would be admitting he is having an A. I read your previous posts that indicate he has not admitted to anything and keeps denying.
What he does instead is get mad at you to throw the heat off of what he is doing. Now you will cower and he has lived to lie/(technically not lie because you didn't ask him any questions, because he was mad and you didn't want to anger him further, possibly, so you turtled and relented just like how he wanted).
He will keep this A going for as long as you let him, and IF you don't confront him, start the 180 even throw his A$$ out then I am afraid this really has no chance of ending the way you want it too. He will eventually leave you because you will have completely become a "whiny, low self-esteemed, resembling nothing of your former self person". That is not attractive to anyone.
So please, please, please, do not let him keep getting away with this. Confront him, call this woman, call her husband if she has one, and get this thing in the open so that your marriage can either start healing, or you can finally have freedom from this prison he is slowly but surely putting you in.