Above all, for now, base your self-esteem on what you read in responses to your posts on SI - you're loving, you're lovable, you're capable, you're terrific.
Then, realize you're under tremendous stress, so you're predisposed to feel lousy. The A followed by job loss is a heavy, heavy load. The best anti-dote is loving self-talk.
Second, keep your eye on the prize - a good M. A guy who does nothing more than no cheat isn't much of a catch. You deserve a lot more than that.
Third, have you talked with your H about how to handle drinks after work? It may help him turn this from temp to permanent, but you can probably work something out to make it safe.