I am so sorry you are here. I am also sorry that I have no new answers for you.
The reality is he is using you. You care for him and his children, work to pay the bills and he gets to screw the neighbour while you work. He is a complete arsehole. (I'm allowed to say that cause mine was too until dday).
You MUST stand up for yourself.
I always say to my kids to look at the worst case scenario when looking at options. Try this. What would be the worst result if:
(a) You cut off all money to him, put a padlock on the pantry and fridge and refused to speak to him about anything? I suspect he would either pull his head out of his arse or leave you and go live with her.
(b) Left him and the children. I suspect he would either beg you to stay or they would all move in with the whore. (Assuming she would have them??? Does the bitch work? I wonder how she would feel if she was saddled with a low life like him AND the kids.)
(c) You stamped your foot and said "Sell the house. We are moving far away. Get a job and do your share or I'm divorcing you and I'll take you to the cleaners for CS, so you'll lose the house anyway." Listen to Sumrlady - you may be in control anyway.
Whatever you do you need to make his life miserable. Do not let him enjoy cake eating.
Take control honey.
Finally, I will send you a pm with a little trick I learnt. It may help you.
Good luck and BIG HUGS