Everytime you post I get more angry. The way he is treating you about fishing, doesn't have time to babysit you, about money, about everything, just makes my blood boil.
Of course he wants you to stay. HE NEEDS YOUR INCOME TO MAINTAIN HIS LIFESTYLE!!!!
He clearly does not, or thinks he does not love you. No one treats someone they love in the manner he is treating you. If he were innocent, instead of getting angry when you accuse him of an affair, he would be heartbroken that you didn't trust him.
Get to an attorney pronto. Ask him to move out. Then he can pay his own bills. If you can susstain yourself financially, then there is no reason for him to live there. He is giving you nothing. Not money, affection, fun, nothing.
Let him have his fishing buddies, his woman, and his own bills. Then you can relax, do things you want to do, and maybe find someone who will add something to your life.
So, so, sorry you are being treated this way. It is just incredibly mean.