It's the only thing I have that's mine, that only belongs to me.
As SandAway says, that sounds selfish -- but welcome to the club. Selfishness is one of my own character flaws that enabled me to have an affair against my own beliefs about right and wrong. In fact, in my opinion, all of us who have had affairs are, to one degree or another, selfish.
It also sounds a bit self-pitying. As in, "Poor me! I don't have anything else, so I can't give up my affair." Well, welcome to that club, too. You'll see that reflected in all sorts of rationalizations for "why I had an affair" -- often in some form of "My marriage/spouse didn't make me happy."
The key is not to get stuck in selfishness and self-pity (or a host of other unhealthy thought patterns that lead us to betraying our spouses) or in shame or self-loathing (or a host of other unhealthy thought patterns that often follow disclosure of an affair).
Here's my suggestion. Take it for what it's worth, coming from someone who is hardly a world-class expert on relationships.
1. Remind yourself, over and over, that you have much more than the affair. Make a list, mentally or on paper, of some of the things that are important to you. Given the season, you might make it a list of things you're thankful for. The list might include your spouse and your marriage, but be honest about that. It might include your son, your extended family, your job, your self-respect, your own personal values, your reputation, your health, your financial well being, your home, etc.
2. Remind yourself over and over that your affair can -- and likely will -- take away, destroy, or at least seriously damage many or all of the things on your list.
3. Remind yourself over and over (even part if you doesn't believe it -- in fact, that's the part you're talking to when you tell yourself this) that some, most, or all of the things that you find appealing about your affair or your affair partner are not true. They are illusory. You created them in your own imagination. For example, you think you're looking for things that are missing in your marriage, so some part of you must think your affair will provide them, but another part of you recognizes your affair partner is emotionally unavailable. Remind yourself of that last little tidbit.
4. Stop the affair. Now. Not just until this afternoon. For good.