As a FWH, I would like to offer my thoughts. You are all correct in that this is totally UNFAIR and the BSes deserve to feel the way you do. In my opinion, things will never just "be ok". We have done fatal damage to our marriages and our spouses. None of the consequences we face are as severe as the hurt and pain the BS feels.
As for what I view as my consequences, they are largely internal. Most glaringly is the shame and remorse I feel for what I've done. Not just the act itself, but the hurt and pain it has caused my wonderful BS. She is the person who chose to be with me (even though she's way out of my league). She is the person who knows me and understands me more than anyone ever can or will. She is my lover, my best friend, and my support. She is the mother of my child and the person who truly makes me happy just thinking about her. And yet, I did this. While it seems like not a lot (in terms of consequences), some days it's hard to look in the mirror and not want to put my fist through it because I hate the reflection I see because that man hurt my wife. Another consequence is having family members, loved ones, and others know how much I have hurt my wife. Being labeled or viewed as a cheater, liar, or simply a terrible spouse makes me want to curl up in a hole somewhere and never come out. While their opinions don't matter as much as my wife's, it's still hard to look those people in the face knowing that your shame is out there and it's hard not to feel judged or looked down upon by them.
Another consequence is the fact that I have to explain to my kids why mommy and daddy are fighting and to see the toll it takes on their lives. They obviously had no part in this, but they are affected no matter what happens in terms of our relationship. I have made a point to talk to my step daughter throughout this to let her know it was my fault that she shouldn't be upset or angry at mom.
For me, I also look back at the marriage we had before. I think about the good times and how we used to be able to sit around laughing and joking. Those days are fewer and far between after DDay and I miss those days. Not because my wife wasn't yelling at me, but because our marriage was innocent and we were just happy to be together. One of my consequences is knowing I had everything I always wanted and was willing to risk it all. I had a beautiful wife, wonderful kids, and a future of memories to live and dreams to fulfill. Now all I have is a big question mark and a prayer that someday I will have the chance to make this up to my wife and show her this will never happen again. The consequence of possibly never having my wife's trust and never having those good days last is a huge consequence because I potentially threw away one of my own dreams.