Like many things that we consider "wrong" I think the fundamental thing, as another poster said, is the damage it does to others. So you get to have some extra orgasms, and for that benefit, you subject your partner to years of torture, maybe subject your kids to a broken family and all the other negative effects we know are a result of an A. The "benefit" gained for you is at a HUGE cost to other people.
I like to think of this in "happiness points" as a way to explain why we consider some things, including A's, so wrong. Let's say you and your spouse are both at a "5" on the happiness scale. You decide to go out and cheat on them, and, while we often dance around the issue, I'm pretty confident when I say, "Cheating is fun". You get to have sex with someone new, live in an imaginary world for a little while, maybe even think your "in love". But, whatever the motivation, let's concede, "cheating is fun" and that fun takes you from a 5 on the happiness scale to an 8. Your partner, who doesn't know, might stay at a 5, or, if your cheater is more like my W and starts to throttle back the sex/affection/attention at home, the cheated on partner might fall to a 3 or 4. But still, overall, it's a good trade, your partner (the WS) adds 3 points, you might lose 1 or 2.
The problem, of course, is discovery. Now the entire equation changes. Let's take my situation, say my wife was an 8 (happy) before d-day, and I was a 4 (she wasn't treating me very well during the A, so it brought me from my normal 5 down to a 4). Well, let's talk through d-day. My happiness upon discovery went from a 4 to a -100. When I confronted her, I'm not sure what happened in her mind, but let's say, I'm pretty sure she fell right off cloud 9, so let's say she fell to a 1 on the happiness scale. When the OM found out I knew, he swiftly moved to damage control, blamed everything on her, and ran immediately back to his wife. The realization that there wasn't "love" in the A, IMHO, really blew my wife's world apart, and she went from a 1 to a -50.
So, pre-A, I was a 5, she was a 5. Post A, I was a -100, she was a -50. And we stayed like that for a LONG time. In fact, I'm pretty sure as my happiness started to slowly creep up, hers fell further. And this went on for years, over an A that was a few months long and included about 10 sexual encounters. Years of time stuck "deep in the negative" for happiness for what??
It's just a terrible trade, and I think that's the fundamental reason it's wrong, the perpetrator gets a little bit happier (or maybe even significantly happier if they are seeking the things an A actually provides) but then, upon discovery, all of that evaporates, and the happy "bonus points" from the A are paid for DEARLY by the negative happy points after discovery.
Most crimes in our society follow this same model in determining the severity we associate with them (also, IMHO, as an aside, why I have real issue with victimless crime, there's no loss of "happy points", so.. What's the problem here?). Even if you have a really good reason to murder someone (like they slept with your W, for example), if you do it, you get a little happier, but the person you kill, his family, his friends, those who love him, they get profoundly less happy, likely for the rest of their lives. And that causes us to view murder as one of the most serious crimes, because it has such an unbalanced cost/benefit analysis when viewed through the "happiness lens".
Summing up, it's wrong because it makes the world a shittier place for everyone, it takes happiness out of the world and, in aggregate, makes all of our lives worse than if it never happened.