It does get better. At first my kids didn't mention OW at all. Then they did start talking about "this person they knew" without mentioning her by name, but it was obvious that's who they meant.
And then, earlier this year, about 3 years after the divorce was final, ex married OW. My young adult boys now freely talk about "the crazy step mother" and it doesn't faze me anymore. In truth, most of it amuses me.
None of it is fair. But with time and healing, you can get to a point where it truly doesn't matter.