I'd take him in to talk with an IC. (Or, rather, to putter around with the IC's toys while you talk--that way the IC can observe him while gathering info.)
It may be that IC is NOT recommended (and I'll tell you why in a minute), but it's best to let someone qualified help you decide.
It wasn't related to infidelity, but we had a trauma when my kids were in preschool and fourth grade.
In a nutshell--their father had a very nearly fatal MI, and just as he was recovering, a truck drove INTO our house. Like, through the garage obliquely (totalling one car and damaging another) into the house, before parking in our living room.
(Alcohol and drug-impaired teen with her license only 2 weeks.)
My daughter's bedroom floor wound up resting atop the truck.
VERY VERY scary--middle-of-the-night, bitter, bitter cold, and of course, immediate chest pain for WH (who was, in fact, a WH then, though I didn't know it).
We were homeless (albeit briefly), with few resources, and two flipped out kids----whose father was in ICU as a result.
I called our pedi (who was my pedi when I was a kid) the next day. I wanted to pre-emptively address fears, etc.
We took his referral to a psych in his office, and I took both kids in.
She very strongly felt that focusing on the trauma (or any trauma) in IC can magnify it for little ones. (My son was 3.5.)
It was recommended that our daughter keep in close touch with her school guidance counselor (and the IC communicated with her), but that we just observe our son, and take him in if he became contrary (um, he was 3.5--I clarified THAT one because "contrary" was his middle name at that point! what the IC meant, really, was combative and really difficult in his contrariness).
Her rationale made sense. By focusing on the bad stuff, it's possible to elevate it in importance when kids that age really are very me-centric, and just don't assign as much importance to it.
Yes, Daddy living somewhere else is a BIG deal. But it can be approached and reframed in a way that makes it almost like a fun adventure. I know that this will likely chafe (and I understand this because being positive about my kids seeing their unfortunately-still-alive father and absorbing the brunt of their angst when they get home quite frankly can suck). But that's the tack I'd try to take.
My son now tells me that the night the truck drove into the house was the coolest night EVER. He got to sit in a firetruck in the middle of the night. The firefighters gave him cocoa and hen Diet Pepsi.
Seriously. Diet Pepsi (something he NEVER got), in the middle of the night in a fire truck. What could be better?!
In a similar vein, my daughter and I came across her journal from that year. I almost died laughing when reading her account of her father's (again, NEAR-FATAL) heart attack. OMG--preteen drama. "Dear Diary, the most horrible thing happened! My dear father had a heart attack. But you'll NEVER guess who sat next to me at lunch! OMG, it was TAYLOR. I think he LIKES me!"
Yeah. Truly, we have to be careful not to magnify issues to be greater than they are. At this point your son's difficulties may just need gentle and loving redirection.
I'd ask a professional for guidance, but consider holding off to see if you can work together to ease the transitions. It would be a shame to make a mountain out of a molehill (not that it really is, but again, small children are very egocentric) if his primary concerns are things you can address on your own.
I hope that made sense.