I'm sorry. My kids have done this several times and it kills me every single time.
Just know that when they are little, there are lots of variables that can cause these visitation meltdowns. Doesn't make it any easier, but sometimes you know that they don't want to go because they are tired or in the middle of something and are annoyed at being interrupted. Sometimes they don't feel good and want mommy. Other times, they have what they believe is a legitimate reason for not wanting to go.
I would say that it would be best if your ex tried to have the same routines as you. Also, he needs to early on make sure that your DS sees his place as another home.
Unfortunately, many wayward exes don't think that far ahead or that much about their children. This is where my kids get hung up. They have a hard time with different house rules and differences in their comfort levels. Ive tried to point out that routines are important, but i cant control if he wants to employ them at his place. He now lives with OW so her rules are their rules. Unlike at my house, the kids there are not allowed to go into the fridge or root around the pantry for a snack. It's amazing - they are treated like visitors but exWH gets pissed if they don't consider his house their home.
Anyway, I know it's hard, but at this point you are doing everything you can. Just keep your eyes and ears open for signs of anything more serious in terms of reasons why he doesn't want to go. Luckily, I would say nine times out of ten, they come out of it and end up having an ok time.