My kids don't have autism, but I work with kids on the spectrum, and I understand how exhausting it can be.
You definitely need some down time, whether it's just to sit around and watch tv, or to get an oil change or a haircut.
When my daughter had some severe medical problems, I was initially very reluctant to send the kids to their father for the weekend. But it was totally wearing me out, and making me ineffective as a parent.
Eventually, when it became apparent that her issues might become permanent, I figured he was going to have to learn to deal with it sooner or later, and he might as well start now. I made sure he had the contact numbers for all of her doctors, and had my cell on and with me at all times when he had the kids.
At the same time though, you can't force someone to take the kids if they don't want to.
My only suggestion is to see if on those days when he's supposed to have your son and won't take him, is there any way you could get him to pay for a sitter? just so you can get out at least for a few hours?
I know it's very difficult to find sitters for some kids, but you may be able to ask his teachers/aides if there is someone who knows him and might be interested in sitting for him on the weekend for a few hours. This might help with transitions, behavior, etc.