Until you have a court order the compels him to tell you where he lives (which is standard), then you need to tread carefully. Otherwise he can claim that you are keeping the kids from him.
What you CAN do is insist that you must have the address and land line IF he is keeping them overnight. You should never allow your children to go anywhere, including with your X, if you do not know where they are and how to reach them in case of emergency.
If he is only taking visitation for a few hours then let them go as long as you have his cell phone number.
Now what is important is that you document, document, document. Get a calendar or a notebook or a word doc and document every communication and interaction he has with the kids. Document the good along with the bad. Document EVERYTHING, that way the court (and he) cannot claim that you were biased in your record keeping. Try to communicate only via email or text so you have a written record of everything that was said. If you ever do talk to him over the phone or in person, follow up with an email summarizing what was discussed.
Finally, you need to let go of control over what he is doing. You cannot control him. And the court believes that what he does during visitation is no concern of yours as long as the kids are safe. It sucks, but you need to embrace it.
Good luck and (((hugs)))