Do you know why they are asking?
Are they looking to go on a trip with dad or looking to visit relatives that they do not see very often?
One, you have no control over him.. you know that. So you can't answer WHY he went without them.
But if they are looking to visit relatives.. can you take them to visit relatives and make yourself scarce for the weekend? I have done that with my kids. XH would not take them to visit his family or friends. But the kids want to spend time with family..it has helped my kids.
It isn't easy parenting with a lower muppet.. but we manage.