My situation is different and I understand your frustration but here is what happens on my end. For months including vacations with friends and even work everyone adjusts on a whim notice if I am able to see my child. People will work for me and sometimes I don't know. That being said everyone is in a different circumstance. He wants to see his child like you would want to see your kids but he is not handling it the proper way. It is not a reflection on you or your children or even your relationship and depending on how often she decides to let you see your child you would jump at the chance as well. It could be something else but if you know him as a kind person then this is most likely it.
I think to fix it:
File for visitation or set up a schedule with her
Stick to that schedule and if she needs him he sees his child when he is not occupied but his visitation schedule includes tons of time where that is reasonable