This is one of those "he is trying to make you play his game" situations. If you play his game, you can't win.
I would send a "Great! I hope the kids have a wonderful time at the beach. Can I get an extra night the week of (whatever his next week) say a Tues. and a Thursday dinner?"
This way if he refuses and you do end up in front of a judge, you look reasonable and flexible and he looks like the ASS he is.
As for the drop off and pick ups, I would minimize the inconvenience to you and let it go. So instead of you sitting around in his driveway, suggest that if his time is imprecise, perhaps he would bring the kids home to you?
If you can't make that change, I would be sure I had a book with me at all times or something I could do in the car.
This is a no win situation for you. It would be expensive to litigate. The likely outcome would be the judge warning him to abide by the agreement. He wouldn't. Eventually he might get found in contempt? But in the meantime, he has bitched and whined about it to the kids, you'll have spent thousands of dollars and time in court.
I would ignore it and hope he grows bored with being an ass when it doesn't appear to piss you off.
The teens are close to phasing out visitation. At worst you have a few more years and this will be a non-issue. Support the kids as best you can and they will appreciate the less drama and strife.
I am sorry. This sucks. It is unfair. I did not like this type of advice when I got it. But truly, when I quit fighting, it got better. Easier. I felt less anxious and stress. The kids are still late (often!) and sometimes they are anxious about it. "Are you mad because we are late Mommy?" I suggest that next time they send me a text so I won't worry, hug them, and move on until next time.
You can only control you.