How long is he going to be in your state?
If it's a week, I'd claim that you were remoting in from somewhere other than your current location.
That is, after your first decline. Maybe start with, "I'm so sorry, but I'm completely booked during that time period - even for coffee. I hope you can find something fun to do in the area."
Then, plan B if he pushes.