My XWH threatened to take me to court all the time when he didn't get his way. Threatened that he was going to take DD away from me. The first time, it scared the shit out of me, I called my mother crying, I emailed my lawyer, I waited for the notice to appear that never came, he did it again, again no court. I realized then that it was just a bullying tactic.
Next time he came to pick up DD, I looked him right in his fucking face (and I NEVER talk to him, ever, even during handoff of DD) and said "You better cut this fucking shit out about how you're going to take her away from me. You hearing me?" well, he was shocked and just mumbled something about how I need to be reasonable, etc. etc. So far he hasn't threatened it again.
He's all talk. Don't give in and don't let him know he upset you. Keep waiting for the court notices that never come.