As a wayward, one of the goals is to completely purge the AP out of your life, including any amount of mental space that they may be taking up in your mind. Hard to do if you are seeing them drive by your house on a regular basis.
I can totally see where your husband is coming from as well. He didn't do anything wrong, why should he have to move? The AP is encroaching upon his territory. Your husband is putting his foot down, and not being intimidated by this guy. That takes some backbone. Your husband is a good man.
I hope your husband can eventually find some peace in his own neighborhood. If not, perhaps the topic of moving should possibly be brought up again eventually. It's not about running away, it is about feeling safe and comfortable where he lives. That goes for you, too.
It has been a year, and I am getting better at seeing her out in public - I not so shaky/hurt/irritated. I honestly think it will be kind of like an innoculation for your BS eventually. Your H will get used to it. But, if he were to feel strongly about moving, then you should. It will get better though, I promise.
The AP may be a bit of a narcissist, or else he is just inconsiderate and clueless, so tell your H the thing that will make AP the angriest, is pretending he doesn't exist. I wouldn't glare at him - I would pretend he is invisible man, driving an invisible truck. Because really, that is how important he is.
Divorced from and remarried to XBH