It will get better. My WH and I try to do something special on those days (the anniversary of his PA with a coworker is coming up on the 26th). We take the kids out apple picking, or go to the movies. Just something to get us out of our heads.
Trust is like paper. Once it's crumpled it can never be perfect again.