My W doesn't organize her experience sequentially. I do, and it looks like you do, too. I'm not sure what you're asking, but I think my response may help.
My W didn't blameshift or minimize, and she actually came clean on d-day. I got more details as time went on, but the details were always congruent with what she told me on 12/22/10. And I asked every question in multiple ways, always getting essentially the same answers. I couldn't get dates, though. She's very smart, but she just doesn't do dates.
By that I mean she doesn't do dates on anything. We have similar memories of events in our lives together (which began in 1965), but she just doesn't associate events with dates.
If your WS remembers dates about many things, but doesn't remember A-related dates, he's probably lying. If he is weak on dates in general, he may just process experience in ways that you don't understand. I sure don't understand my W's methods. He very well might be telling the truth when he says he doesn't remember dates.
You're coming up to what SI used to call an 'antiversary'. Since dates mean a lot to you, it's normal to feel some of what you felt the first time through. My W's A lasted from August to 12/22 in 2010. I was wreck from August to 12/22 in 2011. I was also a wreck during that period in 2012, but less than 2011. 2013 was better than 2012, 2014 better than 2013. Now I don't notice the period, but we still mark 12/22.
My point is that the antiversaries brought up pain for me for at least 4 years, and my W came clean on d-day. Your H did not, and I don't think there's anything unusual in your feeling pain after 4 years. IOW, I think this is part of your healing process.
If I read you right, all I can say is that this sort of angst goes with focusing on time and sequence - it's the downside that comes with the upside of being time-focused.
I think it's a great idea to share what you go through with your H. Letting him know that the dates bring pain with them is a loving act. If he's really on board for R, he'll respond with support, even if you have to ask for specific support - and that support will help you deal with your angst. If he doesn't step up, you'll know there's an issue that needs to be resolved for R to proceed.