I know it seems as though 2 years is a long time however, it was my experience and I've read others accounts on this site that it wasn't really until about year 5 the they really felt like they were past the worst of things and felt as though they hit the full reconciliation point.
What you are experiencing and feeling is normal.
In the 2 years have you had any type of counseling either individual or together? I highly recommend counseling together as a couple.
What type of work that needs to be done is your husband not doing? After 2 years why is there still a need at this point to talk about the affair? Reconciliation should be the focus, finding how to communication and build a relationship should be the focus. It won't ever be like it was before which in my case I'm thankful because it's better and stronger than ever.
Yes, he needs to do his part in helping you gain trust again. That's a hard road but doable in time.
If the affair is the only thing on your mind then that "thing" will consume you whether you shout it from the rooftops or not. At some point YOU need to make the decision to let that go and move on. Not easy, trust me, I know. I had to finally either commit to letting the affair win and consume all of my energy or I had to commit to focusing my energy on me and getting my life along with my marriage back. I thank God every single day that my marriage won! It was a tough battle but worth every minute.