You say you miss revealing your fears. Do you feel restricted in your ability to talk with your husband?
Gently, it's not enough for an affair to end. What is your husband doing to help you heal and feel safe again? What are you doing?
Three years is not unusual, in the spectrum of healing. It's not unusual, even within the context of a healthy and robust R, to have doubts and concerns along the way.
But what concerns me is your statement that you no longer reveal your fears; it suggests rug-sweeping. It makes me wonder if you're channeling your efforts in the right direction. Are you struggling not to rock the boat--using valuable energy there-- or are you working to build a marriage that is healthy and rewarding to both of you?