Oh my can I relate. First of all, I'm so sorry for what you are going through. It's quite possibly the most difficult thing you will ever go through.
I was not a "touch" person for years, which is partly what caused him to think I didn't love him and fall for someone else. His love language is physical touch (if you haven't read Gary Smalley's book on the Five Love Languages, you should). Anyway, somehow ever since the A, I am TOTALLY a touch person. When he first told me, I immediately let him hold me and comfort me, which I find kind of odd.
At any rate, I have figured out "why" I had issues with touch and the fact that I am basically over that has changed everything. Is it possible there's a reason why you haven't wanted to be very affectionate with him in the past, before the A? This is something you might look into if you go to counseling.
Having said all that, it is also very normal for you to have issues being affectionate right now because of the A. It's a strange paradox. And, yes, sadly it will be a roller coaster ride for a long time to come.
Like you, I have done everything I could think of to be the wife he needed, so much so that he really didn't have to do anything. He was remorseful and finally was able to fall back in love with me and all, but it took me packing a bag and leaving last week to get him to realize that him to finally read the book, "How to Help Your Spouse Heal from Your Affair". He just wasn't getting what I was going through and it affected things off and on, and creating an ultimatum was the only thing that seemed to work. The book is amazing and often recommended here. I read it as well and it's good for both spouses to read. It very clearly articulates what the betrayed spouse is going through, and also helps the WS understand some of the typical thinking patterns of WS's before, during and after an affair, so it's good for both partners. It's also relatively short. Could be read in one sitting by some. I think I read it in three sittings.
After he began to read it he said it was a good book and that he should have read it a long time ago. YES! That was amazing to hear. If you can get your husband to read it, the sooner the better for both your healing.