Firstly, I want to say how sorry I am for your pain.
You don't deserve this and I hope your recovery goes quickly.
Your comment about "size" was a "trigger". Many of us feel inadequate in size. This usually begins with the experience of seeing our fathers naked and comparing our "size" to his.
I'm guessing, pure speculation, your husband may have been hurt regarding his penis size in the past.
Women who engage in affairs are, by the "nature of the beast", self-absorbed and self-interested. They have little empathy for their AP and are not
at all careful with what they say and do to him.
His comment to you about his attraction may have been a retaliation.
If he is equating "attraction" to his sexual responses, he is probably not considering the facts.
That he has chosen to place himself into a predicament which can seriously alter his sexual responses for time to come. His responses may now be affected by guilt, by self-recrimination, and even by simple retraining brought on during his affair.
It is also not altogether uncommon for men to develop a period of psychological ED following an affair.