No, you are not being unreasonable.
He has a thing for her, at least you suspect.
While your betrayal was active this man helped to be a mediator (nothing happened I understand, he tried to reconcile both you and her), so it does not scream "affair".
But, he is a man who allegedly "has a thing" for your wife (aka likely sexual attraction). You are not in a good place right now. I do not know your wife, so I can't tell you if she has the hallmarks of a potential cheater or not.
But you feel in your gut that if this guy ever has the chance he would gladly sleep with your wife, correct?
My 2 cents: friendship between man and woman can exist but is conditional. He is attracted to her. We don't know if your wife reciprocates or has reciprocated at some point in time. You are passing through a moment of crisis. She is investing emotional energy outside the relationship into this guy to find comfort.
Those are ALL red flags. So no, you are not being unreasonable.
All depends if your wife is the kind of person who could cheat (low self worth, people pleasing, unresolved ego validation issues), if so then the risk is substantial (she could even do for 'revenge' and discover later that was bad (worse than now, she will not even anything, just mud herself), but you will feel worse, when you will taste the sting of being the BS yourself, trust me).
You should talk to her, not just about your fears (possibly grounded), but about the fact that's not going to help your R:
- The WS heals the WS
- The BS heals the BS
No one else can fill that gap.
If you were right now in the affair, cheating lying and gaslighting her, then it's understandable she needs a shoulder to cry.
Right now I think this is your pain, between the 2 of you, and in that environment should stay.
When I was left by my Wayward Girlfriend when she chose her AP, for a brief period I went down the road of having an harem of girls, just to try to get out that pain. Did not work, but the state I was in, I know where could drive you. But I am a guy, she is a girl, so take this with a pinch of salt.