Sad.
So typical.
A lot of your assumptions about your relationship with your husband may be more flawed and mistakened than you realize.
Life's regrets....
The love but not in love, usually has to do with self and what you are doing or not doing to maintain and nurture the relationship. It usually is significantly more about self than what your partner is doing or not doing. Plus... More often than not, your partner is acting the way he does, more often than not, as a response for what you are bringing to the relationship.
You can't spare your partner the pain from infidelity. It is a debt that has a very high interest rate over time. Pay it up now and let go of the consequences...
Then, build a new one with who both of you are and end up. Grow together or grow apart. Yes, painful for all involved. But, potentially very rewarding for all as well.
Or, pay it later with even more pain and consequences. Or, watch it drag the relationship down to a ghost of what it could be... Usually causing lots of slow steady pain along the way.
Nothing swept under the rug goes away... The dirt is still there. The lump under the carpet. The allergies react even if the dirt, dust, and pollen is out of sight. Black mold is still unhealthy for those who live around it, even if you don't know it is there or where it is hiding.
All of your actions and reactions to your husband now include this variable. Your husbands actions and reactions will be based on your behavior. What you do, or don't do now as a consequence. Only, your husband will be ignorant of the true facts. He will fill in the missing information based on false assumptions. Usually... Blaming himself significantly more than you realize.
Or... Perhaps.... In the doldrums of a failing marriage, he will also reach out for someone else to take care of what is missing in the relationship.
It sounds like you have invested some time and energy growing and developing the relationship with the other man. Time and energy that could have been invested in tge marriage relationship. What would your marriage be like today if you had invested in it instead? Would you be feeling differently about your husband?
Actions and inactions have consequences.