Any change in your status is bound to make you afraid. How your distant friends identify you (involved/single/wayward/betrayed) is another change in the things around you. You cannot ignore that. Just like so many things you have gone through already, take a deep breath and don't borrow trouble that hasn't happened yet.
There are lots of reasons you feel upset. That doesn't mean you are back at the starting line. It is just the next hurdle on your track.
FWIW, I have so many mixed emotions about my friends. I am old enough that I have two groups of friends -- the really old, close friends, and then the group from around the neighborhood/school/whatever.
The people in the neighborhood never need to know. But I kind of want to discuss it with my close friends. I think I'd find support there. At the same time, I am not sure that I'm welcome in my best friend's (since 6th grade) house. He's one of the people who knows. His wife hasn't spoken to me since DDay.
I have a lot of the same anxiety as you do. Honestly, I just haven't gotten far enough along to face it. Where you are at now is still out on the horizon for me. So don't beat yourself up. You'll get through it.
Hang in there.