Those friends need to go. I lost two of my oldest friends as a result of my A and two others who were dear to me. They all knew, they all covered for me, supported me, cheered me on at times, everything.
Yes, my closest one in passing mentioned once about the risk I was taking to ruin my marriage. That was a well placed thought (sounds like your friends have some of those as well) but in the end, he was silent partner to the A which means he was not a friend to the marriage. They can't gain that status back so out they go as part of the healing process.
Good luck! Losing friends is one of the hardest things I have had to do after losing my wife.