Don’t feel rubbish, you’re being honest which is much better than hiding your feelings and spending that weekend going out of your mind with worry. He wouldn’t enjoy it anyway knowing what will be going through your head.
You’re not saying never, you’re just saying not this time. Unfortunately it is his own fault that you don’t want him to go. But the good news is that it sounds like he is totally understanding of the fact that you don’t want him to.
I know how you feel because my husband and I have never been jealous or stopped each other from doing anything and one of the first things I said to him was that I hate that he may have made me into one of ‘those’ wives who doesn’t trust and questions who he is with and where he is going. It just isn’t me – but maybe it sometimes will be from now on and he has said he will accept that and will never go anywhere I do not want him to. It sounds like your husband feels the same.
As another idea – could you go with him? You said you used to train a lot so could you get back into it enough to be able to keep up on the training weekend? Or even just go along and stay with him and he can say to the group that you fancied some fresh air and long walks? Could be a good compromise because then you don’t have to stop him from going if you think it would be good for him but you also know where he is and who he is with.