When is enough enough? When enough time has passed for your BBF to heal. There is no magic timeline that says someone can recover from this in a month, a year or a decade. Two sentences you included cannot coexist.
But at what point is it enough
I love him more than anything in this world and would do anything to stay with him
There are many people in this world who will say they will do anything for their BS/BBF/BGF, but what they really mean is they will do anything as long as it doesn't continue to hurt them. If your BBF taking longer than you would like to heal is your breaking point, then you are not willing to do anything. You are only willing to do something short of anything.
Remember that when we shatter the trust of our BS, no number of times we repeat what did and did not happen can replace their fear. You can say over and over "it was just a kiss", but how can he believe that? His mind is going to the worst case scenario. That you are making light of your actions by thinking of it as a "small A" and that what really happened was something much bigger.
When is enough enough? When either you or he decide that living life this way cannot continue. You do have a choice - to walk away from the relationship. So does he. The fact he asks for you back is a sign that there is enough in him to think there might be a future worth hanging on to.
If you are willing to do anything, then stay. Get IC for yourself, consistently offer to go to couples counseling with him if it will help him, and face the consequences of your actions.
If not, then leave the relationship. But admit to yourself that you were not willing to do anything, just something less than that, to save the relationship.