I think every circumstance is so different. My advice to a 25 year old would be very different than to someone in their late 30s. Even still, there is no one answer, IMO.
H and I have been actively trying since we both committed to R. He had told me during the A that he wanted no more kids, even though we always planned for at least 2, and after DDay I knew I wanted another, even if we divorced and I had to do it on my own. Our MC was strongly in favor of our trying again given how badly we both wanted to, and my IC also is very supportive.
It was not a decision we made lightly, and I suspect if we were 10 years younger we would have delayed it, but I will be 43 shortly and time is running out. Unfortunately we have had 7 miscarriages, and we may never get our wish. Still, if anything positive has come from our losses it is that we have handled them together and our relationship is solid, and thanks to lots of IC I believe that even if we can't have another baby I won't resent him for it. (This was a huge concern of mine.)
I suspect that the earlier on you are in R, the harder the pregnancy might be in terms of the emotional side of it and the infidelity. That doesn't mean it would be a bad decision, though, and I firmly believe that if you can anticipate the difficulties you can both prepare for them. I know for us one of the reasons my H became depressed was that he didn't anticipate how much our lives would change post-baby. Having your eyes wide open will help.
Good luck to you whatever you decide!