Generally an infection would have happened had the condom been left inside for an extended period of time. It doesn't sound like it was.
In any case, the nurse/doctor would have noted an infection. And if they thought an infection were possible, they'd have said something more about it.
Infection does not always interfere with child bearing. In cases where the infection is long term and ongoing and causes scar tissue, that is where the problems arise.