Why can't I get hard all the time. Condom issue?

A little background, I'm in my 20's, healthy, very athletic. I get morning erections. I can have sex daily, even have sex 5+ times (ejaculating, and then having a short 15 min-1 hour break) and then going again with no problem.

However, sometimes with some sex partners I have difficultly getting hard and staying hard. For the most part this is when it comes to putting the condom on.

With a recent sex partner, we had foreplay for about 20 mins. I was (75%) hard, then successfully put the condom on, and tried to penetrate her. However, she was tight (and not too wet), and pushing me out. I think I was panicking trying to get it in before I'd lose the erection. Eventually we realised this wasn't going to work. So we resorted back to oral sex, at which point my erection was intermittent (going from soft to hard temporarily, then back to soft). Eventually we decided to call it a night and accept our (my) failure.

Sometimes I've been successful with condoms. But usually the process of taking a break from the intimacy, and focusing on getting the condom on and penetrating makes me lose my erection. If I'm able to penetrate in sufficient time, then I can be alright, and there is no problem with the sex. It's just this tricky phase.

Any tips. I'm sure this is purely psychological. I think I also have some silly self-esteem issues, whereby I feel I need to prove to other people that I'm better than them, or that I'm worthy. But I don't know how much that affects this of it has anything to do with it.