Limp penis during sex

I have been dating my boyfriend for 3 1/2 months now. About two months ago we decided to have sex. Nothing ended up happening because we did not have protection. A month later or a little less we decided to try again. This time we got all hot and heavy but did no foreplay..he just got right to the point and put the condom on but, he went limp right away. And we did not do anything to try and get it up. He was very embarrassed and all this happened 2 months or so ago. We still haven't done anything. I asked him why he doesn't want to have sex with me and he says it's either because he's too small or I am too big! He said he couldn't feel anything last time. That, to me, is very hard to believe because I do not have any kids and I've only been with 3 men total. But then again I am self-conscious. I just hope it is not me. So what I am trying to ask is, why would he have gone limp? It was our first actual try at sex. He's 21 and has only been with 2 women and he has a child.