I've noticed taller, skinnier guys for some reason have bigger members. I'm 6'2", thin, with a 7 incher. I'm 17. I guess it all comes down to dumb luck. They say black guys are supposed to be biggest, then latinos, then white, middle eastern, asian. I'm not sure how accurate that is, but I know a black guy and a latino guy who aren't very big.
That aside, I would think size doesn't really matter. If the woman you're with is really not cool with your penis size, then she's probably not a keeper anyway. Girls shouldn't judge based on penis size just like guys shouldn't judge based on breast size.