Yes, the average is indeed 5-6 inches. If you doubt that it's probably because your comparisons from porn. You can find a breakdown of the size and percentages online. Many are bigger, many are smaller, but 5-6 is the average.
If you don't grow any bigger then you have to realize a couple things. First, realize the only thing that makes it a big deal is you thinking it's a big deal. Having confidence in yourself is more important than having a big penis. Depending on how much you let this bother you, determines how you will achieve it. If you obsess over it, maybe a therapist would be needed. Otherwise, just having enough confidence to go out and live, get a girlfriend to love, and see how it doesn't bother her might be enough to make you confident.
Second, you need to realize that you don't need a large penis to satisfy a woman. Most women achieve orgasms through clitoral stimulation. That's achieved by your pelvis rubbing on her during sex, or using your hand, tongue, etc. A larger penis does not mean better clitoral stimulation. In fact, a smaller penis might mean shorter strokes, equating to more pelvis contact. Additionally, many women do not achieve orgasm through intercourse at all, so it's important for all men to learn other techniques. Learn to please women with what you've got, and you've got more than a penis. Finally, if a woman is able to achieve non-clitoral orgasm from intercourse, it's usually from the g-spot. The g-spot is reachable by fingers, so certainly reachable by 4.6".
You're absolutely not alone in the no-lube masturbation. It depends on preference, sensitivity, and probably how someone did it when they first started. It's my idea that it would also depend on foreskin. Without foreskin it would probably be uncomfortable run your hand over your glans without lubrication. With foreskin, you're never running your hand over, but instead rolling your foreskin over the glans. That doesn't mean that circumcised men can't masturbate without lube. I'm circumcised, but still have enough foreskin that I can use it to masturbate without lubrication. I'd say that 90 % of my masturbation, my whole life, has been done without lubrication.
Finally, if you're considering looking into enlargement possibilities, read some of my comments in this thread:
http://ehealthforum.com/health/topic82897.
html