In overweight men, it's not uncommon to have something called the "buried penis" syndrome. That means that you get a mound of fat, called the pubic fat pad, around your genitals. This makes your flaccid/soft penis appear smaller than it really is. If you push the fat/skin around your penis inward, your shaft should extend outward making the penis appear larger. When you're hard, it's still buried in the fat mound, so the erection appears smaller too.
If you're overweight, weight loss can make it "grow", but you may also need to do specific exercises to lose that fat pad. If you're not overweight, then have it looked at by an internist/family practitioner or urologist. It may be a hormonal condition.