Penis size is just a mind game that men play with themselves and other men -- and now women are attempting to play it (for whatever reasons they have, none of them usually have anything to do with the actual size of your penis).
Having a smaller penis is only an issue if you're stuck with a woman who has a huge vagina. The real issue is maintaining a good, hard erection...which becomes more a concern with age. But then, it's a completely normal part of the aging process.
So any woman who complains about your penis size is either trying to be cruel because she's mad about you over something else, or is in denial about her over-sized vagina...or just doesn't realize that the real problem for her is she'd like your penis to be harder...and that's a reflection on her, too, when you think about it.