Most men worry and moan that the size of their penis in not monsterous enough. They worry that they will not fully satisfy their wife or girlfriend. Some women are satisfied by a teeny weenie but many are not.
Here is a link to the most complete survey and study on whether size matters.
(edited to remove link)
If you have a hot wife or a girlfriend don't fool yourself into thinking your little inchworm can get the job done. That is why there are men like me. If she is not cheating she wants to.
You should have a frank discussion with her and let her know it is OK for her to take a lover. If you love her you will not want her to miss out on hot sex.