You don't need to be emotionally concerned unless it's disrupting the sex, then there is a problem.
However, if he masturbates too frequently (more than 3 times a week) without replenishing himself or allowing time for recovery, the quality of the sex you have with him will diminish drastically and he can even hurt or bruise himself.
Don't believe me? Look how many posts are in the "premature ejaculation" thread above. I GUARANTEE YOU that all of those guys masturbate too often and don't take care of themselves enough to recuperate from it.
Sex educators say "masturbate all you want! it's harmless!" but this is a lie, because masturbation, and sex in general, require an ENORMOUS amount of energy and nutrients.