This could be a very serious problem, and I would advise you see a urologist immiediately. The reason being it could very well be a hernia in the lower groin, pressing against the tube that carries seman from that particular testicle.
There are a few infections that will cause pain in the testicles, however, a hernia is the most obvious one when only on one side.
Do you have any other symptons, such as pain on your left side, can be very dull or mild, etc?