Left Testicle higher and smaller than right (with pain)

Hi All ... am a 40 year old man that has no problem getting an erection and had no problems having sexual intercourse and even almost became a father in the past. Had testosterone levels checked years ago and they were fine (as per the doctor). Never had serious problems but throughout my life since I can remember, I had:
a) left testicle a bit higher than right one and smaller than the right one
b) left testicle having occasionally pain that can be anything between negligible pinching all the way to annoying pain that shoots throughout the testicle and my left lower abdomen and sometimes connect to nerves that reach my lower back
After a left knee ACL operation in 2003, I had this pain appearing together with knee pain just above the kneecap.
This knee pain went away about 2006. Left testicle pain still there almost every day .. there are some good days that this does not appear but there are most days that it is there and I kind of learned to live with it.
For the last 15 years I had been a smoker and recently I quit (January 2010). Right after I quit smoking, the left testicle pain started appearing more often and being sharper yet still bearable.

All along the years, when I have the pain, I locate it with my fingers coming from the nerves / muscles that hold the testicle ... Never felt that I had torsion because from what I read, it is an emergency and the pain can be extreme but I never got there ...

Don't ask why I did not check this in the past cause I did and doctors told me anything like:
"no body has complete symmetry so don't worry", " testicle size, hair, location seem normal" etc
and since I never had issues with ejaculation and getting an erection, I thought I would follow their orders and not worry about it ... I think it is about time to get this baby taken care of

Thank you all ... Any help and advice is appreciated
Regards