My BF has the same problem - he loves any kind of sex and he can even outlast me, but he can only climax through a handjob. Then again, he requires a lot of attention to reach that point, even via hands. Some guys are just less sensitive than others in that area of their body and require more stimulation than others. I've been with guys who would go off in less than a minute, and others who absolutely would not go off no matter what I did or how long I did it for.
There's not a problem with you - you just may not be as sensitive to touch down there and I don't think it's unusual. Some people are very ticklish, some not at all. Some people can handle a lot of pain, some can't handle any. We're all different, are we not?