I think quite simply, it's a case of you having less control over what's happening. If you're having sex, most times you can control the speed and other factors like that. Whereas with oral sex, you really don't, and in addition the visual stimulation may be more. Don't forget too that oral sex is different to normal sex (obviously), so you can't expect the same response to it. I think the best thing to do is ask your partner to 'go slowly' or something like that. That way you'll last longer. I hope that helps!