I'm in the same boat with you, though I have a small toy instead of a vibrator. I know comfort is a big deal, and even if things feel super great, only my toy works for me.
I suggest you masturbate more often without your toy, and try to learn what you like. Tell your man these things, or even let him use your vibrator on you. I'm nearly positive he'd enjoy it.
That way he can have the same joy you do when you make him go.
Plus, being open and talking about things is very important and can make a big difference. That includes whither or not picky-cummers get to go in bed.
Of course, this doesn't mean you have to go, and I'm sure you know that if you're already enjoying oral and all.