You REALLY need to think this through.
I'm not saying DON'T do it. Plenty of couples do. And plenty stay together afterwards. But if you aren't sure, particularly with this guy not wanting you there, then this may not be the right person. There's NOTHING to say he's ever going to change his mind, and thus you'll never get to see the action... which is what you originally said you both wanted...