Well that shows how much of a mature gentleman that guy is! To tell u the truth, most gentlemen and mature males do not tell about their sex life with a particular girl, unless they dont care about the girl at all! That ususally happens after the couple has broken up. But in ur case, u two arent even dating?? So it's very likely for him to brag about how he popped ur chery.....And yes guys are jerks!
And just forget about him, just let it go, the more u try to defend urself, the more they are gonna talk about it or even add stuff to it and before u know it, it's all over the place, so next time u have sex with a guy again, please know for sure u really can bare those "talks"