When there is a person that is bothering you like that, you can always call him out for harassment and get a restraining warrant....or you could just plain ignore him. It's very hard to figure out exactly who is the father when you are thinking of the many dates and when your period was and how many times you had sex and when you did...... Sperm can stick around for a while, so you can't be sure which one made it, and douche isn't really that effective for preventing pregnancy.
Since you are going to have the baby anyways but still want to be sure about the paternity, a simple DNA test will solve the problem after the baby is born. I hope the father is the one you are hoping it is.