Not to be rude but how old are you, the father and your son? How much time has this guy spent with his son? I have a 6 year old son, a 2 year old son and a 5 month old daughter. My husband works out of town for 2 to 3 months at a time. When he comes home all the kids act fine. None of them forget who he is (granted our daughter will have no recollection of him but she doesn't fuss in the slightest around him). He rarely feeds, bathes, changes diapers, etc but just because he doesn't do it often doesn't mean he can't if he has to. I think this is more you trying to control the situation, even to the detriment of your son. Why would you even want to keep the two of them apart? Sooner or later the father is going to completely walk out and then you will be complaining about an absentee father when in reality you played a large part in him walking out of your son's life.
You need to set up days and times when this guy can come get his child. Yes,
get his child. He doesn't need to hang out at the house with you just to see his son. Set the schedule firmly and advise him that if there are to be any changes you will 24 hour notice. Otherwise, have his diaper bag packed and have your son dressed and ready to go. You might as well wise up to the fact that this is how it has to be because I assure you if he goes to court over visitation
he will win. The courts recognize a father's right to unsupervised visitation (barring him being convicted of child abuse, neglect or endangerment) and overnight or weekend visitation. I suggest you stop thinking of yourself and start thinking about what is best for your son.