Actually, that wasn't true, what that police man said to you. You can file a restraining order based on what you have written here. The threats, walking into your house uninvited, attempting to assault both you and your brother inlaw.
Those things are called harrassment and it is illegal. Go to the court house, fill out the paper and file the papers. The 'worst' a judge will say is no. But a judge will more than likely agree with it.
Call the police every single time he shows up. You have asked him not to, so make sure to make a trail of evidence to his harrassment (whether you do the restraining order or not).
There is no way any family court judge would let someone who is violent and does drugs take a child for visitation. But it will be up to you to make sure all this is documented!
Good luck.