Tell him that.
Tell him how you feel, that you do not want the baby around the people he hangs out with because they are bad influences.
If he wants to be in the babies life, thats great. But hearing from a friend he's sober could just be a lie to get a good word in for him.
He as to prove to you that he is sober and can be a good father.
If you dont want your baby living in the environment that he is in right now, tell him. and talk to your mom about that too. I would not want my baby living in a dead beat environment either and thats a good call for you as a mother.
But you have to be honest with him, or else things wont change.
Otherwise, if you think he wont change, tell him he can come see the baby-when you and your mom are home- when he wants to, and can show his friends pictures, but he is to come alone and can spend a few hours with you and the baby so he can bond with his child, if he wont move out of that place.
hope that was a little helpful.