I agree with you all the way, I wouldn't want my child around a person like that & to me it sounds like he's only pissed because you're pregnant.
Do what you feel is right, it's your baby and it's your job to make the right choices for it, having your baby around a person like that will not only hurt it emotionally, but growing up around that type of person that hits and talks nasty language that like in front of your baby will do nothing but make your child think thats okay to do, and will do it when it's older.
I have a potty mouth i'll admit, but I haven't and will never say cusswords in front of my son, nor will I ever let him see me angry or upset.