I let my 5 year old son go live with his father 17 years ago. It was the worst thing I've ever done in my life.
If he was abusive towards you, he is bound to be abusive towards the "new gf" and possibly your daughter.
Get yourself and your daughter involved in activities outside...take her to a park... meet new people there...go to the library~they always have kids groups going on and you may make a new Mom friend too!
My son is a wonderful man today. But, our lives were marked with a lot of B.S. from my ex and his now ex wife.
Take hold of your little one and keep her close!