Divorce, Children and Ex Harrassment

We have been legally separated for almost 10 months now. We have an 8 and 9 years old. My ex was emotionally abusive to me. He was not much of an interactive father but he was good to the kids. We have a parenting plan in place but he uses that time to play games with me on scheduling and not picking kids up on time or being late at drop off. He also uses that time to leave me multiple voice mails about the past (venting) also saying bad things to me about me. He sends me pics of girls he is dating, etc... I don't care about any of that. I just want him to stick with the schedule with the kids and leave me out of it. He says he can't be a father unless he has contact with me. We go around and around about this. I end up blocking his phone, it is very peaceful. Then he want to start seeing kids again, I unblock his phone to communicate about kids, and the abuse starts again through voice mail and text. I block his phone and we keep doing this over and over again. He wants to go to my son's b-day party and I told him maybe. Then the pick up and drop off went soar again this weekend. So I told him no, I don't feel comfortable with you there. Now he is AGAIN threatening to walk out of the kid's lives. He does have my son on Wed, which is his actually b-day. He can do something fun with kids instead of him always wanting to be in my plans, because he doesn't know how to be a dad. He did it through me for 9 years.

My question is: My son wants dad to go to b-day, but I struggle because I don't want to take a chance of him doing or saying something to upset me in front of family and friends. Should I take the chance for him to be there for my son?