Yeah, honey it is tough. See, I would be exactly like you are so it is hard for me to give you advice! I know telling you to keep him out is absolutely wrong, but I understand where you are coming from
but, let me put it this way. There are so very many girls/women out there who get pregnant and the guy runs off. Or he beats on her, kicks her in the stomach trying to make her lose the baby, or he runs off as soon as she has the kid, or he fights for custody and makes her pay child support. There are just so many scenarios.
Try to be happy that he wants to be there! Even wanting to go to the doctor's appointments, that is virtually unheard of. Take pride in what you have.
He can be a part of you and your baby's life without you having to be "with" him. Maybe just be friends through the pregnancy? Like, don't see him everyday, all the time so that fights break out, but let him go to the doctor with you and everything. That may be best.
I do understand what you mean though, and it doesnt' make you a bad person or anything.