Bipolar boyfriend maybe draining me

I have been in a relationship with a guy who is Bipolar/Manic/Schizophrenia. He told me in the beginning of the relationship that he has issues, but I don't see him acting out or being that type of person. When I first meet him we were together for 6 months something I said to him got him upset and we broke apart he didn't want to deal with me a year passed and the following year I decided that I wanted to be back with him. It was enough to realize what type of person I was dealing with. We finally were able to make it threw a year. The past few month he would come over often and spend time with me just hanging out. What I really have noticed about him is that he is so paranoid like he doesn't want to be touch, even though I men well and its away of showing my affection I have to touch him. This makes him really upset and he says stuff like he can hurt me and that he want's to do mean things to me. Even when I give him a light touch he hits me back. I'm always asking my self why the hell does he do that and I'm realizing it has to be because of his Bipolar, lately he doesn't want to come over and has been staying home. When I ask him about coming over he tells me yes he well, or will say he doesn't know I just feel that he's going threw one of his phases and I need to fall back, it's so hard to do that when you love someone so much. I feel like I need him around and he just rather stay home.