Ohh. I'm sorry you have to go through this. How old are you? As long you aren't so young that you and your 'brother' would have to live together as siblings, it's really not a big deal. It may be frustrating to explain for the rest of your lives (if you plan on getting married or whatever), but that's life.
Me and my husband are different races and people have asked me if our children are mine when I am alone with them in public. I get angry sometimes, b/c people can be so ignorant, but that's life. I will have to deal with this forever I suppose. Unless, my girls start to look more like me one day *praying!!*.
I'm not sure what exactly you're upset about, but if it's what other people think...You've got a life time of other's opinions to deal with. Do what's right and what makes you happy. I hope it works out for you.