Well, are you wanting to gain weight or muscle mass? If you're wanting to gain weight, then you don't need to be exercising your behind off. What you should try is eating ice cream or cookies before you go to bed. Hell, have a big glass of whole milk with it, too. You just have a high metabolism and you need to shock your body.
I dated a guy that was about your size and he couldn't pack on the pounds no matter how hard he tried. But, after we broke up, he was walking everywhere and was going to the gym and drinking these protein shakes and stuff and I noticed that he was actually getting a little bit bigger. Sometimes it works, sometimes it doesn't.
Regardless of what works or not, you should feel good about yourself whether you're big or small. Definitely work on your self-esteem and love yourself for who you are. I wish you all the best! :-)