Was I Wrong Or Had Lack of Confidence?

I am 25 now. When I was 17, I didn't want to go to college because of 2 reasons.
First, the colleges around my neighborhood were expensive and unfordable and if I want to go to college, I would have to go far away and dorm. I didn't dorm because I fear dorm life. I have never been anywhere before except for the city I live in and don't know anyone from another city. If I went away, I would have been lost in the world. I would feel like I am in another world. How can I survive in a place on my own like that? That time I feel that I am a stupid person because I have this problem. Whenever I see or hear someone else dorming in school or work, I would be very proud of them and wish I wasn't scared of dorm life and be like them. My parents were very upset that I didn't go to college. They thought I was a bad person but I wasn't. I just didn't know how to explain to them about my fear.
Second, I feel that school is not for me. Since 1st to 12th grade, I always have this problem. At the beginning of every schoolyear, I had a lot of stress adjusting to school, classes, and homework. I look at everyone able to adjust except for me, I feel proud of them and wish I was like them. I feel that they are all smarter than me. During the summer months, I feel that I am on top of the world and knows everything. When I go back to school, I feel like I am the worst and stupidest person in the world. I would not understand why I felt that way in the summer. I had always thought about dropping out of school because of these problems. If I went to college that way, I would feel that I am wasting my parent's money.
Now when my younger siblings graduate from college and I am the only one out of us that don't have a college degree. I begin to regret that I felt that way. I asked myself, why I was like that. 1) whenever a person goes to a new place, he/she don't know anyone and is scared. You just need to adjust to change. Life is all about change. You will face more changes in your life later on. 2) school is a challenge for everyone. It can be harder on some person, like me. All you need to do is study. If you study, you can get good grades.
I think back and feel that there was something wrong with me that I felt that way that cause me to not go to college. Right? Was I wrong, have lack of confidence, or what? Please help me.