I'm absolutely positive that your previous life experiences don't have much to do with your anxiety. There're other mental conditions which correlate with negative memories of the past, but not anxiety.
Anxiety i'm sure has a physiological basis to it. During a panic attack, massive amount of adrenaline is released into your system, that's why you are willing to scream and run for your life when it happens. But, seeing how there's no immediate danger, that freaks you out even more.
But, to stay on the subject, I grew up in a family where there non stop arguments about the most menial things. I hanged out with my friends a lot, did well at school, knew many people around the town, and didn't have to work because my parents were rich and provided me with everything. I learned what depression/anxiety/depersonalization is after a smoked pot. This condition is to stay with me forever, though it hardly ever manifests itself now. Still, every now and then I get my "healthy dosage of fun".