Zoloft like the other ssris have a potential for lowering sex drive. I suppose it's a matter of choice... If you're getting significant relief (enough that you don't mind losing the sex drive) than you have to pick and choose..
Honestly, I don't want to speak for everyone, but I have my suspicions about the real nature of anti-depression drugs and their long-term effectiveness. It seems to me most people with anxiety have a significant event or series of events that occured in childhood which essentially led to their current fearful perception of reality. The anxieties go and and hand with their most essential thought process; which is why therapy is the best route to fix the problem. Medication is some cases however, could be helpful in the short term to at least calm symptoms enough to embark upon therapy.