Pot addiction is probably why you have such anxiety issues. Most people think it just calms you down, but it can actually do the opposite, especially in certain people and when you become addicted to it.
People think pot is a safe and harmless drug that is not addictive, but that's not true. For some people, it can be very addictive. It may be too difficult to do this on your own.
You said you are in recovery from other drugs? Did you go through a program? This is probably your best bet when it comes to this addiction, especially because a part of you doesn't want to quit, mostly because you seem scared. But you don't have to do this by yourself. There are other people going through the exact same thing you are and there are doctors that specialize in addiction medicine and recovery.
I would look into working a program, whether it is 12 step or whatever is available. Find what will work for you. It can be inpatient or outpatient. Do you go to meetings? If so, there are meetings for drug addiction and sometimes even specifically for pot addiction. This will help your long-term sobriety. Get a sponsor who has been through what you are going through. You can lean on that person, talk to them day or night when you are feeling an urge. Wouldn't that be nice rather than having to go it alone?
Do some research and find a program or group in your area and start going and talking to people.
It's not going to be easy, but with a program and support group, you can make it easier on yourself. You aren't alone--just remember that.