You are definitely not alone in this. I have had driving anxiety for almost 5 years and it hasn't lightened up much since the beginning. I ask myself the same things you mentioned and I always have a minor panic/anxiety attack.
Lately, I have been doing everything in my power to work with this problem. I am a wife and a mother, and not driving is a real block in my everyday life. I find myself jealous of those who can drive and it is very upsetting.
The one thing I can suggest is just practice, and tell yourself that you can do it. That's what everyone told me to do and I never believed it, but it has been working so far. Slowly but surely, I am becoming more aware of my ability to drive.
Try pinpointing what bothers you the most and start there. Sit down and make a list of what stops you from driving anxiety-free. Evaluate those points and find a solution for each. By seeing the specific issues on paper, you will have a better chance of finding a solution. Have an experienced driver do this with you, or evaluate what you have written to give you some pointers. You will have solutions to each of your weak points in no time.
For example, if you feel that your depth perception is shaky, you could have someone ride with you when you drive and tell you in minute intervals that you are doing a good job of staying in your lane. My husband does that for me and it makes a tremendous difference.
If you forget some of the "common" rules of the road, like what to do at a 4 way stop, or how/when to turn at a red light, you can look them up, put them on a sticky note, and put them within view from behind the wheel (on your wheel, visor, etc.) for a quick reminder.
If you need any more help, feel free towrire me and we can work through this together and hold each other accountable. I am fixing my problem, and I know you can too!
With Sincerity,
Nicole