Like you I started an a very anxious driver (20+ years ago now). When I first got my licence I told my instructor: 'thanks, but I am never going to drive again!', as I was so exhausted from the anxiety.
However, like you I slowly expanded my comfort zone, firstly I would get up in the middle of the night, when all was quiet, and take little trips on my own, so that I could learn the driving route without the pressure of congestion. I found it nerve wracking but in this way, I built my confidence to drive to work (I lived close to inner city and had to drive in peak hour in the worst of it).
The same method I used to expand my driving further in baby steps over several years, until I was fairly comfortable driving city, rural, interstate, I have even driven in England (but that was very stressful as busy + unfamiliar + hire car). Basically manage the size of the task, and break it into achievable steps. Plan a small deviation from your work routine to visit a new shop or place to eat once a week. Or take small drives to new areas at quiet times and allow yourself to pull over and take a breather if you feel overwhelmed. .
Don't stay in your comfort zone, set yourself a goal, and then just set smaller achievable steps that build your confidence gradually until you can reach that goal. Then set a new one. You will learn best by just slowly working on the edges of your comfort zone. Good Luck