I'm so sorry to hear what you're going through. I read your post as it is similar to the situation I am in. Our situation has been going on for some 12 years though and while it has improved, it's not perfect and I do struggle because of decisions I made.
My husband also suffers anxiety and in the early days panic attacks. It would come quickly and we'd have to leave the event we were at. It got to the point where he couldn't go to public events or large gatherings. Christmas was with 2-4 people and in 2 sittings at our house. To support my husband I wouldn't go to the functions either and so we'd always have to decline invites. It got to the point where we were getting fewer and fewer invites and in the end none, now it's literally just immediate family.
I can tell you I've missed so many milestones, family functions and events that I'd usually go to and have fun at. He was never really a go to family functions type of guy so it wasn't a big loss for him. When you say you have a birthday trip coming up, it's great that he is not saying to cancel and don't go. But I can see your stress knowing what is coming. I could see this too and so I did cancel events or not suggest trips that would pressure him. The problem is fast forward those years to now and I have friends I can count on one hand (and don't see very often) and am only close to my immediate family. I've not seen my extended family in over 8 years. I was close to them all now they assume I've just snobbed them.
I guess what I am saying is, please make sure you don't stop your life for them. I know it's hard and I felt so guilty even thinking about going to events and places which is why I didn't. I thought I was supporting him. But now that he is a little better and can manage going to events and places, we only go when he wants to. I don't go to events or dinner with friends on my own. I now suffer depression because of the life we lead, but he can't see why. Like you, when I say anything to do with his anxiety I get tared with the worst person tag and I'm just bullying him again and I'm not supportive. The guilt trip cycles continue.
I wish I was stronger to stand up for myself earlier so I didn't end up alone with no real support close by. I just felt guilty and thought I was being supportive, but after reading the forums, I realise now (10 yrs too late) that I needed to keep myself alive and not enable his anxiety.
It's tough and we're here for you.