Hi Sebastian,
I can relate quite well with the anxiety you're experiencing. Long before taking stimulants, I was riddled with anxiety as a kid - even exhibiting Obsessive Compulsive tendencies (based heavily in anxiety). I'm not one to push meds, but if the anxiety seems irrational and oppressive like it has been with me, an SSRI antidepressant may be helpful. I've been on many antidepressants in my life, but recently I got to rediscover my "natural" state of being without any meds. My anxiety came back like when I was a kid. Anyways, I started taking Lexapro (an SSRI I tried about 10 years ago that worked well for me). That seemed to help and I stopped having the melt-downs I was having prior to it.
Let me clarify that SSRI antidepressants are administered for depression as well as anxiety - which is why I wanted to bring it up despite your aversion. I feel I can relate a lot to how you're feeling and it might be something to consider. In my experience, if you do consider taking an antidepressant, be very wary of doctors pushing new meds. They're are plenty of SSRIs that have been around for years, thoroughly researched, and far cheaper than new meds seeking profit. Also, STAY AWAY FROM BENZOs (Klonopin, Zanax, Valium, etc). I know from personal experience that Benzos are a nasty med to get involved with.
I wish you the best!