I had similar issues with my Handy. Two things saved it for me, and even elevated it beyond any other toy I’ve tried - including the SR6 and Syncbot.
First is the Sunshine 3d printed sleeve holder.
Improved Hold for Handy Sleeves - Sunshine Review - DIY - EroScripts
Second is having a high end microphone arm attached to my desk to hold my Handy.
Live Streamer ARM - BA311 | AVerMedia