Kicked out of Voice call

How do I not get kicked out from my own jitsi phone voice conference. I find it a big issue that anyone can kick me out of my own phone conference. Is there a way to solve it ?