I am unable to remove the moderator rights from participant panel
i had removed from remote pane but not from above one
Just make sure the participant is not moderator at the serverside.
I had similar issues, you can use this css tweak:
a[id^=grantmoderator]
{
display:none;
}