Can I speak to a single person in a conference?

Is there a feature to speak to a single person in a conference call? Like mute everyone else, like wise I can talk to a person so then others cannot hear me!

Muting others mean they won’t be able to speak, not that they won’t be able to hear the conversation.
If you want to have only you and another participant speaking, just mute everyone else, but they will still hear your talk.
There is no option to deafen the other participants. Maybe you are looking for something like a breakout sub-room, there is no such feature currently in Jitsi Meet.
You can achieve something similar by sending in the chat a private message the the other person a URL for another room, then leave, go there, talk and return to the main room.