Is it possible to run Jitsi in private network without public domain for exploring purpose?
Welcome to the forum.
Yes, you can. Search the forum for installation on LAN/installation in private network.
use of a private certificate is possible when the operating system allows to use one without problem (AFAIK everything but Android). If you want to do your testing with an Android phone, you will have to run though some hooplas to use a public certificate on a private domain.