Easy way to create a Jitsi cluster

Hello,

I made some tools, installer and documentation to create a Jitsi cluster in an easy way. I really mean “easy”. For detailed information, please see the link below:

Easy way to create a Jitsi cluster based on Debian Buster (Debian 10)

These are the steps:

  • Jitsi Meet Server (JMS) installation

    • download the installer
    • set the hostname
    • run the installer
    • get a Let’s Encrypt certificate
  • Adding an additional Jitsi Videobridge (JVB) node to the cluster

    • copy the JMS’s public key to the node (newly installed Debian machine)
    • run the add-jvb-node command on JMS
  • Adding an additional Jibri node to the cluster

    • copy the JMS’s public key to the node (newly installed Debian machine)
    • run the add-jibri-node command on JMS

Please send me feedback if you have an issue

1 Like

@emrah

Thanks for this, we have ran the script and come up with the following errors;

This is right at the end of the install process, I ran the script twice, the first time as a sudo user and got this error at the set-letsencrypt-cert meet.mydomain.com stage, the 2nd time I ran as root and got the same error at the end of the install script.

We are running a Azure VM with 5.5.0-0.bpo.2-amd64 #1 SMP Debian 5.5.17-1~bpo10+1 (2020-04-23) x86_64 GNU/Linux

lxc-attach: eb-jitsi: attach.c: lxc_attach: 1095 Failed to get init pid

Something went wrong. The installation couldn’t be completed!

Thanks

Can you paste the output of the following commands?

lxc-ls -f
lxc-start -n eb-jitsi
lxc-attach -n eb-jitsi -- bash -c 'ls'

ip a

NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
eb-buster STOPPED 0 - - - false
eb-jibri-template RUNNING 1 eb-group, eb-jibri 172.22.22.223 - false
eb-jitsi STOPPED 1 eb-group, onboot - - false

No output

bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0d:3a:d9:80:ec brd ff:ff:ff:ff:ff:ff
inet 10.0.2.4/24 brd 10.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20d:3aff:fed9:80ec/64 scope link
valid_lft forever preferred_lft forever
3: eb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether fe:28:e0:b2:5a:db brd ff:ff:ff:ff:ff:ff
inet 172.22.22.1/24 brd 172.22.22.255 scope global eb0
valid_lft forever preferred_lft forever
inet6 fe80::cf1:3eff:febf:222a/64 scope link
valid_lft forever preferred_lft forever
7: vethJ7E07N@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master eb0 state UP group default qlen 1000
link/ether fe:28:e0:b2:5a:db brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::fc28:e0ff:feb2:5adb/64 scope link
valid_lft forever preferred_lft forever
9: veth3SSI1K@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master eb0 state UP group default qlen 1000
link/ether fe:60:20:0a:f4:ab brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::fc60:20ff:fe0a:f4ab/64 scope link
valid_lft forever preferred_lft forever

Hello @Greenzom,

I sent you a private message. I can connect to the server and check it if it is OK for you.

Hello @Greenzom,

I deleted the containers and I started the installer again.

To delete the containers:

lxc-stop -n eb-jitsi
lxc-stop -n eb-jibri-template
lxc-destroy -n eb-jitsi
lxc-destroy -n eb-jibri-template

It’s working now

I changed the structure a bit. Now, the cluster is using two different host addresses for jitsi and coturn.

For example, meet.mydomain.com for JMS and turn.mydomain.com for coturn

Easy way to create a Jitsi cluster

Hello @emrah!
Thanks very much for the script, it helps a lot!
I had installed the core and 4 jvb node(in differents VM) for my jitsi intance, and I wanted to know how can I know if the nodes are indeed functional.

Last day I have checked that CPU of the 4 JVB VMs are not increasing, so all conferences were using only the jvb of the “core” jitsi. I enter in these VMs and saw the log with some timeout to access the internet domain of my jitsi instace. I have resolved that, and the logs are now getting this output, with 10 seconds interval:

2020-09-09 16:03:42.528 INFO: [29] Videobridge.createConference#255: create_conf, id=e7f7d958a7e7980e gid=-1 logging=false
2020-09-09 16:03:42.537 INFO: [29] AbstractHealthCheckService.run#182: Performed a successful health check in PT0.00925S. Sticky failure

I was thinking that it was functional, but when I tried to create conference, with some time of running it goes to the “Unfortunately something went wrong” message, so I had to stop the jitsi-videbridge2 service in all nodes to jitsi get back to work.

I had used the add_jvb_node command in the core to do the install, and it executed without errors.

Hello @Roberto_Moura,

I check the additional JVB by stopping the JVB which is on the JMS server.

On JMS

lxc-attach -n eb-jitsi
systemctl stop jitsi-videobridge2.service

The conferences work fine in my tries. Check this at least with 3 clients in a room. When there are only 2 clients, a P2P connection is established and no traffic on the JVBs.

To restart the JVB again on JMS

systemctl start jitsi-videobridge2.service

Let me know if there is a problem when you stop the main JVB

1 Like

I activated colibri for the new installation too. So you can get the conference list using the following command on JVB

lxc-attach -n eb-jvb
curl http://127.0.0.1:8080/colibri/conferences
1 Like

jibri is now configured using the new jibri.conf file.

1 Like

Thanks for answers, @emrah.

The environment is in production, so I had no time yet to disable the JVB on JMS. The conferences are running in this only JVB, but I’m getting some issues.

First one is that some participants that enter the conference some time after the begin of the conference is getting no audio at all of the others participants, but I think that is something about the moodle plugin that I’m using, because this problem is not ocurring directly on the JMS website.

Second one is less frequent, but it is critical, the conference stop and the following error appears:

Jicofo 2020-09-11 14:00:06.265 SEVERE: [4019] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant – no bridge available.

Do you know why it happens?

And other thing: How can I get this updates you are making? In my installation I don’t have colibri.

Do you have enough resources (CPU & RAM) and could you check videobridge log
/var/log/jitsi/jvb.log

There are two changes to activate colibri (in the container eb-jitsi and eb-jvb)

  • /etc/jitsi/videobridge/config
JVB_OPTS="--apis=rest"
  • /etc/jitsi/videobridge/sip-communicator.properties
org.jitsi.videobridge.rest.private.jetty.port=8080
org.jitsi.videobridge.rest.private.jetty.host=0.0.0.0

Needed to restart the videobridge

systemctl restart jitsi-videobridge2.service
1 Like

Yes, the VM have 110GB RAM and 24vCPU, but if it happens again I will check the jvb.log too. Today there are already some conferences running. Thanks for prompt response!

Can it be the sip-communicator-properties? In the main jvb I disabled TCP harvester, but as I confirmed this configuration is not in the jvb nodes. Can other tunnings made in JMS impact on this?
Today I had some “SCTP send errors” too, I think that is because the jvb nodes is only with the service stopped, but it is somehow “connected” to JMS, does it make sense?

TCP harvester is needed on JMS to coturn works correctly.

And your JMS and JVBs are not in the same local network, right?

1 Like

Hmm… I had disabled it for performance tunning, and it was the best configuration to reduce cpu load… But it was in time I didn’t use this container solution yet… So I need tu turn this on again?

My JMS have a public exit, but it communicates with the JVBs within the same local network. I said in previous post that I needed to add a new entrance in /etc/hosts to JVB use local network to communicate with the JMS.

I tried to stop the JVB on JMS and let run only one of the JVB nodes, but it have the same behavior when I have the JVB on JMS running: The conference starts, I can see the participants getting inside, but after a while all of them get disconnected.

In jvb.log at JVB node:

2020-09-14 20:59:11.077 INFO: [100] [confId=e33dd1d05777079f epId=857deccf gid=161652 stats_id=Nickolas-7R8 conf_name=teste] AbstractEndpoint.expire#233: Expiring.
2020-09-14 20:59:11.079 INFO: [46] [confId=e33dd1d05777079f gid=161652 conf_name=teste] Conference.dominantSpeakerChanged#464: ds_change ds_id=e33dd1d05777079f
2020-09-14 20:59:11.080 INFO: [100] [confId=e33dd1d05777079f epId=857deccf gid=161652 stats_id=Nickolas-7R8 conf_name=teste] Transceiver.teardown#319: Tearing down
2020-09-14 20:59:11.080 INFO: [100] [confId=e33dd1d05777079f epId=857deccf gid=161652 stats_id=Nickolas-7R8 conf_name=teste] RtpReceiverImpl.tearDown#296: Tearing down
2020-09-14 20:59:11.081 INFO: [100] [confId=e33dd1d05777079f epId=857deccf gid=161652 stats_id=Nickolas-7R8 conf_name=teste] RtpSenderImpl.tearDown#268: Tearing down
2020-09-14 20:59:11.082 INFO: [100] [confId=e33dd1d05777079f epId=857deccf gid=161652 stats_id=Nickolas-7R8 conf_name=teste] DtlsTransport.stop#184: Stopping
2020-09-14 20:59:11.082 INFO: [100] [confId=e33dd1d05777079f epId=857deccf local_ufrag=b06r1ei7gsop2 gid=161652 stats_id=Nickolas-7R8 conf_name=teste] IceTransport.stop#235: Stopping
2020-09-14 20:59:11.084 INFO: [100] [confId=e33dd1d05777079f gid=161652 stats_id=Nickolas-7R8 conf_name=teste ufrag=b06r1ei7gsop2 epId=857deccf local_ufrag=b06r1ei7gsop2] Agent.setState#963: ICE state changed from Running to Terminated.
2020-09-14 20:59:11.084 INFO: [100] [confId=e33dd1d05777079f epId=857deccf local_ufrag=b06r1ei7gsop2 gid=161652 stats_id=Nickolas-7R8 conf_name=teste] IceTransport.iceStateChanged#321: ICE state changed old=Running new=Terminated
2020-09-14 20:59:11.085 INFO: [100] [confId=e33dd1d05777079f gid=161652 stats_id=Nickolas-7R8 componentId=1 conf_name=teste ufrag=b06r1ei7gsop2 name=stream-857deccf epId=857deccf local_ufrag=b06r1ei7gsop2] MergingDatagramSocket.close#142: Closing.
2020-09-14 20:59:11.086 INFO: [100] [confId=e33dd1d05777079f epId=857deccf gid=161652 stats_id=Nickolas-7R8 conf_name=teste] Endpoint.expire#778: Expired.
2020-09-14 20:59:13.466 INFO: [27] VideobridgeExpireThread.expire#144: Running expire()
2020-09-14 20:59:13.466 INFO: [27] VideobridgeExpireThread.expire#150: Conference e33dd1d05777079f should expire, expiring it
2020-09-14 20:59:13.467 INFO: [94] [confId=e33dd1d05777079f gid=161652 conf_name=teste] Conference.expire#541: Expiring.
2020-09-14 20:59:13.467 INFO: [94] [confId=e33dd1d05777079f gid=161652 conf_name=teste] Conference.updateStatisticsOnExpire#620: expire_conf,duration=35,has_failed=true,has_partially_failed=false
2020-09-14 20:59:13.831 INFO: [29] Videobridge.createConference#255: create_conf, id=64a0daade97784a3 gid=-1 logging=false
2020-09-14 20:59:13.846 INFO: [29] AbstractHealthCheckService.run#182: Performed a successful health check in PT0.015162S. Sticky failure: false

In jvb.log at JMS, in the moment I start jvb on node, it get stuck at these erros:

2020-09-14 20:51:27.926 WARNING: [36] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener: Connection XMPPTCPConnection[jvb@auth.my-domain/E7N-gZ_o] (0) closed with error
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
stream:errorReceived SIGTERM</stream:error>
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-09-14 20:51:27.941 WARNING: [36] [hostname=localhost id=shard] MucClient$1.connectionClosedOnError#314: Closed on error:
org.jivesoftware.smack.XMPPException$StreamErrorException: system-shutdown You can read more about the meaning of this stream error at http://xmpp.org/rfcs/rfc6120.html#streams-error-conditions
stream:errorReceived SIGTERM</stream:error>
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1064)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:1000)
at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:1016)
at java.base/java.lang.Thread.run(Thread.java:834)
SCTP JNI load: Linux OS detected
SCTP lib loaded
=====>: org_jitsi_modified_sctp4j_SctpJni.c calling init
=====>: org_jitsi_modified_sctp4j_SctpJni.c about to set SCTP_DEBUG_ALL
2020-09-14 20:51:35.217 SEVERE: [30] AbstractHealthCheckService.run#185: Health check failed in PT4.316294S:
java.lang.Exception: Performing a health check took too long: PT4.316294S
at org.jitsi.health.AbstractHealthCheckService.run(AbstractHealthCheckService.kt:170)
at org.jitsi.utils.concurrent.RecurringRunnableExecutor.run(RecurringRunnableExecutor.java:216)
at org.jitsi.utils.concurrent.RecurringRunnableExecutor.runInThread(RecurringRunnableExecutor.java:292)
at org.jitsi.utils.concurrent.RecurringRunnableExecutor.access$000(RecurringRunnableExecutor.java:36)
at org.jitsi.utils.concurrent.RecurringRunnableExecutor$1.run(RecurringRunnableExecutor.java:328)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
SCTP JNI load: Linux OS detected
SCTP lib loaded
=====>: org_jitsi_modified_sctp4j_SctpJni.c calling init
=====>: org_jitsi_modified_sctp4j_SctpJni.c about to set SCTP_DEBUG_ALL

All this stuff was after enabling again TCP harverster and reset all services on JMS.

I will let only the JVB on JMS running until get things working. Thanks for the attention!

Do JMS and JVBs use the same external IP?

1 Like

No, only the JMS has public IP, the JVBs have only local address (192.168.120/24). Do JVBs need to have public address?? Didn’t see it in nowhere.