Jitsi meet won't start at boot


#1

I’d been using jitsi meet successfully for a couple of years. Recently, I started having a problem where it would fail to start at boot. The logged messages would be wildly inconsistent from boot to boot (and, very occasionally, the program would start and run properly until the next restart). I tried the dev’s suggestions, I wiped all the jitsi software and re-installed, but the symptoms remained consistently inconsistent (and the program remained unusable). Finally, I provisioned a brand-new server box with a clean install of Devuan Ascii and installed jitsi meet using apt and the quick install instructions. It worked fine until an update about a month ago, and I was back to not being able to get the program to launch no matter what I did. The log messages were at least consistent this time, if completely unhelpful (stack traces).

After many, many hours of fiddling I discovered that there appears to be some sort of race condition between components (prosody, jicofo, and jitsi-videobridge). I now have an init.d script which will consistently work, but it’s ugly:

/etc/init.d/prosody start
/etc/init.d/jicofo start
/etc/init.d/jitsi-videobridge start
sleep 25
/etc/init.d/prosody restart
sleep 5
/etc/init.d/jicofo restart
sleep 5
/etc/init.d/jitsi-videobridge restart

Ugh. Obviously, there’s a lot of empiricism here and the sleep timing (1) has not been rigorously optimized and (2) might be different on different systems. All I know is it has worked here for 20 consecutive restarts, and having totally blown my time budget for several projects I simply have to put it into production without fiddling more. I wanted to get this posted so, until there’s some explanation or solutions, others having the same issues will at least have access to a workaround.

Here’s what I know for sure:

  • You have to start at least one of the components (I have only tested all three) and then restart again to get things to work.
  • There must be some delay (5 seconds wasn’t enough, 25 is probably overkill given how consistently it’s working now) between the first series of starts and the restarts.
  • There must be some delay between re-starting prosody, re-starting jicofo, and re-starting jitsi-videobridge or it will fail. Two seconds was not sufficient, but five was. I don’t know whether both delays are needed.

I don’t know if this is a race condition that is somehow covered up by systemd (Devuan is a Debian fork without systemd) or whether this would happen on distros that use systemd; I don’t have time to test right now. But I assume this isn’t breaking on all distros or there would be some noise about it here.

I hope this is helpful to others, and I’d appreciate any thoughts on a cleaner workaround than this one.

–2p


#2

The order you are starting the components is not correct.
You can try:

/etc/init.d/prosody start
/etc/init.d/jitsi-videobridge start
sleep 5
/etc/init.d/jicofo start

This should work. The thing is that by default jicofo discovers jitsi-videobridge component and if it doesn’t find one will retry after 30 or 60 seconds. I think if you leave the default startup order (basically no order) it should work after boot, but you need to wait at least a minute.

There is another option if you connect your jvb, not as a component but using a xmpp muc, then this re-discovery is not needed and jvb are available at the moment they connect.

Thanks for the report, I’m currently looking in some systemd changes for the videobridge, and will try taking a look at the startup order, those changes will not affect you but we can try addressing that problem.


#3

Ahhh. I was following the start order recommended in a comment to the quick start guide:

However, I can assure you that even waiting longer than 24 hours didn’t result in the program starting properly without doing the start-then-restart sequence I posted.

The server is currently in transit from the lab to the office data center. I’ll try playing with the load order once it’s back up (likely Wednesday).

Thanks!

–2p


#4

In the quick install guide there is nothing mentioned for restarting order. Generally, there should be no order, you just need to wait a bit. If that is not true, there can be a bug. For example, I had seen trying to start any of the services before the network is up was resulting unusable system.

If you can, can you leave the default order, do a restart, wait 5 minutes, try a conference and send the jicofo and jvb logs and one of the clients js console logs, you can send it to me personally if you want.


#5

Correct. I was referencing one of the comments to the guide.

Will do, but it will have to wait until Wednesday after the server is re-installed on site. There won’t be a client log, as the server never becomes reachable when using the default load order.

Thanks for your help!

–2p


#6

Ok, whenever you can.

From the description makes me think you are running with jetty and jvb fails to start at all, it seems to me it is maybe the same issue where it starts to load before there is a network or something like that, but the logs will say more.


#7

I captured logs from a failed start. I can send them to you, but I don’t have your address. Let me know where you would like me to send them; I’m ron /at/ risley /dot/ net.

Since this is a clean install of the OS and Jitsi Meet on fresh new server hardware, and the symptoms are the same as I was having before on the old hardware, and nobody else seems to be having the problem, I think the issue must be related to my choice of Linux distro (Devuan, a systemd-free fork of Debian). We’re going to have to abandon the project altogether if we don’t have something working reliably by Monday, so I’m going to wipe the box and install something mainstream like Ubuntu server and hope that fixes it.

If it turns out to be distro-related (and not just something idiotic I’ve done and then forgotten about), I’ll bring up a testing system in the lab and we can troubleshoot further if you’d like.


#8

You can send it to me at damencho at jitsi dot org.
I’m testing regularly installations on Debian and Ubuntu / latest LTS and haven’t seen any problems. The only thing is with ubuntu missing the universe channel by default, but this is described in the quick install docs.