Random crashes when joining meetings

Hi,

we recently updated to the latest Jitsi version [1] (on our Debian 10 server and noticed random crashes when joining meetings. When this occurs, people get no authentication request after the pre-join screen and just “hang” between the meeting and the pre-join screen without joining the meeting at all. The original meeting crashes and every participants is left in their own room.

We could not find a fix for this error and were forced to do a rollback to the version before that, which still runs stable.

[1] jitsi-videobridge2 2.1-492, jitsi-meet-web-config 1.0.4985, jitsi-meet-web 1.0.4985

Jicofo 2021-05-27 14:00:04.277 SEVERE: [27] org.jivesoftware.smack.AbstractXMPPConnection$6.run: Exception in packet listener
java.lang.NullPointerException
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.getLocalNickname(ChatRoomImpl.java:170)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.isFocusMember(JitsiMeetConferenceImpl.java:1001)
        at org.jitsi.jicofo.JitsiMeetConferenceImpl.onMemberJoined(JitsiMeetConferenceImpl.java:564)
        at org.jitsi.jicofo.ChatRoomRoleAndPresence.memberPresenceChanged(ChatRoomRoleAndPresence.java:144)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.lambda$fireMemberPresenceEvent$1(ChatRoomImpl.java:535)
        at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.fireMemberPresenceEvent(ChatRoomImpl.java:535)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.processOtherPresence(ChatRoomImpl.java:794)
        at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.processPresence(ChatRoomImpl.java:843)
        at org.jivesoftware.smackx.muc.MultiUserChat$3.processStanza(MultiUserChat.java:251)
        at org.jivesoftware.smack.AbstractXMPPConnection$6.run(AbstractXMPPConnection.java:1263)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Anyone else with this problem? Is there a fix for this?

Hi Wurzelmann,

Had the same issue right after the last update on the server.
The be precise, the room started to heavily crash after approx 20-30 people joined.

I red a few posts on the forum and understood that last prosody version (0.11.2-1+deb10u2) might be the problem.
So I downgraded it to the previous one (0.11.2-1).

I then tried to test with as many people as I could on that day (18 people) and it worked.

Could you describe which packages you downgraded to which version ?
That could help other people as well.

Best

We’re using ejabberd, not prosody, so this should not concern us, to be honest…

We simply rolled back all packages from the last Jitsi version via the repository, these are the ones currently installed (after the rollback):

jitsi-videobridge2_2.1-478-gc6da57bd-1_all.deb
jicofo_1.0-740-1_all.deb
jitsi-meet-web_1.0.4900-1_all.deb
prosody_0.11.2-1+deb10u2_amd64.deb
jitsi-meet_2.0.5765-1_all.deb
jitsi-meet-web-config_1.0.4900-1_all.deb
jitsi-meet-prosody_1.0.4900-1_all.deb

What is your java version?

java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-post-Debian-1deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.11+9-post-Debian-1deb10u1, mixed mode, sharing)

We found the root cause of this issue: in /etc/jitsi/jicofo/jicofo.conf there is a setting that tells jicofo how long to wait for the first participant: initial-timeout.

Setting initial-timeout to a much higher value (6 hours instead of 15 seconds in our case as we use our own reservation system) fixes the issue we had with vanishing conferences. The setting itself has been there for quite some time but was seemingly not honoured before.

It seems the stack traces we encountered in the logs are “normal” and not responsible for the crashes at all, which suggests erroneous catching of exceptions and logging IMO.

Anyway, I hope someone can use this information to fix their Jitsi instances too.

/etc/jitsi/jicofo/jicofo.conf:

jicofo {
  // ...
  conference {
    // How long to wait for the initial participant in a conference.
    // wait for 6 hours for the first participant
    //initial-timeout = 15 seconds
    initial-timeout = 6 hours
  }
}
2 Likes

I don’t think this is a fix, but rather a workaround for your particular case. That timeout was always like that and we had never seen problems for 5 years now.
What does crashing mean? What is the exception? Is it the NPE from the first post? Do you reproduce the problem with the latest jicofo?

As pointed out in the first post, every participant ends up in their own little room, with no one else and the original meeting is not usable and produces a NPE like in my post. This was tested with the latest jicofo and the only way we could our Jitsi instance to work again was downgrading all components until we found this workaround.

What is the output for

netstat -taunp | grep 5222

So this is not using jitsi-meet and lib-jitsi-meet?

We wrote a meeting manager that basically schedules and manages meetings via xmpp. It creates a new muc, invites the focus user and, after focus joined, sets a password on the room. For this very reason the focus user is unable to join the same meeting again as we did not find any way to send the password to the focus user before joining the meeting (eg. XEP-0249).

Our problem was that we got distracted by the uncaught exception in the log that is completely unrelated to our issue. Up until the last release, focus was happy when there was any other user in the meeting (our bot).
Something seems to have changed in jicofo that only counts users with A/V as participants, for this reason, focus terminated the meeting after the default timeout of 15 seconds.

We’d love more documentation on the xmpp interaction in order to be able to use other cool jitsi features like waiting rooms from within our bot or a pointer on how we might add an authentication button to the web interface to make a user administrator from within a meeting and the like… :wink:

The password thingy you can control from a custom prosody module. You don’t need a participant and creating a room in advance.
The flow is that jicofo is admin of the rooms, it is always the one creating the rooms and joins first.
All clients first send an IQ to jicofo for a room and if they receive a successful replay they join the room. And the replay is after jicofo creates and joins the room.
You can check the reservation module, I think it already does that, on an IQ to jicofo checks for data, in your case is there a password for this room. And when the room is created and you can wait for jicofo to join you can set the password.
There are multiple ways to solve your problem with custom prosody module and for most of that there are already examples in the code …