java.lang.NullPointerException at org.jitsi.videobridge.shim.ChannelShim.<init>(ChannelShim.java:163)

I’ve got a Jitsi-meet install on Ubuntu 18.04, that runs a meet for 20 to 30 people, most of whom are not using video/audio, and we’re crashing occasionally with this error:

   2020-07-23 13:20:33.826 INFO: [26] [confId=ac4a24c9d0687762 epId=6a8855bc local_ufrag=5t39h1edtt9ohh gid=ffeb96 stats_id=Jesse-nv6 conf_name=26386] IceTransport.iceStateChanged#321: ICE state changed old=Running new=Terminated
   2020-07-23 13:20:33.826 INFO: [26] [confId=ac4a24c9d0687762 gid=ffeb96 stats_id=Jesse-nv6 componentId=1 conf_name=26386 ufrag=5t39h1edtt9ohh name=stream-6a8855bc epId=6a8855bc local_ufrag=5t39h1edtt9ohh] MergingDatagramSocket.close#142: Closing.
   2020-07-23 13:20:33.826 INFO: [26] [confId=ac4a24c9d0687762 epId=6a8855bc gid=ffeb96 stats_id=Jesse-nv6 conf_name=26386] Endpoint.expire#809: Expired.
   2020-07-23 13:20:33.828 SEVERE: [25] XmppCommon.handleIQRequest#251: Exception handling IQ request
   java.lang.NullPointerException
           at org.jitsi.videobridge.shim.ChannelShim.<init>(ChannelShim.java:163)
           at org.jitsi.videobridge.shim.ContentShim.createRtpChannel(ContentShim.java:135)
           at org.jitsi.videobridge.shim.ContentShim.getOrCreateChannelShim(ContentShim.java:276)
           at org.jitsi.videobridge.shim.VideobridgeShim.processChannels(VideobridgeShim.java:143)
           at org.jitsi.videobridge.shim.VideobridgeShim.handleColibriConferenceIQ(VideobridgeShim.java:345)
           at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:600)
           at org.jitsi.videobridge.Videobridge.handleColibriConferenceIQ(Videobridge.java:585)
           at org.jitsi.videobridge.xmpp.XmppCommon.handleIQRequest(XmppCommon.java:228)
           at org.jitsi.videobridge.xmpp.XmppCommon.handleIQInternal(XmppCommon.java:179)
           at org.jitsi.videobridge.xmpp.XmppCommon.handleIQ(XmppCommon.java:150)
           at org.jitsi.videobridge.xmpp.ClientConnectionImpl.handleIq(ClientConnectionImpl.java:108)
           at org.jitsi.xmpp.mucclient.IQListener.handleIq(IQListener.java:50)
           at org.jitsi.xmpp.mucclient.MucClient.handleIq(MucClient.java:547)
           at org.jitsi.xmpp.mucclient.MucClient.access$500(MucClient.java:50)
           at org.jitsi.xmpp.mucclient.MucClient$2.handleIQRequest(MucClient.java:511)
           at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)

I’m planning to add some instrumentation to try to discover more about it, and to hopefully recover without bringing the whole of jitsi down, but thought I should first enquire here whether anybody else has encountered this, and might know who the issue is?
Best regards,
Craig

Is this the latest version from stable?

Hi @damencho, it’s the latest jitsi-videobridge of stable apt.

apt_repository:
repo: “deb https://download.jitsi.org stable/”
state: present

I think it’s this error: Random disconnection mid meeting on scalable setup We seem to see it both as an individual being bumped off, and as the whole system going down.
This is in the manifest:

Implementation-Title: jitsi-videobridge
Implementation-Version: 2.1-202-g5f9377b9
Build-Jdk-Spec: 1.8

Latest jvb is 2.1-273, you should update jitsi-meet.
And report do you still see the problem.

Thank you - I will try that. We’re running a whole sequence of conferences tomorrow, so I will know soon whether it works :slight_smile: Newbie question: how do I fetch that release from git?

What do you mean?

Sorry, that’s a very newbie question. I want to pull the source of jitsi-videobridge that was used for that build. Is it tagged stable/jitsi-meet_4857 ?


Or go to the github interface, find the tag and there is a link to download archive of it.

Thank you @damencho. I managed it using the git checksum in the version name :slight_smile: Is the latest stable release always tagged as stable/jitsi-meet_N with N increasing?

It is always https://github.com/jitsi/jitsi-meet/releases/latest

1 Like

https://github.com/jitsi/jitsi-videobridge/releases/latest
https://github.com/jitsi/jicofo/releases/latest

1 Like

And this also yes.

We’re still getting this error. I instrumented it to return null rather than crash, but this is what I now see:

2020-07-24 07:33:06.154 SEVERE: [4804] [confId=37a1d457b0699a06 gid=271767 type=audio conf_name=26387] ContentShim.createRtpChann
el#135: CMJ: conference.getLocalEndpoint(`db2f5cee`) returned NULL

Just before this we see in our logs:

2020-07-24 07:33:02.584 WARNING: [4455] [confId=c1013ed1a8a2d5da epId=c40367e7 gid=271767 stats_id=Lurline-XKp conf_name=26387] EndpointMessageTransport.endpointMessage#590: Unable to find endpoint to send EndpointMessage to: db2f5cee
2020-07-24 07:33:06.153 INFO: [4827] [confId=37a1d457b0699a06 epId=db2f5cee gid=271767 stats_id=Adela-35G conf_name=26387] AbstractEndpoint.expire#233: Expiring.

But for that endpoint ID, when it first appears:

grep db2f5cee jvb.log | head
2020-07-24 07:32:26.143 WARNING: [4755] [confId=37a1d457b0699a06 epId=d0440d59 gid=271767 stats_id=jibri conf_name=26387] EndpointMessageTransport.endpointMessage#590: Unable to find endpoint to send EndpointMessage to: db2f5cee
2020-07-24 07:32:26.146 WARNING: [4819] [confId=37a1d457b0699a06 epId=6558e1fc gid=271767 stats_id=Adolph-IOo conf_name=26387] EndpointMessageTransport.endpointMessage#590: Unable to find endpoint to send EndpointMessage to: db2f5cee
2020-07-24 07:32:26.150 WARNING: [4787] [confId=37a1d457b0699a06 epId=a72610c7 gid=271767 stats_id=Tomasa-BnQ conf_name=26387] EndpointMessageTransport.endpointMessage#590: Unable to find endpoint to send EndpointMessage to: db2f5cee
2```

The error does seem to only occur when we’re live streaming… not sure if that’s a clue…

We pretty stable without streaming, but once we start the live-streaming, the system can crash. It can also, though, run for 45 minutes or more without crashing, until we say, “I think it’s working!” And then of course it crashes :slight_smile: !

You see the NPE and with the latest version you say?

Yes. I need to fix this by Monday, so am planning to dig into the code over the weekend. Any pointers would be appreciated (haha, just not null pointers!) - it looks to me like it’s trying to send to an Endpoint that has expired, but right now I’m not sure why it’s doing that, or why that Endpoint looks like it’s been bad from the beginning…

I’m seeing plenty of warnings in the logs:

2020-07-24 13:37:37.617 WARNING: 
    [2405] [confId=fb48223c01a4c763 epId=820c5219 gid=284712 
    stats_id=Nelle-gBh conf_name=26390] 
    EndpointMessageTransport.endpointMessage#590: 
    Unable to find endpoint to send EndpointMessage to: 7d602ca7

Should these be cause for concern?

I’ve posted a Pull Request that I think fixes this: https://github.com/jitsi/jitsi-videobridge/pull/1357
Would really appreciate any feedback, since I need this fixed by Monday, which gives me very little time to be wrong :slight_smile:
Sorry - decided this PR doesn’t cover all bases - there’s an issue of an endpoint being ‘in flight’, during which time it needs to be valid on the conference…