Jitsi errors, some or all users getting dropped from calls at random

Warchamp7 · February 3, 2022, 9:22pm

This only started happening recently, and upgrading to latest stable does not appear to have solved it. We’ve had tens and hundreds of people on Jitsi before with no issues, across multiple bridges, but now recently seeing this. I’ve disconnected all our additional bridges and even with only a single server running everything including the bridge, still encountering this problem.

No obvious errors in jicofo or prosody logs, but the jvb log does get flooded with errors like below when it happens.

JVB 2022-02-03 21:11:34.002 WARNING: [456] [confId=f529b4d7a854dc1d gid=12620 stats_id=Valentine-8Lz conf_name=CONFNAME@conference.jitsi.DOMAIN.COM ufrag=631br1fr0m8s73 epId=712cd382 local_ufrag=631br1fr0m8s73] ConnectivityCheckClient.startCheckForPair#374: Failed to send BINDING-REQUEST(0x1)[attrib.count=6 len=92 tranID=0x729F6DC17E014BB1B53BF695]
java.lang.IllegalArgumentException: No socket found for 142.---.---.---:4443/tcp->76.---.---.---:53598/tcp
        at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:631)
        at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:581)
        at org.ice4j.stack.StunClientTransaction.sendRequest0(StunClientTransaction.java:267)
        at org.ice4j.stack.StunClientTransaction.sendRequest(StunClientTransaction.java:245)
        at org.ice4j.stack.StunStack.sendRequest(StunStack.java:680)
        at org.ice4j.ice.ConnectivityCheckClient.startCheckForPair(ConnectivityCheckClient.java:335)
        at org.ice4j.ice.Agent$StunKeepAliveRunner.sendKeepAlive(Agent.java:2693)
        at org.ice4j.ice.Agent$StunKeepAliveRunner.run(Agent.java:2671)
        at org.ice4j.util.PeriodicRunnable.executeRun(PeriodicRunnable.java:206)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
JVB 2022-02-03 21:11:34.003 WARNING: [456] [confId=f529b4d7a854dc1d gid=12620 stats_id=Valentine-8Lz conf_name=CONFNAME@conference.jitsi.DOMAIN.COM ufrag=631br1fr0m8s73 epId=712cd382 local_ufrag=631br1fr0m8s73] ConnectivityCheckClient.startCheckForPair#374: Failed to send BINDING-REQUEST(0x1)[attrib.count=6 len=92 tranID=0x739F6DC17E017F28B54D328B]
java.lang.IllegalArgumentException: No socket found for 142.---.---.---:4443/tcp->76.---.---.---:53598/tcp
at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:631)
        at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:581)
        at org.ice4j.stack.StunClientTransaction.sendRequest0(StunClientTransaction.java:267)
        at org.ice4j.stack.StunClientTransaction.sendRequest(StunClientTransaction.java:245)
        at org.ice4j.stack.StunStack.sendRequest(StunStack.java:680)
        at org.ice4j.ice.ConnectivityCheckClient.startCheckForPair(ConnectivityCheckClient.java:335)
        at org.ice4j.ice.Agent$StunKeepAliveRunner.sendKeepAlive(Agent.java:2693)
        at org.ice4j.ice.Agent$StunKeepAliveRunner.run(Agent.java:2671)
        at org.ice4j.util.PeriodicRunnable.executeRun(PeriodicRunnable.java:206)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

I also had the below in the console before the page attempted to reconnect and refresh

damencho · February 3, 2022, 10:10pm

The console shows that there is 502 coming from the bosh connection, most probably prosody error, have you checked prosody logs for errors.

Are you using TCP with the bridge? This is not recommended and for TCP connections you should use cotrun (turnserver).

Warchamp7 · February 3, 2022, 10:15pm

Nothing in prosody logs unless I’m looking in the wrong place.

Nothing in /var/log/prosody.err from today.

The lines around this failure time in /var/log/prosody.log don’t note anything of interest

Feb 03 21:24:09 mod_bosh        info    New BOSH session, assigned it sid 'db7d051a-442b-493d-adfa-52ddeef1c44b'
Feb 03 21:24:09 boshdb7d051a-442b-493d-adfa-52ddeef1c44b        info    Authenticated as 3aipkjptwnt3fs4v@jitsi.DOMAIN.COM
Feb 03 21:25:36 mod_bosh        info    Client tried to use sid 'd2000f69-b980-4633-8754-33d3a2fc0d70' which we don't know about
Feb 03 21:25:36 speakerstats.jitsi.DOMAIN.COM:speakerstats_component   warn    A module has been configured that triggers external events.
Feb 03 21:25:36 speakerstats.jitsi.DOMAIN.COM:speakerstats_component   warn    Implement this lib to trigger external events.
Feb 03 21:25:37 mod_bosh        info    New BOSH session, assigned it sid '6e1b9655-bda2-4e2f-aba7-3bfceda64f11'
Feb 03 21:25:38 bosh6e1b9655-bda2-4e2f-aba7-3bfceda64f11        info    Authenticated as k0kwnbxs6k1fejq1@jitsi.DOMAIN.COM
Feb 03 21:27:02 mod_bosh        info    New BOSH session, assigned it sid 'a9ac9939-0b65-422e-b4f9-26d53facd2b6'
Feb 03 21:27:02 bosha9ac9939-0b65-422e-b4f9-26d53facd2b6        info    Authenticated as eyugenj4xoywpa5x@jitsi.DOMAIN.COM
Feb 03 21:27:14 mod_bosh        info    New BOSH session, assigned it sid '7859eefc-db7c-441c-94e0-fcc81ed7788f'
Feb 03 21:27:15 bosh7859eefc-db7c-441c-94e0-fcc81ed7788f        info    Authenticated as abebq7kxlhwbxvjs@jitsi.DOMAIN.COM
Feb 03 21:28:13 speakerstats.jitsi.DOMAIN.COM:speakerstats_component   warn    A module has been configured that triggers external events.
Feb 03 21:28:13 speakerstats.jitsi.DOMAIN.COM:speakerstats_component   warn    Implement this lib to trigger external events.

Are you using TCP with the bridge? This is not recommended and for TCP connections you should use cotrun (turnserver).

Certainly not intentionally. Where would that be configured? Perhaps I made an error.

Edit: org.jitsi.videobridge.DISABLE_TCP_HARVESTER was indeed set to false. Changed it to true and will report back

damencho · February 3, 2022, 10:29pm

Check and nginx logs, you should see the 502 there

Warchamp7 · February 3, 2022, 10:34pm

Yep, for only a brief period when the outage happens

76.---.---.--- - - [03/Feb/2022:22:26:25 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:26 +0000] "POST /http-bind?room=confroom15 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM15" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:28 +0000] "POST /http-bind?room=confroom14 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM14" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:29 +0000] "POST /http-bind?room=confroom12 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM12" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:29 +0000] "POST /http-bind?room=confroom10 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM10" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:30 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:30 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:30 +0000] "POST /http-bind?room=confroom13 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM13" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:30 +0000] "POST /http-bind?room=confroom6 HTTP/2.0" 504 585 "https://jitsi.DOMAIN.COM/CONFROOM6" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:31 +0000] "POST /http-bind?room=confroom3 HTTP/2.0" 504 585 "https://jitsi.DOMAIN.COM/CONFROOM3" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:31 +0000] "POST /http-bind?room=confroom3 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM3" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:32 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:32 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:32 +0000] "POST /http-bind?room=confroom8 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM8" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:32 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:32 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:32 +0000] "POST /http-bind?room=confroom8 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM8" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom11 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM11" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom11 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM11" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 173 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom7 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM7" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom9 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM9" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom5 HTTP/2.0" 504 585 "https://jitsi.DOMAIN.COM/CONFROOM5" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 173 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom7 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM7" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom9 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM9" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=confroom5 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM5" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:33 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:34 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:34 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:35 +0000] "POST /http-bind?room=confroom4 HTTP/2.0" 504 585 "https://jitsi.DOMAIN.COM/CONFROOM4" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:35 +0000] "POST /http-bind?room=confroom4 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM4" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:35 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:35 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:36 +0000] "POST /http-bind?room=confroom15 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM15" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:36 +0000] "POST /http-bind?room=confroom15 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM15" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:36 +0000] "POST /http-bind?room=confroom1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:36 +0000] "POST /http-bind?room=confroom1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:38 +0000] "POST /http-bind?room=confroom14 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM14" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:38 +0000] "POST /http-bind?room=confroom14 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM14" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:39 +0000] "POST /http-bind?room=confroom12 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM12" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:39 +0000] "POST /http-bind?room=confroom10 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM10" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:39 +0000] "POST /http-bind?room=confroom12 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM12" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:39 +0000] "POST /http-bind?room=confroom10 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM10" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom13 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM13" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom13 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM13" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:40 +0000] "POST /http-bind?room=confroom3 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM3" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:41 +0000] "POST /http-bind?room=confroom8 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM8" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:41 +0000] "POST /http-bind?room=confroom11 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/CONFROOM11" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:41 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 502 173 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
76.---.---.--- - - [03/Feb/2022:22:26:41 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 502 575 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:42 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 499 0 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:42 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:42 +0000] "POST /http-bind?room=confroom7 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM7" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
72.---.---.--- - - [03/Feb/2022:22:26:42 +0000] "POST /http-bind?room=confroom9 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM9" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:42 +0000] "POST /http-bind?room=taylortest1 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/TaylorTest1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
76.---.---.--- - - [03/Feb/2022:22:26:43 +0000] "POST /http-bind?room=confroom16 HTTP/2.0" 499 0 "https://jitsi.DOMAIN.COM/CONFROOM16" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
72.---.---.--- - - [03/Feb/2022:22:26:43 +0000] "POST /http-bind?room=confroom9 HTTP/2.0" 200 312 "https://jitsi.DOMAIN.COM/CONFROOM9" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 OBS/27.1.3 Safari/537.36"

damencho · February 3, 2022, 10:50pm

Have you monitored prosody cpu usage at that time? Maybe it was overloaded …

Warchamp7 · February 3, 2022, 11:11pm

Have not checked prosody specifically but the server itself has not even crossed 50% at any point

gpatel-fr · February 3, 2022, 11:16pm

Your server has probably several cores and Prosody will ever use only one of them. So if your server has 4 cores and Prosody maxes out its one and only core, it will mean 25% load.

Warchamp7 · February 3, 2022, 11:23pm

Any guesses how we’ve gone almost an entire year without seeing this issue until the past few days though?

damencho · February 3, 2022, 11:42pm

Did anything change? Something updated? Number of participants at the same time? What was the biggest conference at that time?

Warchamp7 · February 4, 2022, 1:37am

We updated to stable in November or so after not updating since around August I think. Didn’t have any notable problems on that version till this recent issue. Updated to stable again after a full purge to verify and it didn’t solve it.

Now I am here

damencho · February 4, 2022, 1:43am

Monitor to see is prosody when this happens.

Agitatra · February 4, 2022, 3:23pm

I have the same problem.
For me it looks as if this behaviour startet to appear aften an update. Not an update of Jitsi, but an up(date|grade) of the underlying Ubuntu system. After the log4shell disaster I upgraded Jitsi to the newest version at the time being (I guess it was tag: “7409d5c” or: “stable-6726-1”). This worked fine until January, 21st 2022. Then Jistsi(jvb) ceased to get the peer-2-peer connections running, by being unable to determine the external (non-NAT) ip addresses of the participants.
The Jitsi-server was last used successfully on January, 14th. On January 17th a regular “apt upgrade” (upgraded packages see below) has been executed.
Afterwards I checked if the server was available, which it was, but unfortunately, not if two peers cound connect. The first usage afterwards war January, 21st which failed.
So I would blame the changes in Ubuntu, but this is only a gut feeling.

linux-image-5.4.0-94-generic:amd64,
linux-modules-extra-5.4.0-94-generic,
linux-modules-5.4.0-94-generic:amd64
linux-libc-dev:amd64,
libsystemd0:amd64,
linux-image-generic:amd64,
ubuntu-advantage-tools:amd64,
udev:amd64
ubuntu3.14,
245.4-4ubuntu3.15,
libudev1:amd64,
systemd-timesyncd:amd64,
systemd-sysv:amd64,
libpam-systemd:amd64,
systemd:amd64
libnss-systemd:amd64
linux-firmware:amd64

pandel · March 10, 2022, 12:47pm

@Agitatra and @Warchamp7

Did anybody of you solve this problem? I do have the same since two days. I am on Debian Bullseye and my packages have been updated as follows:

2022-03-08 21:14:41 upgrade containerd.io:amd64 1.4.12-1 1.4.13-1
2022-03-08 21:14:43 upgrade cryptsetup-initramfs:all 2:2.3.5-1 2:2.3.7-1+deb11u1
2022-03-08 21:14:43 upgrade libcryptsetup12:amd64 2:2.3.5-1 2:2.3.7-1+deb11u1
2022-03-08 21:14:43 upgrade cryptsetup-bin:amd64 2:2.3.5-1 2:2.3.7-1+deb11u1
2022-03-08 21:14:44 upgrade cryptsetup:amd64 2:2.3.5-1 2:2.3.7-1+deb11u1
2022-03-08 21:14:44 upgrade cryptsetup-run:all 2:2.3.5-1 2:2.3.7-1+deb11u1
2022-03-08 21:14:44 upgrade libexpat1-dev:amd64 2.2.10-2 2.2.10-2+deb11u2
2022-03-08 21:14:44 upgrade libexpat1:amd64 2.2.10-2 2.2.10-2+deb11u2
2022-03-08 21:14:44 upgrade libsasl2-modules-db:amd64 2.1.27+dfsg-2.1 2.1.27+dfsg-2.1+deb11u1
2022-03-08 21:14:44 upgrade libsasl2-2:amd64 2.1.27+dfsg-2.1 2.1.27+dfsg-2.1+deb11u1
2022-03-08 21:14:44 upgrade libsasl2-modules:amd64 2.1.27+dfsg-2.1 2.1.27+dfsg-2.1+deb11u1
2022-03-08 21:14:44 upgrade linux-image-amd64:amd64 5.10.92-1 5.10.92-2
2022-03-08 21:14:44 upgrade linux-image-5.10.0-11-amd64:amd64 5.10.92-1 5.10.92-2
2022-03-08 21:14:48 upgrade linux-libc-dev:amd64 5.10.92-1 5.10.92-2

Sadly, I am not able to simply downgrade, as some of the packages are not available anymore.

Don’t know what to do now…

Agitatra · March 10, 2022, 1:36pm

@pandel
No, I’m sorry I’m still stuck.
First of all: unless your Jitsi-server is only used in the intranet, DO NOT REVERT TO A FORMER VERSION, otherwise you are prone to the Log4Shell-exploit (Log4Shell - Wikipedia), whose outcome will be worse than having no Jitsi-Server at all.
The “fun-part” is, after the log4shell-upgrade some friends of mine and myself managed to get four Jitsi-instances running. After upgrades of the underlying Ubuntu, two of them ceased to work, the others are still operational. The machines are basically identical, we have no idea why the problems only show up on some machines. It looks like as if the address resolution, required to establish a peer-2-peer connection between the Jitsi-clients is somehow distorted but not with all machines.

pandel · March 10, 2022, 1:51pm

@Agitatra
I am using a Jitsi version not prone to Log4Shell so I am fine.

But even though I managed to downgrade every package during the last hour, the error still appears randomly.

FWIW…:

JVB 2022-03-10 14:47:08.893 SEVERE: [91] [confId=eee73c37214d033f epId=03836e92 gid=110332 stats_id=Raegan-Syp conf_name=1234@conference.video.XX.de] TlsServerImpl.notifyAlertReceived#247: Alert received: level=2, description=46 (certificate_unknown)
JVB 2022-03-10 14:47:08.899 SEVERE: [91] [confId=eee73c37214d033f epId=03836e92 gid=110332 stats_id=Raegan-Syp conf_name=1234@conference.video.XX.de] DtlsServer.accept#52: Error during DTLS connection: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
JVB 2022-03-10 14:47:08.899 SEVERE: [91] [confId=eee73c37214d033f epId=03836e92 gid=110332 stats_id=Raegan-Syp conf_name=1234@conference.video.XX.de] DtlsTransport.startDtlsHandshake#110: Error during DTLS negotiation, closing this transport manager
org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at org.bouncycastle.tls.DTLSRecordLayer.processRecord(Unknown Source)
        at org.bouncycastle.tls.DTLSRecordLayer.receive(Unknown Source)
        at org.bouncycastle.tls.DTLSReliableHandshake.receiveMessage(Unknown Source)
        at org.bouncycastle.tls.DTLSServerProtocol.serverHandshake(Unknown Source)
        at org.bouncycastle.tls.DTLSServerProtocol.accept(Unknown Source)
        at org.bouncycastle.tls.DTLSServerProtocol.accept(Unknown Source)
        at org.jitsi.nlj.dtls.DtlsServer.accept(DtlsServer.kt:45)
        at org.jitsi.nlj.dtls.DtlsServer.start(DtlsServer.kt:41)
        at org.jitsi.nlj.dtls.DtlsStack.start(DtlsStack.kt:150)
        at org.jitsi.videobridge.transport.dtls.DtlsTransport.startDtlsHandshake(DtlsTransport.kt:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Interestingly, the error appears randomly throughout a conference. Sometimes after a few seconds, sometimes after 10 or more minutes. It is simply not clear…

Freddie · March 10, 2022, 3:47pm

Are you using a self-signed certificate, by chance?

pandel · March 10, 2022, 4:07pm

No, it is not self signed. But just a few minutes ago we got a notice from our data center that they also have problems with GotoMeeting sessions the same way as we have with our Jitsi setup. So it seems more likely that it is maybe a provider problem. Our workstations don’t connect directly to the Internet but through a rather complex internal network with proxies and such, thus we a re using a TURN based setup. As this has been working flawlessly from day one, we first searched for the problem on our side, but maybe this time it is somewhere else…