How do I troubleshoot looping of "Unfortunately something went wrong" after apt update?

Running on Ubuntu 18.04.5 LTS

Meet keeps crashing after an upgrade, I would like to know what logs to look at to troubleshoot this. I have a snapshot of my server so I can revert back to a working version but every time I do an apt update/upgrade the Jitsi meet server stops working…

Log of packages that were upgraded and led to server breaking

apt list --upgradable

Listing... Done
jibri/stable 8.0-83-g204354d-1 all [upgradable from: 8.0-61-g99288dc-1]
jicofo/stable 1.0-740-1 all [upgradable from: 1.0-692-hf-1]
jitsi-meet/stable 2.0.5765-1 all [upgradable from: 2.0.5390-3]
jitsi-meet-prosody/stable 1.0.4900-1 all [upgradable from: 1.0.4628-1]
jitsi-meet-turnserver/stable 1.0.4900-1 all [upgradable from: 1.0.4628-1]
jitsi-meet-web/stable 1.0.4900-1 all [upgradable from: 1.0.4628-1]
jitsi-meet-web-config/stable 1.0.4900-1 all [upgradable from: 1.0.4628-1]
jitsi-videobridge2/stable 2.1-478-gc6da57bd-1 all [upgradable from: 2.1-416-g2f43d1b4-1]



apt upgrade

Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  fonts-lato javascript-common libjs-jquery libruby2.5 rake ruby ruby-did-you-mean ruby-hocon ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit ruby2.5 rubygems-integration zip
The following packages will be upgraded:
  jibri jicofo jitsi-meet jitsi-meet-prosody jitsi-meet-turnserver jitsi-meet-web jitsi-meet-web-config jitsi-videobridge2
8 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
Need to get 115 MB of archives.
After this operation, 6,669 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 fonts-lato all 2.0-2 [2,698 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 rubygems-integration all 1.11 [4,994 B]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 rake all 12.3.1-1ubuntu0.1 [44.9 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-did-you-mean all 1.2.0-2 [9,700 B]
Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-minitest all 5.10.3-1 [38.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-net-telnet all 0.1.1-2 [12.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-power-assert all 0.3.0-1 [7,952 B]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby-test-unit all 3.2.5-1 [61.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libruby2.5 amd64 2.5.1-1ubuntu1.9 [3,072 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ruby2.5 amd64 2.5.1-1ubuntu1.9 [48.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 ruby amd64 1:2.5.1 [5,712 B]
Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 ruby-hocon all 1.2.5-1 [74.8 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 javascript-common all 11 [6,066 B]
Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-jquery all 3.2.1-1 [152 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 zip amd64 3.0-11build1 [167 kB]
Get:16 https://download.jitsi.org stable/ jitsi-videobridge2 2.1-478-gc6da57bd-1 [33.4 MB]
Get:17 https://download.jitsi.org stable/ jitsi-meet 2.0.5765-1 [3,344 B]
Get:18 https://download.jitsi.org stable/ jicofo 1.0-740-1 [21.7 MB]
Get:19 https://download.jitsi.org stable/ jitsi-meet-web 1.0.4900-1 [9,139 kB]
Get:20 https://download.jitsi.org stable/ jitsi-meet-web-config 1.0.4900-1 [18.8 kB]
Get:21 https://download.jitsi.org stable/ jitsi-meet-prosody 1.0.4900-1 [45.3 kB]
Get:22 https://download.jitsi.org stable/ jitsi-meet-turnserver 1.0.4900-1 [6,248 B]
Get:23 https://download.jitsi.org stable/ jibri 8.0-83-g204354d-1 [44.6 MB]
Fetched 115 MB in 16s (7,171 kB/s)
Preconfiguring packages ...
Selecting previously unselected package fonts-lato.
(Reading database ... 110476 files and directories currently installed.)
Preparing to unpack .../fonts-lato_2.0-2_all.deb ...
Unpacking fonts-lato (2.0-2) ...
Preparing to unpack .../jitsi-videobridge2_2.1-478-gc6da57bd-1_all.deb ...
Unpacking jitsi-videobridge2 (2.1-478-gc6da57bd-1) over (2.1-416-g2f43d1b4-1) ...
Setting up jitsi-videobridge2 (2.1-478-gc6da57bd-1) ...
Installing new version of config file /etc/jitsi/videobridge/logging.properties ...
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-lxd-inotify.conf ...
fs.inotify.max_user_instances = 1024
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /etc/sysctl.d/20-jvb-udp-buffers.conf ...
net.core.rmem_max = 10485760
net.core.netdev_max_backlog = 100000
* Applying /usr/lib/sysctl.d/50-default.conf ...
net.ipv4.conf.all.promote_secondaries = 1
net.core.default_qdisc = fq_codel
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.conf ...
(Reading database ... 110489 files and directories currently installed.)
Preparing to unpack .../00-jitsi-meet_2.0.5765-1_all.deb ...
Unpacking jitsi-meet (2.0.5765-1) over (2.0.5390-3) ...
Selecting previously unselected package rubygems-integration.
Preparing to unpack .../01-rubygems-integration_1.11_all.deb ...
Unpacking rubygems-integration (1.11) ...
Selecting previously unselected package rake.
Preparing to unpack .../02-rake_12.3.1-1ubuntu0.1_all.deb ...
Unpacking rake (12.3.1-1ubuntu0.1) ...
Selecting previously unselected package ruby-did-you-mean.
Preparing to unpack .../03-ruby-did-you-mean_1.2.0-2_all.deb ...
Unpacking ruby-did-you-mean (1.2.0-2) ...
Selecting previously unselected package ruby-minitest.
Preparing to unpack .../04-ruby-minitest_5.10.3-1_all.deb ...
Unpacking ruby-minitest (5.10.3-1) ...
Selecting previously unselected package ruby-net-telnet.
Preparing to unpack .../05-ruby-net-telnet_0.1.1-2_all.deb ...
Unpacking ruby-net-telnet (0.1.1-2) ...
Selecting previously unselected package ruby-power-assert.
Preparing to unpack .../06-ruby-power-assert_0.3.0-1_all.deb ...
Unpacking ruby-power-assert (0.3.0-1) ...
Selecting previously unselected package ruby-test-unit.
Preparing to unpack .../07-ruby-test-unit_3.2.5-1_all.deb ...
Unpacking ruby-test-unit (3.2.5-1) ...
Selecting previously unselected package libruby2.5:amd64.
Preparing to unpack .../08-libruby2.5_2.5.1-1ubuntu1.9_amd64.deb ...
Unpacking libruby2.5:amd64 (2.5.1-1ubuntu1.9) ...
Selecting previously unselected package ruby2.5.
Preparing to unpack .../09-ruby2.5_2.5.1-1ubuntu1.9_amd64.deb ...
Unpacking ruby2.5 (2.5.1-1ubuntu1.9) ...
Selecting previously unselected package ruby.
Preparing to unpack .../10-ruby_1%3a2.5.1_amd64.deb ...
Unpacking ruby (1:2.5.1) ...
Selecting previously unselected package ruby-hocon.
Preparing to unpack .../11-ruby-hocon_1.2.5-1_all.deb ...
Unpacking ruby-hocon (1.2.5-1) ...
Preparing to unpack .../12-jicofo_1.0-740-1_all.deb ...
Unpacking jicofo (1.0-740-1) over (1.0-692-hf-1) ...
Preparing to unpack .../13-jitsi-meet-web_1.0.4900-1_all.deb ...
Unpacking jitsi-meet-web (1.0.4900-1) over (1.0.4628-1) ...
Preparing to unpack .../14-jitsi-meet-web-config_1.0.4900-1_all.deb ...
Unpacking jitsi-meet-web-config (1.0.4900-1) over (1.0.4628-1) ...
Preparing to unpack .../15-jitsi-meet-prosody_1.0.4900-1_all.deb ...
Unpacking jitsi-meet-prosody (1.0.4900-1) over (1.0.4628-1) ...
Setting up jitsi-meet-web-config (1.0.4900-1) ...
----------------

You can now switch to a Let’s Encrypt certificate. To do so, execute:
/usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh

----------------
(Reading database ... 111789 files and directories currently installed.)
Preparing to unpack .../jitsi-meet-turnserver_1.0.4900-1_all.deb ...
Unpacking jitsi-meet-turnserver (1.0.4900-1) over (1.0.4628-1) ...
Selecting previously unselected package javascript-common.
Preparing to unpack .../javascript-common_11_all.deb ...
Unpacking javascript-common (11) ...
Preparing to unpack .../jibri_8.0-83-g204354d-1_all.deb ...
Unpacking jibri (8.0-83-g204354d-1) over (8.0-61-g99288dc-1) ...
Selecting previously unselected package libjs-jquery.
Preparing to unpack .../libjs-jquery_3.2.1-1_all.deb ...
Unpacking libjs-jquery (3.2.1-1) ...
Selecting previously unselected package zip.
Preparing to unpack .../zip_3.0-11build1_amd64.deb ...
Unpacking zip (3.0-11build1) ...
Setting up libjs-jquery (3.2.1-1) ...
Setting up zip (3.0-11build1) ...
Setting up jitsi-meet-web (1.0.4900-1) ...
Setting up fonts-lato (2.0-2) ...
Setting up ruby-did-you-mean (1.2.0-2) ...
Setting up jitsi-meet-prosody (1.0.4900-1) ...
Setting up jibri (8.0-83-g204354d-1) ...
Installing new version of config file /etc/jitsi/jibri/xorg-video-dummy.conf ...
Setting up ruby-net-telnet (0.1.1-2) ...
Setting up rubygems-integration (1.11) ...
Setting up javascript-common (11) ...
Setting up ruby-minitest (5.10.3-1) ...
Setting up ruby-power-assert (0.3.0-1) ...
Setting up jitsi-meet-turnserver (1.0.4900-1) ...
------------------------------------------------

turnserver is already configured on this machine.

------------------------------------------------
Setting up ruby-test-unit (3.2.5-1) ...
Setting up rake (12.3.1-1ubuntu0.1) ...
Setting up libruby2.5:amd64 (2.5.1-1ubuntu1.9) ...
Setting up ruby2.5 (2.5.1-1ubuntu1.9) ...
Setting up ruby (1:2.5.1) ...
Setting up ruby-hocon (1.2.5-1) ...
Setting up jicofo (1.0-740-1) ...
Installing new version of config file /etc/init.d/jicofo ...
Installing new version of config file /etc/jitsi/jicofo/logging.properties ...
Setting up jitsi-meet (2.0.5765-1) ...
We found the old key EF8B479E2DC1389C, which is no longer valid and we removed it.
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Processing triggers for systemd (237-3ubuntu10.47) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for fontconfig (2.12.6-0ubuntu2) ...
Processing triggers for ureadahead (0.100.0-21) ...


This might help - [How to] Why do I see "Unfortunately something went wrong"? Why does my room 'crash'?

Thanks, I had a look at that. I am not using it externally at the moment so no ports would be blocked and I put a tail on the jicofo.log and jvb.log but nothing of interest shows up even when I try to start a conference…

tail -f /var/log/jitsi/jvb.log
JVB 2021-05-14 11:18:18.319 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:18:28.319 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:18:38.319 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:18:48.320 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:18:58.309 INFO: [19] VideobridgeExpireThread.expire#140: Running expire()
JVB 2021-05-14 11:18:58.320 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:19:08.320 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:19:18.319 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:19:28.319 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
JVB 2021-05-14 11:19:38.319 INFO: [20] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false
tail -f /var/log/jitsi/jicofo.log
Jicofo 2021-05-14 08:43:04.349 INFO: [1] org.eclipse.jetty.util.log.Log.initialized: Logging initialized @1058ms to org.eclipse.jetty.util.log.Slf4jLog
Jicofo 2021-05-14 08:43:04.350 INFO: [19] JvbDoctor.addBridge#153: Scheduled health-check task for: jvbbrewery@internal.auth.meet.me.com/fba5687e-14e3-4da
Jicofo 2021-05-14 08:43:04.351 INFO: [13] [type=jibri brewery=jibribrewery] BaseBrewery.start#172: Joined the room.
Jicofo 2021-05-14 08:43:04.351 INFO: [19] [type=jibri brewery=jibribrewery] BaseBrewery.addInstance#338: Added brewery instance: jibribrewery@internal.auth.me
Jicofo 2021-05-14 08:43:04.351 INFO: [13] [xmpp_connection=client] XmppProviderImpl.fireRegistrationStateChanged#338: Set replyTimeout=PT15S
Jicofo 2021-05-14 08:43:04.411 INFO: [1] org.eclipse.jetty.server.Server.doStart: jetty-9.4.35.v20201120; built: 2020-11-20T21:17:03.964Z; git: bdc54f03a5e0a7
Jicofo 2021-05-14 08:43:04.773 WARNING: [1] org.glassfish.jersey.internal.inject.Providers.checkProviderRuntime: A provider org.jitsi.rest.Version registered o constraint configuration problems the provider org.jitsi.rest.Version will be ignored.
Jicofo 2021-05-14 08:43:04.963 INFO: [1] org.eclipse.jetty.server.handler.ContextHandler.doStart: Started o.e.j.s.ServletContextHandler@151ef57f{/,null,AVAILA
Jicofo 2021-05-14 08:43:04.972 INFO: [1] org.eclipse.jetty.server.AbstractConnector.doStart: Started ServerConnector@3efe7086{HTTP/1.1, (http/1.1)}{0.0.0.0:88
Jicofo 2021-05-14 08:43:04.972 INFO: [1] org.eclipse.jetty.server.Server.doStart: Started @1682ms



When I connect in the browser and open the js console I see a lot of errors…

2021-05-14T11:26:18.224Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>:  UnhandledError: Focus error, retry after 1000 Script: null Line: null Column: null StackTrace:  Error: Focus error, retry after 1000
    at l._allocateConferenceFocusError (https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:10:182368)
    at https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:10:181419
    at I.Handler.handler (https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:1:31531)
    at I.Handler.run (https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:1:26830)
    at https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:1:35268
    at Object.forEachChild (https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:1:18492)
    at I.Connection._dataRecv (https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:1:35117)
    at O.Bosh._onRequestStateChange (https://meet.me.com/libs/lib-jitsi-meet.min.js?v=4900:1:55102)
	
	
2021-05-14T11:27:19.170Z [modules/xmpp/moderator.js] <l._allocateConferenceFocusError>:  Focus error, retry after 2000 

2021-05-14T11:28:05.963Z [conference.js] <ne._onConferenceFailed>:  CONFERENCE FAILED: conference.focusDisconnected focus.meet.me.com 1


Sounds like a prosody issue.

Depending on your instance age, I would suggest a clean install if possible, since the last update things are changing on files not touched by the updates.

So you might be missing out of some features or new directions.

  • conf.json > jibri.config
  • toolbar at config.js
  • lastN UI control

Some of what I can think of.

Jitis is also broken for me after the latest update Jitsi meet broken after update ubuntu 20.04.2 LTS