Jitsi not working after update

Hi,
Having issue with jitsi after performing upgrade.
Running on AWS, deploying with ansible on ubuntu 18.04.
Want to describe all steps that I did, maybe this will help to understand better my issue.
Working versions (separate signal server):

  • ‘jicofo=1.0-626-1’
  • ‘jitsi-meet-web-config=1.0.4370-1’
  • ‘jitsi-meet-prosody=1.0.4370-1’
  • ‘jitsi-meet-turnserver=1.0.4370-1’
  • ‘jitsi-meet-web=1.0.4370-1’
  • ‘jitsi-meet=2.0.4966-1’
  • ‘jitsi-meet-tokens=1.0.4370-1’
    Working version (separate jvb):
  • ‘jitsi-videobridge2=2.1-304-g8488f77d-1’

Upgraded to latest available versions (ubuntu):
signal-meet:

  • ‘jicofo=1.0-747-1’
  • ‘jitsi-meet-web-config=1.0.4985-1’
  • ‘jitsi-meet-prosody=1.0.4985-1’
  • ‘jitsi-meet-turnserver=1.0.4985-1’
  • ‘jitsi-meet-web=1.0.4985-1’
  • ‘jitsi-meet=2.0.5870-1’
    JVB:
  • ‘jitsi-videobridge2=2.1-492-g5edaf7dd-1’

First thing that I faced after upgrade - link from /etc/nginx/modules-enabled/60-jitsi-meet.conf to /usr/share/jitsi-meet-turnserver/jitsi-meet.conf wasn’t created automatically like before and in browser I had “Connection error.Your device may be offline or our servers may be experiencing problems.”
After creating link, domain name with jitsi opened fine, but I coudn’t connect to any room - failed with “Unfortunately, something went wrong”.
After investigating logs, found this entries in prosody err log;
datamanager error Failed to load account storage (…accounts/jvb.dat: expected near ‘[’’ for user jvb )
(same error was for focus user)
And also had such error in JVB log:
org.jivesoftware.smack.sasl.SASLErrorException: SASLError using SCRAM-SHA-1: not-authorized

After investigating more, found that password is not written correct in following configs:
/var/lib/prosody/auth%2eexample%2ecom/accounts/jvb.dat
/var/lib/prosody/auth%2eexample%2ecom/accounts/focus.dat
it looked like this both times:
return {
[“stored_key”] = “a5e86ab7eea2…ed7e842d2”;
[“server_key”] = “387b61e74…6681aa”;
[“salt”] = “5628…7c3ae”;
[“iteration_count”] = 4096;
};
[“password”] = “password_here”;
Looks like that parameters stored_key/server_key/salt/iteration_count wasn’t defined there before, in previous jitsi version, and now, after adding them, ansible didn’t put actual password in the right way.
So, this was fixed too (moved }; to the end) and issue with JVB gone away.

But, I still have now the issue with “Unfortunately, something went wrong”. when trying to connect to any room.
In prosody log I have following lines now:
May 24 12:43:29 boshd97…4dd info Authenticated as 673...c10@guest.example.com
May 24 12:43:29 focus.example.com:component warn Component not connected, bouncing error for:

And in browser console I have error about focus:
handledError: Focus error, retry after 1000 Script: null Line: null Column: null StackTrace: Error:

But I have no idea why it’s happening now - password is written ok in /var/lib/auth…/focus.dat now and is the same as password in /etc/jitsi/jicofo/config (JICOFO_AUTH_PASSWORD=password_here)
Tried to update prosody version to 0.11.9 and set password with
prosodyctl passwd focus@auth.example.com
but this didn’t change anything. Multiple restarts of prosody/jicofo/videobridge after each config change didn’t help either.

Hope you can help me with this.

This had been dropped long time ago from the automated install.

We no longer use components across all projects, you are using outdated config file. Your config needs to look like: jitsi-meet/prosody.cfg.lua-jvb.example at 598d0149607cc8fb8f7a5f4f7fce14a159f240ee · jitsi/jitsi-meet · GitHub
And there is one command executed on installing the debian package which is needed for normal operation jitsi-meet/jitsi-meet-prosody.postinst at 598d0149607cc8fb8f7a5f4f7fce14a159f240ee · jitsi/jitsi-meet · GitHub

My advice is to make a clean VM install using the quick install guide by hand and looking at the configs adjust your ansible.

Hi again, sorry for late reply.
It seems that config wasn’t so old - I had to change only turn part and

Component “focus.example.com” “client_proxy”
target_address = “focus@auth.example.com

Also issue prosodyctl command you mentioned and - now all is working fine, thank you.
Probably only issue that left - octo stopped working with update as well. I have 2 JVB instances and all connections now go to one bridge again - however that worked fine before update with following config:

jvb - sip-communicator.properties file:

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=example.com
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.example.com
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=password_here
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.example.com
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=89fcaa…eb69410e
org.jitsi.videobridge.octo.BIND_ADDRESS=172.31.20.100
org.jitsi.videobridge.octo.BIND_PORT=4096
org.jitsi.videobridge.REGION=eu-west-1

(same config for another jvb - just MUC_NICKNAME and BIND_ADDRESS are different)

meet/jicofo - sip-communicator.properties file:

org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.example.com
org.jitsi.jicofo.jibri.BREWERY=JibriBrewery@internal.auth.example.com
org.jitsi.jicofo.jibri.PENDING_TIMEOUT=90
org.jitsi.jicofo.BridgeSelector.BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy
org.jitsi.jicofo.DISABLE_AUTO_OWNER=true

config.js - added octo as well

    p2pTestMode: false,
    octo: {
        probability: 1
    }

Previously I saw in log something like this in jicofo log:

Jicofo 2021-05-27 13:10:49.926 INFO: [20] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=8838 octo_enabled= true: [[eu-west-1, eu-west-1, eu-west-1][eu-west-1, eu-west-1]]

But now it’s empty, don’t see any log entries about octo.
Any idea what might be wrong?

Blockquote
datamanager error Failed to load account storage (…accounts/jvb.dat: expected near ‘[’’ for user jvb )
(same error was for focus user)

If the passwords in the .dat are plain, try authentication = “internal_plain” in /etc/prosody/conf.avail/[DOMAIN].cfg.lua, or try prosodyctl register jvb auth.meet.example.com PASSWORD.

Upgrading to the new stable can be kinda tricky, a fresh install saves you a lot of time and energy!

Thanks for replying. Yes, we already have authentication = “internal_plain” in config file. Issue was that ansible didn’t wrote password in correct way.
Upgrading is done with fresh install every time anyway - we have list of packages with versions defined in ansible vars, so we just recreate ASG and all software is installed automatically on new ASG from scratch.

You need to explicitly enable octo in jicfo and jvb.

Thank you for reply.
Added octo explicitly to configs:

/etc/jitsi/videobridge/jvb.conf

videobridge {
http-servers {
public {
port = 9090
}
}
octo {
enabled = true
region=“eu-west-1”
bind-address=address from sip config
bind-port=4096
recv-queue-size=1024
send-queue-size=1024
}
websockets {
enabled = true
domain = “localhost:443”
tls = true
}
}

/etc/jitsi/jicofo/jicofo.conf

Jicofo HOCON configuration. See /usr/share/jicofo/jicofo.jar/reference.conf for
available options, syntax, and default values.
jicofo {
octo: {
enabled = true
id = 1234
}
xmpp: {
client: {
client-proxy: focus.jitsi-staging.bemyapp.com
}
}
}

Added line to sip also:

org.jitsi.jicofo.SHORT_ID=1234

Restarted all components, but still nothing changed. Any other idea?

How do you test that it is not working?
Check jicofo logs, which strategy is used? I think by default it will use SingleBridgeSelectionStrategy

So when bridge load reaches:

then it will start using a second bridge.

Hm. Well, I’m actually using SplitBridgeSelectionStrategy (defined in sip config).
And yes, now I see it in log

Jicofo 2021-05-27 17:15:10.174 INFO: [15] BridgeSelectionStrategy.select#104: Selected initial bridge Bridge[jid=jvbbrewery@internal.auth.example.com/57fee1…51ebc576ec, relayId=172.13.13.107:4096, region=eu-west-1, stress=0.00] with reported stress=0.0 for participantRegion=eu-west-1 using strategy SplitBridgeSelectionStrategy

But after joining 4 peoples to test conference - I see that all them are connecting to 1 bridge - it’s jvb log is constanly updating with related stuff and other is silently adding only

JVB 2021-05-27 17:22:35.917 INFO: [21] HealthChecker.run#171: Performed a successful health check in PT0S. Sticky failure: false

So I assume that still 1 bridge is being used.

And I don’t see octa-related log in jicofo log which I seen before software version update.

Jicofo 2021-05-27 13:10:49.926 INFO: [20] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=8838 octo_enabled= true: [[eu-west-1, eu-west-1, eu-west-1][eu-west-1, eu-west-1]]

With SplitBridgeSelectionStrategy you will get other people on another bridge when the current bridge load becomes high.

Ugh…But in the doc you sent earlier it states that each participant will use separate bridge?

// SplitBridgeSelectionStrategy: Use a separate bridge for each participant (for testing).

Am I missing something?

Ah you are right … The one I was talking is IntraRegionBridgeSelectionStrategy … Hum that is strange.
Do you see both in jicofo and jvb logs that octo is enabled?

Hi again.
No, nothing in log indicates that octo is enabled - however, I performed all steps described in octo guide and all steps you tell me before.
Actually, I wanted to try to setup JVB in another AWS region (USA), connect that VPC to current region (EU), add them as bridges from another region in config and then try to enable RegionBasedBridgeSelectionStrategy or IntraRegionBasedBridgeSelectionStrategy. This setup looks similar to this:

And you answered there in first reply that for such case nginx/haproxy is needed+ aws route53.
Is this still a working solution or this can be done in better/simpler way?
Because I checked this community call Jitsi Community Call - YouTube and it seems that for your case, you were forced to move away from haproxy - but it’s still ok if not have such big amount of load that you have now.

Hi @damencho I’m having the same issue too, " Connection error Your device may be offline or our servers may be experiencing problems" I followed the manual installation guide with few changes ( I got videobridge from the package).
Could you please point me towards an updated guide for manual installation?
TIA

There is no such, sorry. You better start clean with quick install to have something working and the do anything manual.