Jicofo crashes with two participants - Failed to select initial bridge

Am a bit lost trying to setup jvb+jicofo+jitsi-meet from latest stable tag 5142.

Am trying to set it up in an LXD container on meet.rys.pw with host forwarding ports 10000 and 8443 to the container.

When a second participant connects, Jitsi-meet crashes, and jicofo has the following in the log:
Oct 30 02:19:52 meet jicofo.sh[994]: Jicofo 2020-10-30 02:19:52.319 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelectionStrategy.log() Failed to select initial bridge for participantRegion=europe
Oct 30 02:19:52 meet jicofo.sh[994]: Jicofo 2020-10-30 02:19:52.319 SEVERE: [28] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant – no bridge available.

Am not sure where the issue could be, here is my full setup:

nginx.conf https://haste.rys.pw/akamazohus.nginx
jitsi-meet config.js https://haste.rys.pw/raw/egotojufav
jitsi-meet interface_config.js https://haste.rys.pw/raw/inixexitol
jicofo sip-communicator.properties - org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.meet.rys.pw
jicofo config https://haste.rys.pw/raw/emamalijan
jvb sip-communicator.properties https://haste.rys.pw/raw/ewubumehoq
jvb config https://haste.rys.pw/raw/fafazavaga
prosody config https://haste.rys.pw/raw/zepidiwera

Accounts were created via:
prosodyctl register jvb auth.meet.rys.pw 31qvd4fak4wknb7k
prosodyctl register focus auth.meet.rys.pw efo3n2n3sec39rgv

I had a similar setup working some months ago on an older version.
Any pointers are welcome.

one the one hand you say it’s based on stable, on the other hand you created jvb and focus yourself, something stable decidedly does. So it seems your setup is rather specific and it’s very difficult to debug this remotely by reading config files.
Generally this kind of problem could comes from jvb not registering with prosody. In your place I’d validate telnet interface to prosody (add module admin_telnet to host) and test with
telnet localhost 5582
and then
c2s:show()
you should see focus and jvb. If not maybe secret is wrong.

c2s:show()
| auth.meet.rys.pw
|    focus@auth.meet.rys.pw/focus573712825333920 [c2s560af2ac73b0] available(0) (encrypted)
| OK: Total: 1 clients

Well, jvb is missing.

JVB log: https://haste.rys.pw/jepaxebabo
The only big thing seems to be the statsCollector error, but that seems to be a harmless bug related to statistics that is already fixed in master: https://github.com/jitsi/jitsi-videobridge/pull/1507

maybe secret is wrong

How can I verify this?
As you can see I provided the configs with the secrets(you can see the .service files in the links below), and they should be alright from what I see.

one the one hand you say it’s based on stable, on the other hand you created jvb and focus yourself, something stable decidedly does.

It’s both based on the stable tag and built from the following packages (which I maintain)




wow, that’s really a pared-down log. Standard deb install says a lot more. Your jvb log config could be more verbose. This warning is disturbing:

WARNING: Running with open files limit 1048576 (hard 1048576), thread limit null (hard null)

I don’t know if this is a problem, but maybe you could take some inspiration from the project service file.

[Service]
SuccessExitStatus=143
# allow bind to 80 and 443
AmbientCapabilities=CAP_NET_BIND_SERVICE
EnvironmentFile=/etc/jitsi/videobridge/config
Environment=LOGFILE=/var/log/jitsi/jvb.log
User=jvb
RuntimeDirectory=jitsi-videobridge
RuntimeDirectoryMode=0750
PIDFile=/var/run/jitsi-videobridge/jitsi-videobridge.pid
# more threads for this process
TasksMax=65000
# allow more open files for this process
LimitNPROC=65000
LimitNOFILE=65000
ExecStart=/bin/bash -c "exec /usr/share/jitsi-videobridge/jvb.sh ${JVB_OPTS} < /dev/null >> ${LOGFILE} 2>&1"
ExecStartPost=/bin/bash -c "echo $MAINPID > /var/run/jitsi-videobridge/jitsi-videobridge.pid"
Restart=on-failure
RestartSec=2

in deb config, secret is set at 2 different locations (this seems cheesy but it’s how it is)

config.ssh:JVB_SECRET=mysecret
sip-communicator.properties:org.jitsi.videobridge.xmpp.user.shard.PASSWORD=mysecret

I find it in
/var/lib/prosody/auth%2etestreunion%2ek%2dlogic%2efr/accounts/jvb.dat
(it’s in the clear in my config)
Well, what is happening if I change the prosody password ? obviously nothing works, but nothing of interest in the prosody log, no smoking gun like ‘jvb tried to authenticate but it went all wrong’ :slight_smile:
However, If you validate debug in the prosody.cfg.lua file
debug = “/var/log/prosody/prosody.dbg”;
if will report exactly that.
But it’s not necessary to enable prosody debug log, if jvb log is set to WARNING, it will report it too.

The openfiles limit should be a problem only for large instances, which mine isn’t.

I enabled debug logging in prosody, restarted everything and crashed it by joining two people, but nothing stands out to me in the log: https://haste.rys.pw/raw/cenujekagi

Except for the TLS cert error, but unsure what that’s about.

Password seems correct.

(ssh) root@meet : ~
[0] # cat /var/lib/prosody/auth%2emeet%2erys%2epw/accounts/jvb.dat 
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /var/lib/prosody/auth%2emeet%2erys%2epw/accounts/jvb.dat
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ return {
   2   │     ["password"] = "31qvd4fak4wknb7k";
   3   │ };
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

(ssh) root@meet : ~
[0] # cat /etc/jitsi/videobridge/config | grep JVB_SECRET
JVB_SECRET=31qvd4fak4wknb7k

(ssh) root@meet : ~
[0] # cat /etc/jitsi/videobridge/sip-communicator.properties | grep PASSW
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=31qvd4fak4wknb7k

the warning is about thread limits. A thread limit of zero could be a problem (not sure if it’s interpreted as unlimited, though)

it don’t matter since jvb is not even attempting to connect. You should find the string ‘jvb’ in the debug log.
So my guess is that jvb either don’t start, or is trying to connect to the wrong host, or the wrong port.

I tested the limits just in case, but the issue remains.

Oct 31 00:42:27 meet jvb.sh[5619]: INFO: Running with open files limit 65000 (hard 65000), thread limit 65000 (hard 65000).

Well, it seems to start. It’s eating 78MB~ RAM when it’s finished up, so it’s doing something at least.

I have tried launching jvb from the terminal just in case to confirm my sanity, but it same result.

[130] # ./jvb.sh --host=localhost --domain=meet.rys.pw --port=5347 --secret=31qvd4fak4wknb7k --apis="''"

I have also tried with 127.0.0.1 instead of localhost.

The port seems bound fine.

[0] # ss -tulpn | grep 5347
tcp   LISTEN 0      128                          127.0.0.1:5347       0.0.0.0:*    users:(("lua5.2",pid=127,fd=5))                    
tcp   LISTEN 0      128                              [::1]:5347          [::]:*    users:(("lua5.2",pid=127,fd=17))  

I also tried compiling the git version instead in case it was related to the stats error, but that does not seem to be the case, git JVB log: https://haste.rys.pw/jokexutiso

2 things:

  • the port 5347 is irrelevant for jvb. Jicofo is using ports 5347 and 5222 to connect to Prosody. Jvb is using only port 5222 (MUC connection)

  • your jvb does not even attempt to connect. In my log I have this:

Initializing a new MucClient

So the main problem is very probably that your jvb does not use the right config file.

2020-10-30 20:32:10.089 INFO: [1] JitsiConfig.<clinit>#47: Initialized newConfig: merge of /etc/jitsi/videobridge/jvb.conf: 1,application.conf @ jar:file:/usr/share/jitsi-videobridge/jitsi-videobridge.jar!/application.conf: 1,system properties,reference.conf @ jar:file:/usr/share/jitsi-videobridge/jitsi-videobridge.jar!/reference.conf: 1,reference.conf @ jar:file:/usr/share/jitsi-videobridge/lib/ice4j-3.0-22-g67ffceb.jar!/reference.conf: 1,reference.conf @ jar:file:/usr/share/jitsi-videobridge/lib/jitsi-media-transform-1.0-198-g1babb83.jar!/reference.conf: 1
2020-10-30 20:32:10.295 INFO: [1] ReadOnlyConfigurationService.reloadConfiguration#51: loading config file at path /etc/jitsi/videobridge/sip-communicator.properties

while your log show this:

2020-10-31 01:18:19.153 INFO: [1] JitsiConfig.<clinit>#47: Initialized newConfig: merge of application.conf @ jar:file:/usr/share/jitsi-videobridge/jitsi-videobridge.jar!/application.conf: 1,system properties,reference.conf @ jar:file:/usr/share/jitsi-videobridge/jitsi-videobridge.jar!/referenceidge/lib/ice4j-3.0-22-g67ffceb.jar!/reference.conf: 1,reference.conf @ jar:file:/usr/share/jitsi-videobridge/lib/jitsi-media-transform-1.0-198-g1babb83.jar!/reference.conf: 1
2020-10-31 01:18:19.167 INFO: [1] ReadOnlyConfigurationService.reloadConfiguration#40: net.java.sip.communicator.SC_HOME_DIR_LOCATION not set

here is the jvb command line for a working jvb

ps fauxww | grep -i jvb
jvb      15894  0.0  5.3 6902616 109428 ?      Ssl  Oct30  40:43 java -Xmx3072m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djdk.tls.ephemeralDHKeySize=2048 -Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties -Dcom.sun.management.jmxremote.port=6666 -Dcom.sun.management.jmxremote.rmi.port=6666 -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -cp /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/* org.jitsi.videobridge.MainKt --apis=rest,

Would you be so kind and post the following files? I thought there’s just one config, but it seems two are used now?

/etc/jitsi/videobridge/config
/etc/jitsi/videobridge/jvb.conf

Seems like had the exact some problems like you, @Martin_Rys, when using thc arch packages.
However, it seems like I was able to figure out workarounds.
In difference to the defaults when installing the arch packages, I did the following:

In prosody I added another Component (stolen from the debian packages):

Component "internal.auth.meet.sensing-the-world.de" "muc"
    admins = { "focus@auth.meet.sensing-the-world.de", "jvb@auth.meet.sensing-the-world.de" }

and I changed the authentication for auth.sensing-the-world.de to internal_hashed.

And then using this sip-communicator.properties for jicofo:

org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.meet.sensing-the-world.de

In jvb sip-communicator.properties, I have also:

org.jitsi.videobridge.user.shard.MUC_JIDS=JvbBrewery@internal.auth.meet.sensing-the-world.de
org.jitsi.videobridge.user.shard.MUC_NICKNAME=jvb2

Maybe something of this helps you, too.

As you can see in the configs, I already have all of that configured, sans internal_hashed instead of internal_plain, which seems to just be a security related change (if you have a problem with the passwords being saved unhashed on the FS).

Are you behind NAT? I think that’s part of my issue.
Could you post full configs? (with secrets omitted of course)

Yes, I am behind a NAT, too. Hm. Sorry, I missed your changes in the configs.
Here are mine.

jvb config https://pastebin.com/EUjqj4wU
jvb sip-communicator.properties https://pastebin.com/ABpgyr0W
prosody.cfg.lua https://pastebin.com/DFu2Tudq
jicofo config https://pastebin.com/Pqw2Letr
jicofo sip-communicator.properties https://pastebin.com/5jH6g3CT

Also, I did

peter@ark ~> sudo prosodyctl register jvb auth.meet.sensing-the-world.de SECRET1
peter@ark ~> sudo prosodyctl register focus auth.meet.sensing-the-world.de SECRET3

sorry, as a new user I was not allowed to post so many direct links

I tried replicating your setup to no avail.
You explicitly enable xmpp API, which I know used to break JVB back in May 2020, so that’s even weirder

I tried cherry picking everything different from your configs into mine to no avail

What version of JVB are you running and which ports have you forwarded?

I am not sure whether apis=xmpp, matters. I put it there because it was in the arch wiki.

tcp/443, tcp/4443, udp/10000

peter@ark ~> yay -Qi jitsi-videobridge
Name                     : jitsi-videobridge-bin
Version                  : 2.1+376+g9f12bfe2-1

Yeah so same as mine, except you’re using -bin, which should not matter. Tried with -bin just in case, nothing.

jicofo-1.0_644-1-x86_64.pkg.tar.zst
jitsi-meet-1.0.4466-1-x86_64.pkg.tar.zst
jitsi-videobridge-2.1_376_g9f12bfe2-1-x86_64.pkg.tar.zst

I was forwarding tcp/8443 and udp/10000. Added tcp/80,443 and 4443 now and still nothing.

I can already see this being some dumb typo, I just can’t figure out what’s wrong.

Grasping at straws here, but are you by chance using nginx? If so, could you post the config?

config is still used as before
sip-communicator.properties is supposed to be replaced by jvb.conf but as most people using this forum copy and paste from old posts, nobody is using the new format. The 2 formats can coexist.

Dont’ bother with nginx configuration, if jvb can’t connect to prosody no amount of nginx configuration will solve the problem.

config

JVB_HOSTNAME=meet.myurl.mytld
JVB_HOST=
JVB_PORT=5347
JVB_SECRET=mysecret
JVB_OPTS="--apis=rest,"

jvb.conf

videobridge {
    http-servers {
        public {
            port = 9090
        }
    }
    websockets {
        enabled = true
        domain = "meet.myurl.mytld:443"
        tls = true
    }
}

sip-communicator.properties

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.meet.myurl.mytld
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=mysecret
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.meet.myurl.mytld
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=93ba3a55-c9e2-4ae8-b73d-e974b67a4491

Edit: looking up my post, I realize that the config includes a ref to port 5357 in config file. Port 5347 is NOT used in default config (and probably not in not so default configs). I think that’s a relic from another time, the port to watch is 5222 (ss -tapnu | grep 5222 -> you should see 2 different Java instances connected to Prosody)