Exception while attempting load-balancing for Jitsi conferences

I referred to this video and set up a 2nd videobridge on a separate EC2 instance.
Tutorial video: How to Load Balance Jitsi Meet - Jitsi

When I start the videobridge, I get an exception “Connection refused”. TCP port 5347 is open on inbound security rule. I have pasted config from Jitsi server (Server1) and exception log from videobridge2 (Server2). Please let me know what might be wrong in the configuration. Appreciate your help.

Jitsi server (Server1)

/etc/prosody/conf.d/meet.MYDOMAIN.com.cfg.lua
Component “videobridge2.meet.MYDOMAIN.com
component_secret = “xxxxxxxxx”

/etc/jitsi/jicofo/jicofo.conf
jicofo {
xmpp: {
client: {
client-proxy: “focus.meet.MYDOMAIN.com
xmpp-domain: “meet.MYDOMAIN.com
domain: “auth.meet.MYDOMAIN.com
username: “focus”
password: “Fv9ZAAlrVLcoa0V8”
}
trusted-domains: [ “recorder.meet.MYDOMAIN.com” ]
}
bridge: {
brewery-jid: “JvbBrewery@internal.auth.meet.MYDOMAIN.com
}
authentication: {
enabled: false
type: XMPP
login-url: meet.MYDOMAIN.com
}
}

/etc/prosody/prosody.cfg.lua
– Prosody Example Configuration File

– Information on configuring Prosody can be found on our
– website at Configuring Prosody – Prosody IM

– Tip: You can check that the syntax of this file is correct
– when you have finished by running this command:
– prosodyctl check config
– If there are any errors, it will let you know what and where
– they are, otherwise it will keep quiet.

– The only thing left to do is rename this file to remove the .dist ending, and fill in the
– blanks. Good luck, and happy Jabbering!

---------- Server-wide settings ----------
– Settings in this section apply to the whole server and are the default settings
– for any virtual hosts

– This is a (by default, empty) list of accounts that are admins
– for the server. Note that you must create the accounts separately
– (see Creating accounts – Prosody IM for info)
– Example: admins = { “user1@example.com”, “user2@example.net” }
admins = { }

component_ports = { 5347 }
component_interface = “xxx.xxx.xxx.xxx”

– Enable use of libevent for better performance under high load
– For more information see: libevent – Prosody IM
–use_libevent = true

– Prosody will always look in its source directory for modules, but
– this option allows you to specify additional locations where Prosody
– will look for modules first. For community modules, see https://modules.prosody.im/
– For a local administrator it’s common to place local modifications
– under /usr/local/ hierarchy:
plugin_paths = { “/usr/local/lib/prosody/modules” }

– This is the list of modules Prosody will load on startup.
– It looks for mod_modulename.lua in the plugins folder, so make sure that exists too.
– Documentation for bundled modules can be found at: Prosody Modules – Prosody IM
modules_enabled = {

    -- Generally required
            "roster"; -- Allow users to have a roster. Recommended ;)
            "saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
            "tls"; -- Add support for secure TLS on c2s/s2s connections
            "dialback"; -- s2s dialback support
            "disco"; -- Service discovery

    -- Not essential, but recommended
            "carbons"; -- Keep multiple clients in sync
            "pep"; -- Enables users to publish their avatar, mood, activity, playing music and more
            "private"; -- Private XML storage (for room bookmarks, etc.)
            "blocklist"; -- Allow users to block communications with other users
            "vcard4"; -- User profiles (stored in PEP)
            "vcard_legacy"; -- Conversion between legacy vCard and PEP Avatar, vcard
            "limits"; -- Enable bandwidth limiting for XMPP connections

    -- Nice to have
            "version"; -- Replies to server version requests
            "uptime"; -- Report how long server has been running
            "time"; -- Let others know the time here on this server
            "ping"; -- Replies to XMPP pings with pongs
            "register"; -- Allow users to register on this server using a client and change passwords
            --"mam"; -- Store messages in an archive and allow users to access it
            --"csi_simple"; -- Simple Mobile optimizations

    -- Admin interfaces
            "admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
            --"admin_telnet"; -- Opens telnet console interface on localhost port 5582

    -- HTTP modules
            --"bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
            --"websocket"; -- XMPP over WebSockets
            --"http_files"; -- Serve static files from a directory over HTTP

    -- Other specific functionality
            "posix"; -- POSIX functionality, sends server to background, enables syslog, etc.
            --"groups"; -- Shared roster support
            --"server_contact_info"; -- Publish contact information for this service
            --"announce"; -- Send announcement to all online users
            --"welcome"; -- Welcome users who register accounts
            --"watchregistrations"; -- Alert admins of registrations
            --"motd"; -- Send a message to users when they log in
            --"legacyauth"; -- Legacy authentication. Only used by some old clients and bots.
            --"proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use

}

– These modules are auto-loaded, but should you want
– to disable them then uncomment them here:
modules_disabled = {
– “offline”; – Store offline messages
– “c2s”; – Handle client connections
– “s2s”; – Handle server-to-server connections
}

– Disable account creation by default, for security
– For more information see Creating accounts – Prosody IM
allow_registration = false

– Debian:
– Do not send the server to background, either systemd or start-stop-daemon take care of that.

daemonize = false;

– Debian:
– Please, don’t change this option since /run/prosody/
– is one of the few directories Prosody is allowed to write to

pidfile = “/run/prosody/prosody.pid”;

– Force clients to use encrypted connections? This option will
– prevent clients from authenticating unless they are using encryption.

c2s_require_encryption = true

– Force servers to use encrypted connections? This option will
– prevent servers from authenticating unless they are using encryption.

s2s_require_encryption = true

– Force certificate authentication for server-to-server connections?

s2s_secure_auth = false

– Some servers have invalid or self-signed certificates. You can list
– remote domains here that will not be required to authenticate using
– certificates. They will be authenticated using DNS instead, even
– when s2s_secure_auth is enabled.

–s2s_insecure_domains = { “insecure.example” }

– Even if you disable s2s_secure_auth, you can still require valid
– certificates for some domains by specifying a list here.

–s2s_secure_domains = { “jabber.org” }

– Enable rate limits for incoming client and server connections

limits = {
c2s = {
rate = “10kb/s”;
};
s2sin = {
rate = “30kb/s”;
};
}

– Select the authentication backend to use. The ‘internal’ providers
– use Prosody’s configured data storage to store the authentication data.

authentication = “internal_hashed”

– Select the storage backend to use. By default Prosody uses flat files
– in its configured data directory, but it also supports more backends
– through modules. An “sql” backend is included by default, but requires
– additional dependencies. See Data storage – Prosody IM for more info.

–storage = “sql” – Default is “internal” (Debian: “sql” requires one of the
– lua-dbi-sqlite3, lua-dbi-mysql or lua-dbi-postgresql packages to work)

– For the “sql” backend, you can uncomment one of the below to configure:
–sql = { driver = “SQLite3”, database = “prosody.sqlite” } – Default. ‘database’ is the filename.
–sql = { driver = “MySQL”, database = “prosody”, username = “prosody”, password = “secret”, host = “localhost” }
–sql = { driver = “PostgreSQL”, database = “prosody”, username = “prosody”, password = “secret”, host = “localhost” }

– Archiving configuration
– If mod_mam is enabled, Prosody will store a copy of every message. This
– is used to synchronize conversations between multiple clients, even if
– they are offline. This setting controls how long Prosody will keep
– messages in the archive before removing them.

archive_expires_after = “1w” – Remove archived messages after 1 week

– You can also configure messages to be stored in-memory only. For more
– archiving options, see mod_mam – Prosody IM

– Logging configuration
– For advanced logging see Logging – Prosody IM

– Debian:
– Logs info and higher to /var/log
– Logs errors to syslog also
log = {
– Log files (change ‘info’ to ‘debug’ for debug logs):
info = “/var/log/prosody/prosody.log”;
error = “/var/log/prosody/prosody.err”;
– Syslog:
{ levels = { “error” }; to = “syslog”; };
}

– Uncomment to enable statistics
– For more info see Statistics – Prosody IM
– statistics = “internal”

– Certificates
– Every virtual host and component needs a certificate so that clients and
– servers can securely verify its identity. Prosody will automatically load
– certificates/keys from the directory specified here.
– For more information, including how to use ‘prosodyctl’ to auto-import certificates
– (from e.g. Let’s Encrypt) see Certificates – Prosody IM

– Location of directory to find certificates in (relative to main config file):
certificates = “certs”

– HTTPS currently only supports a single certificate, specify it here:
–https_certificate = “/etc/prosody/certs/localhost.crt”

----------- Virtual hosts -----------
– You need to add a VirtualHost entry for each domain you wish Prosody to serve.
– Settings under each VirtualHost entry apply only to that host.
– It’s customary to maintain VirtualHost entries in separate config files
– under /etc/prosody/conf.d/ directory. Examples of such config files can
– be found in /etc/prosody/conf.avail/ directory.

------ Additional config files ------
– For organizational purposes you may prefer to add VirtualHost and
– Component definitions in their own config files. This line includes
– all config files in /etc/prosody/conf.d/

VirtualHost “localhost”

–VirtualHost “example.com
– certificate = “/path/to/example.crt”

------ Components ------
– You can specify components to add hosts that provide special services,
– like multi-user conferences, and transports.
– For more information on components, see Components in Prosody – Prosody IM

—Set up a MUC (multi-user chat) room server on conference.example.com:
–Component “conference.example.com” “muc”
— Store MUC messages in an archive and allow users to access it
–modules_enabled = { “muc_mam” }

—Set up an external component (default component port is 5347)

– External components allow adding various services, such as gateways/
– transports to other networks like ICQ, MSN and Yahoo. For more info
– see: Components in Prosody – Prosody IM

–Component “gateway.example.com
– component_secret = “password”
Include “conf.d/*.cfg.lua”

TCP/5222 should be open on JMS

Yes, TCP 5222 is also open. The following inbound ports are open -

3478 UDP
5222 TCP
10000 UDP
5347 TCP
443 TCP
80 TCP
5349 TCP

Are you talking about JMS or JVB?

TCP/5222 should be open on JMS

JVB. The videobridge set up on 2nd server is what is throwing “Connection refused” exception while trying to connect to Jitsi/Prosody on 1st server

I meant TCP/5222 should be open on JMS (your 1st server), not on JVB

I understood, all these ports are open on the 1st server -
3478 UDP
5222 TCP
10000 UDP
5347 TCP
443 TCP
80 TCP
5349 TCP

I did the changes as I understood from that Youtube video. Config files were different in 1st server compared to what was stated the video, as you can see from what I pasted. Result is that I have 1st videobridge on 1st server (same server as Prosody etc) and 2nd videobridge on a separate instance. 1st videobridge is working, I am able to run a conference with updated configuration but 2nd server is throwing an exception “Connection refused”.

I have opened up all those ports in 1st server. Am I missing 5347 port configuration in some location ?

I checked running ports using netstat and do not find 5347 among the listed ports. I have provided port 5347 in prosody config. Did I miss some other entry ?

/etc/prosody/prosody.cfg.lua
component_ports = { 5347 }
component_interface =

I found a message from damencho in Feb '21 on another post - “Only prosody listens on those ports, it is not jicofo. On the next stable release 5347 will not be used so that can go away”

Does that mean prosody no longer listens on 5347 ?

I changed the port on 2nd server (Videobridge) to 443 and now that error is gone. This is the log -

JVB 2022-12-19 05:05:11.155 INFO: [28] org.jitsi.videobridge.IceUdpTransportManager.log() Initialized TCP harvester on port 443, using SSLTCP:true
JVB 2022-12-19 05:05:11.184 INFO: [28] org.jitsi.videobridge.health.Health.log() Performed a successful health check in 926ms. Sticky failure: false

I have a fundamental question now - to scale up conferences, is it enough to add just videobridges as stated in that Youtube video or has the architecture changed after that video was published ?

Can you share your second JVB’s /etc/jitsi/videobridge/sip-communicator.properties?

Don’t forget to mask privates.

I abandoned that approach and set up load balancing as outlined in this article - Install Jitsi Meet and configure load balancing - Samuel Nitsche

It worked fine. Now I need to find out how to determine current load of a videobridge so that I can use a limit to do auto-scale up. Please help with that.

If you calibrate the load-threshold value in your JVB to match the maximum packet rate it should handle, you can then use stress_level from colibri stats to infer current load and inform your scaling decisions.

Great, thank you, let me read and try to understand the approach