Videobridge2 startup errors

I had a working videobridge1 setup that I broke doing something stoooopid and then got distracted before I made a note of what I had changed.

So decided to upgrade to videobridge2 and see what happened.

As expected the same error occurs… no surprise there, but it confirms it is my configs and stupidity!

I have tried ever config item I can see (and there is a plethora of conflicting documentation and advice) but get a continual failure to start or run correctly.

I get the meet web page. All rooms are user/password protected with no anonymous access.

If I try and login I now get “Waiting for host, conference has not yet started” with “CONFERENCE FAILED: conference.authenticationRequired” in the console which I believe is because videobridge is not starting as we can see in the logs below.

Server is host : ispare.testdomain.net
Single public IP
Apache reverse proxy

jitsi-meet 2.0.4384-1
jitsi-meet-prosody 1.0.3969-1
jitsi-meet-web 1.0.3969-1
jitsi-meet-web-config 1.0.3969-1
jitsi-videobridge2 2.1-164-gfdce823f-1
jitsi-meet-prosody 1.0.3969-1
prosody 0.11.5-1~stretch6

Setups below.

Any advice greatly appreciated. I am pretty certain it is something pretty small I have missed.

jvb.log

2020-04-04 00:09:34.215 INFO: [11] AbstractJettyBundleActivator.start#613: Not starting the Jetty service for org.jitsi.videobridge.rest.PublicRESTBundleActivator(port=-1)
2020-04-04 00:09:34.216 INFO: [11] AbstractJettyBundleActivator.start#613: Not starting the Jetty service for org.jitsi.videobridge.rest.PublicClearPortRedirectBundleActivator(port=8080)
2020-04-04 00:09:34.262 WARNING: [11] UlimitCheck.printUlimits#111: Running with open files limit 4096 (hard 4096), thread limit 15736 (hard 15736). These values are too low and they will limit the number of participants that the bridge can serve simultaneously.
2020-04-04 00:09:34.264 INFO: [11] VideobridgeExpireThread.start#92: Starting with 60 second interval.
2020-04-04 00:09:34.268 WARNING: [11] Videobridge.start#918: No authorized source regexp configured. Will accept requests from any source.
2020-04-04 00:09:34.300 INFO: [17] Videobridge.createConference#326: create_conf, id=99bf081d6ad678ae gid=null logging=false
2020-04-04 00:09:34.339 INFO: [17] TaskPools.<clinit>#81: TaskPools detected 2 processors, creating the CPU pool with that many threads
2020-04-04 00:09:34.507 INFO: [11] JitsiConfig$Companion.reload#40: Reloading.
2020-04-04 00:09:34.513 INFO: [11] NewConfig$1.invoke#88: Loaded NewConfig with origin: merge of system properties,system properties,reference.conf @ jar:file:/usr/share/jitsi-videobridge/jitsi-videobridge.jar!/reference.conf: 1,reference.conf @ jar:file:/usr/share/jitsi-videobridge/lib/jitsi-media-transform-1.0-139-g33e0a49.jar!/reference.conf: 1
2020-04-04 00:09:34.513 INFO: [11] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-04-04 00:09:34.513 INFO: [11] LegacyConfigFileLoader$Companion.load#40: Attempting to load legacy config file at path /etc/jitsi, videobridge, sip-communicator.properties
2020-04-04 00:09:34.536 INFO: [11] OctoRelayService.start#62: Octo relay is disabled.
2020-04-04 00:09:34.553 SEVERE: [17] RecurringRunnableExecutor.run#230: The invocation of the method org.jitsi.videobridge.health.Health.run() threw an exception.
java.lang.NoSuchMethodError: org.ice4j.ice.Agent.<init>(Ljava/lang/String;Lorg/jitsi/utils/logging2/Logger;)V
	at org.jitsi.videobridge.ice.IceTransport.createIceAgent(IceTransport.java:252)
	at org.jitsi.videobridge.ice.IceTransport.<init>(IceTransport.java:157)
	at org.jitsi.videobridge.DtlsTransport.<init>(DtlsTransport.java:128)
	at org.jitsi.videobridge.Endpoint.<init>(Endpoint.java:269)
	at org.jitsi.videobridge.Endpoint.<init>(Endpoint.java:285)
	at org.jitsi.videobridge.Conference.createLocalEndpoint(Conference.java:692)
	at org.jitsi.videobridge.health.Health.check(Health.java:83)
	at org.jitsi.videobridge.health.Health.doCheck(Health.java:161)
	at org.jitsi.videobridge.health.Health.doRun(Health.java:266)
	at org.jitsi.utils.concurrent.PeriodicRunnableWithObject.run(PeriodicRunnableWithObject.java:87)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.run(RecurringRunnableExecutor.java:216)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.runInThread(RecurringRunnableExecutor.java:292)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.access$000(RecurringRunnableExecutor.java:36)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor$1.run(RecurringRunnableExecutor.java:328)
2020-04-04 00:09:34.642 INFO: [23] org.ice4j.ice.harvest.StunMappingCandidateHarvester.discover: Discovered public address 1.2.3.4:56275/udp from STUN server 52.47.138.178:443/udp using local address 1.2.3.4:0/udp
2020-04-04 00:09:34.642 INFO: [19] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Using org.ice4j.ice.harvest.StunMappingCandidateHarvester, face=/1.2.3.4, mask=/1.2.3.4
2020-04-04 00:09:34.642 INFO: [19] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Initialized mapping harvesters (delay=115ms).  stunDiscoveryFailed=false
2020-04-04 00:09:34.662 INFO: [26] ComponentBase.loadConfig#202: Component org.jitsi.videobridge. config: 
2020-04-04 00:09:34.662 INFO: [26] ComponentBase.loadConfig#203:   ping interval: 10000 ms
2020-04-04 00:09:34.662 INFO: [26] ComponentBase.loadConfig#204:   ping timeout: 5000 ms
2020-04-04 00:09:34.662 INFO: [26] ComponentBase.loadConfig#205:   ping threshold: 3
2020-04-04 00:09:34.840 INFO: [21] [hostname=localhost id=shard] MucClient$1.connected#266: Connected.
2020-04-04 00:09:34.840 INFO: [21] [hostname=localhost id=shard] MucClient.lambda$getConnectAndLoginCallable$7#648: Logging in.
2020-04-04 00:09:34.911 INFO: [21] [hostname=localhost id=shard] MucClient$MucWrapper.join#751: Joined MUC: jvbbrewery@internal.auth.ispare.testdomain.net
2020-04-04 00:09:44.554 INFO: [17] Videobridge.createConference#326: create_conf, id=3caaafc2e08a5d93 gid=null logging=false
2020-04-04 00:09:44.556 SEVERE: [17] RecurringRunnableExecutor.run#230: The invocation of the method org.jitsi.videobridge.health.Health.run() threw an exception.
java.lang.NoSuchMethodError: org.ice4j.ice.Agent.<init>(Ljava/lang/String;Lorg/jitsi/utils/logging2/Logger;)V
	at org.jitsi.videobridge.ice.IceTransport.createIceAgent(IceTransport.java:252)
	at org.jitsi.videobridge.ice.IceTransport.<init>(IceTransport.java:157)
	at org.jitsi.videobridge.DtlsTransport.<init>(DtlsTransport.java:128)
	at org.jitsi.videobridge.Endpoint.<init>(Endpoint.java:269)
	at org.jitsi.videobridge.Endpoint.<init>(Endpoint.java:285)
	at org.jitsi.videobridge.Conference.createLocalEndpoint(Conference.java:692)
	at org.jitsi.videobridge.health.Health.check(Health.java:83)
	at org.jitsi.videobridge.health.Health.doCheck(Health.java:161)
	at org.jitsi.videobridge.health.Health.doRun(Health.java:266)
	at org.jitsi.utils.concurrent.PeriodicRunnableWithObject.run(PeriodicRunnableWithObject.java:87)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.run(RecurringRunnableExecutor.java:216)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.runInThread(RecurringRunnableExecutor.java:292)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor.access$000(RecurringRunnableExecutor.java:36)
	at org.jitsi.utils.concurrent.RecurringRunnableExecutor$1.run(RecurringRunnableExecutor.java:328)

And repeat…

I have on occasion had another repeating failure when messing with configs like this:

2020-04-03 23:15:20.831 INFO: [17] Videobridge.createConference#326: create_conf, id=3e4c3a129f0054ce gid=null logging=false
2020-04-03 23:15:20.834 SEVERE: [17] RecurringRunnableExecutor.run#230: The invocation of the method org.jitsi.videobridge.health.Health.run() threw an exception.
java.lang.NoSuchMethodError: org.ice4j.ice.Agent.<init>(Ljava/lang/String;Lorg/jitsi/utils/logging2/Logger;)V
	at org.jitsi.videobridge.ice.IceTransport.createIceAgent(IceTransport.java:252)

videobridge conf

JVB_HOSTNAME=ispare.testdomain.net
JVB_PORT=5347
JVB_SECRET=9NKpWPtN
#JVB_OPTS="--apis=, "
JVB_OPTS="--apis=xmpp,rest,"
JAVA_SYS_PROPS=" -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"

sip-communicator.properties

org.jitsi.impl.neomedia.transform.srtp.SRTPCryptoContext.checkReplay=false
org.jitsi.videobridge.TCP_HARVESTER_PORT=4443
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.ispare.testdomain.net
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=9NKpWPtN
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.ispare.testdomain.net
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=f9a86dfc-2227-4631-95c2-cc3f0e8d794a

prodosody.cfg.lua

admins = { }
plugin_paths = { "/usr/lib/prosody/modules" }
modules_enabled = {
		"roster"; -- Allow users to have a roster. Recommended ;)
		"saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
		"tls"; -- Add support for secure TLS on c2s/s2s connections
		"dialback"; -- s2s dialback support
		"disco"; -- Service discovery
		"carbons"; -- Keep multiple clients in sync
		"pep"; -- Enables users to publish their avatar, mood, activity, playing music and more
		"private"; -- Private XML storage (for room bookmarks, etc.)
		"blocklist"; -- Allow users to block communications with other users
		"vcard4"; -- User profiles (stored in PEP)
		"vcard_legacy"; -- Conversion between legacy vCard and PEP Avatar, vcard
		"version"; -- Replies to server version requests
		"uptime"; -- Report how long server has been running
		"time"; -- Let others know the time here on this server
		"ping"; -- Replies to XMPP pings with pongs
		"register"; -- Allow users to register on this server using a client and change passwords
		"admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
}
modules_disabled = {
}
allow_registration = false
c2s_require_encryption = true
s2s_require_encryption = true
s2s_secure_auth = false
pidfile = "/var/run/prosody/prosody.pid"
authentication = "internal_hashed"
storage = "internal"
archive_expires_after = "1w" -- Remove archived messages after 1 week
log = {
	info = "/var/log/prosody/prosody.log"; -- Change 'info' to 'debug' for verbose logging
	error = "/var/log/prosody/prosody.err";
}
Include "conf.d/*.cfg.lua"

ispare.prodosody.cfg.lua

https_certificate = "/var/lib/prosody/ispare.testdomain.net.crt";
https_key =  "/var/lib/prosody/ispare.testdomain.net.key";
VirtualHost "ispare.testdomain.net"
        authentication = "internal_plain"
        ssl = {
                key = "/var/lib/prosody/ispare.testdomain.net.key";
                certificate = "/var/lib/prosody/ispare.testdomain.net.crt";
        }
        modules_enabled = {
            "bosh";
            "pubsub";
            "ping"; -- Enable mod_ping
        }
        c2s_require_encryption = false
VirtualHost "auth.ispare.testdomain.net"
        authentication = "internal_plain"
        ssl = {
                key = "/var/lib/prosody/auth.ispare.testdomain.net.key";
                certificate = "/var/lib/prosody/auth.ispare.testdomain.net.crt";
        }
admins = { "focus@auth.ispare.testdomain.net" }
Component "conference.ispare.testdomain.net" "muc"
Component "jitsi-videobridge.ispare.testdomain.net"
    component_secret = "9NKpWPtN"
Component "focus.ispare.testdomain.net"
    component_secret = "VcrolK4W"
Component "internal.auth.ispare.testdomain.net" "muc"
    storage = "memory"
    modules_enabled = { "ping"; }
    admins = { "focus@auth.ispare.testdomain.net", "jvb@auth.ispare.testdomain.net" }

prosody log:

Apr 03 22:44:28 c2s5651420e8ad0	info	Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Apr 03 22:44:28 c2s5651420e8ad0	info	Authenticated as jvb@auth.ispare.testdomain.net
Apr 03 22:44:43 jitsi-videobridge.ispare.testdomain.net:component	warn	Component not connected, bouncing error for: <iq to='jitsi-videobridge.ispare.testdomain.net' id='4KDpJ-4398' from='focus@auth.ispare.testdomain.net/focus11017025851' type='get'>

Attempt to login

Apr 03 22:50:39 mod_bosh	info	New BOSH session, assigned it sid '3a072758-2a21-460b-be90-f9e879485a7e'
Apr 03 22:50:40 bosh3a072758-2a21-460b-be90-f9e879485a7e	info	Authenticated as impamark@ispare.testdomain.net
Apr 03 22:50:43 jitsi-videobridge.ispare.testdomain.net:component	warn	Component not connected, bouncing error for: <iq to='jitsi-videobridge.ispare.testdomain.net' id='4KDpJ-4593' from='focus@auth.ispare.testdomain.net/focus11017025851' type='get'>
Apr 03 22:50:57 mod_bosh	info	New BOSH session, assigned it sid '05931552-94f0-4e77-a245-dec82e932e49'
Apr 03 22:50:58 bosh05931552-94f0-4e77-a245-dec82e932e49	info	BOSH client disconnected: session close

Well, here’s some fun.

I decided to rip the lot out and start over. That certainly cleared up a bit of a mess.

However, the new installer is pretty brutal and makes no allowances for saving existing configs so it entirely trampled my existing apache reverse proxy setup without a “by your leave”. I had saved my jitsi setups but forgot apache - I had no idea that JITSI install would trash an APACHE file.

How on earth you can make the assumption that there is nothing of interest that I might want in a file is beyond me. You should AT LEAST make a backup copy if an existing conf is found…

That is just plain amateurish. Sorry.

Finally just about got it running again.

Had to sort out the certificate permissions (still a nightmare) and ‘storage’ errors

“error Error restoring room myroom@conference.mydomain.com from storage: no data storage active”

Seems storage is set to none? Changed it to internal and that shut it up.

storage = "internal"

But have finally got at least tow people connected.