JVB spams log with Stack Trace

Hi,

Our JVB2 is spamming the jvb.log file with messages like this one below. Does anyone have an idea were this comes from ? We are using JVB2 version 2.1-308-g6cb64128-1 from the unstable repo

    Exception in thread "Global IO pool-31346" java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.deser.std.JdkDeserializers
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1852)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:167)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:131)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:476)
        at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4389)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4198)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.ja
va:3205)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3188)
        at org.jitsi.videobridge.message.BridgeChannelMessage$Companion.parse(BridgeChannelMessage.kt:373)
        at org.jitsi.videobridge.message.BridgeChannelMessage.parse(BridgeChannelMessage.kt)
        at org.jitsi.videobridge.AbstractEndpointMessageTransport.onMessage(AbstractEndpointMessageTransport.java:75)
        at org.jitsi.videobridge.EndpointMessageTransport.onDataChannelMessage(EndpointMessageTransport.java:192)
        at org.jitsi.videobridge.datachannel.DataChannel.onIncomingMsg(DataChannel.java:131)
        at org.jitsi.videobridge.datachannel.DataChannelStack.onIncomingDataChannelPacket(DataChannelStack.java:84)
        at org.jitsi.videobridge.Endpoint$DataChannelHandler.consume(Endpoint.java:1465)
        at org.jitsi.videobridge.Endpoint.lambda$null$6(Endpoint.java:888)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Which java version is used by jvb?

Hi @damencho, We are using

openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~18.04-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

@bbaldino @Boris_Grozev @Jonathan_Lennox Any idea? This is the stable jvb we released few days ago …

Strange, I don’t think I’ve ever seen this. I’ll see if I can repo. @plasma any specific steps to reproduce?

@bbaldino Nothing special configured here. We have to separate JVB’s and the config looks like this

config JVB1:

# Jitsi Videobridge settings

# sets the XMPP domain (default: none)
JVB_HOSTNAME=meet.xyz.de

# sets the hostname of the XMPP server (default: domain if set, localhost otherwise)
JVB_HOST=host1.domain.de

# sets the port of the XMPP server (default: 5275)
JVB_PORT=5347

# sets the shared secret used to authenticate to the XMPP server
JVB_SECRET=abc

# extra options to pass to the JVB daemon
JVB_OPTS="--apis=rest"


# adds java system props that are passed to jvb (default are for home and logging config file)
JAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties" 

sip-communicator.properties JVB1:

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=host1.domain.de
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.meet.xyz.de
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=abc
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.meet.xyz.de
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=c4e817da-54b1-40b1-9ec8-1307bbda1311
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=10.5.2.141
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=<external ip redacted>
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true

config JVB2:

# Jitsi Videobridge settings

# sets the XMPP domain (default: none)
JVB_HOSTNAME=meet.xyz.de

# sets the hostname of the XMPP server (default: domain if set, localhost otherwise)
JVB_HOST=host1.domain.de

# sets the port of the XMPP server (default: 5275)
JVB_PORT=5347

# sets the shared secret used to authenticate to the XMPP server
JVB_SECRET=abc

# extra options to pass to the JVB daemon
JVB_OPTS="--apis=rest"


# adds java system props that are passed to jvb (default are for home and logging config file)
JAVA_SYS_PROPS="-Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.config.file=/etc/jitsi/videobridge/logging.properties"

sip-communicator.properties JVB2:

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
#org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=host1.domain.de
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.meet.xyz.de
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=abc
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.meet.xyz.de
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=efbafe92-e9be-452b-8891-968b91d2ac8a
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=10.5.2.151
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=<external ip redacted>
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true

I wasn’t able to repro. This is just doing a normal upgrade via apt?

Yes. I ran a normal apt upgrade to get the latest version

I dunno…maybe it’s getting some incorrect version of that jar? It looks like we don’t use a fat jar for the debian package.

Should I purge and reinstall the jvb ?

That would be worth a shot. If that doesn’t work would you be able to build the JVB from source so we can test that? I can walk you through it.

@bbaldino I purged the JVB on one server and reinstalled it but as soon as a meeting starts on that videobridge it spits out the stack traces again. They are fewer than before but still there.

Also I noticed that with the latest version the video quality is very poor. Does this have something to do with this message ?

2020-09-07 12:30:19.013 INFO: [17] JvbLoadManager.loadUpdate#50: Load measurement RTP packet rate (up + down) of 730 pps is below threshold of RTP packet rate (up + down) of 50000 pps, maybe running recovery
2020-09-07 12:30:19.013 INFO: [17] JvbLoadManager.maybeRun#68: Load reducer started running PT20S ago, and we wait PT1M between runs, so will skip running recovery

No, that message is unrelated. We’re looking into the stack trace problem.

We’ve got the same problem over here (Debian 10 with openjdk-11-jre-headless). As this happens for the data channel, we tried to switch to ‘websocket’ which resulted in the stack trace popping up in the browser’s Javascript console instead of the jvb log.

Long story short: this results in broken client <-> bridge communication, which leaves us stuck at low definition video and breaks screen-sharing in Chrome/Chromium (fun fact: in Firefox, screen-sharing works anyways).

Sorry, forgot to update here but the stacktrace issue has been fixed in master

1 Like

Great news, thank you. ETA for an updated Debian package? :slight_smile:

1 Like

It’s in jitsi-videobridge2_2.1-318-gcc20ad4b in unstable, but not sure when the next stable package will be.

1 Like

Thank you!

To add to that:

When using Jitsi on Debian 10 (videobridge version 2.1-304-g8488f77d-1), there seem to be too many libraries installed, the fix is here:

Afterwards, our problems with low definition video were gone.

Any ETA when we can expect that fix in the Debian stable repo? We are having this issue with jitsi-videobridge 2:2.1-304-g8488f77d-1 and we consider to roll back.