Testing Octo Setup

Yes. I have 3 bridges running and connected/detected by jicofo, and even though SplitBridgeSelectionStrategy is set, participants still only join a single bridge.

I seem to have figured out the error. Looking at my JVB logs I discovered octo relay was failing to start. Apparently using my public IP in the bind_address was causing issues, even though port 4096 was open. I changed my bind_address to 127.0.0.1 and octo was tested successfully. Hopefully using the local IP as the bind_address does not cause issues.

1 Like

I have the same behaviour, participants always join on the same jvb… Can you please check the config files, what may be wrong, to have OCTO enabled ? i want to test each participant to join to different jvb with SplitBridgeSelectionStrategy… Installation is on aws ec2 c5.xlarge servers with security group all traffic allowed on each…
When i check with tcpdump -i any src port 10000 always same jvb has traffic, it is not distributed and in stats i always see octo_conferences = 0, but region is correct = “region1”

Jitsi-Meet Server settings

/etc/jitsi/jicofo/sip-communicator.properties

org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.meet1.mydomain.com
org.jitsi.jicofo.BridgeSelector.BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy
#org.jitsi.jicofo.BridgeSelector.BRIDGE_SELECTION_STRATEGY=RegionBasedBridgeSelectionStrategy

/etc/jitsi/videobridge/sip-communicator.properties

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc,colibri
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.meet1.mydomain.com
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=Ia1Nvvrh
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.meet1.mydomain.com
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jvb1
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true
org.jitsi.videobridge.octo.BIND_ADDRESS=172.31.36.162
org.jitsi.videobridge.octo.PUBLIC_ADDRESS=publicIP
org.jitsi.videobridge.octo.BIND_PORT=4096
org.jitsi.videobridge.REGION=region1
#org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=172.31.36.162
#org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=publicIP

“/etc/jitsi/meet/meet1.mydomain.com-config.js”

testing: {
    // Enables experimental simulcast support on Firefox.
    enableFirefoxSimulcast: false,

    // P2P test mode disables automatic switching to P2P when there are 2
    // participants in the conference.
    p2pTestMode: false,

    octo: {
    probability: 1
}

    // Enables the test specific features consumed by jitsi-meet-torture
    // testMode: false

    // Disables the auto-play behavior of *all* newly created video element.
    // This is useful when the client runs on a host with limited resources.
    // noAutoPlayVideo: false
},

Second JVB

/etc/jitsi/videobridge/sip-communicator.properties

org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc,colibri
org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=meet1.mydomain.com
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=auth.meet1.mydomain.com
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=Ia1Nvvrh
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@internal.auth.meet1.mydomain.com
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=jvb2
org.jitsi.videobridge.xmpp.user.shard.DISABLE_CERTIFICATE_VERIFICATION=true
org.jitsi.videobridge.octo.BIND_ADDRESS=172.31.1.115
org.jitsi.videobridge.octo.PUBLIC_ADDRESS=publicIP
org.jitsi.videobridge.octo.BIND_PORT=4096
org.jitsi.videobridge.REGION=region1
#org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=172.31.1.115
#org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=publicIP

deploymentInfo: {
    shard: "shard",
    region: "region1",
    userRegion: "region1"
},

In Jicofo logs i see
Jicofo 2020-05-06 11:54:27.772 INFO: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Using org.jitsi.jicofo.bridge.SingleBridgeSelectionStrategy
Jicofo 2020-05-06 11:54:27.772 INFO: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Bridge failure reset threshold: 300000
Jicofo 2020-05-06 11:54:27.772 INFO: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Local region: null
Jicofo 2020-05-06 11:54:28.270 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meet1.mydomain.com/jvb1 v: null
Jicofo 2020-05-06 11:54:28.270 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet1.mydomain.com/jvb1
Jicofo 2020-05-06 11:54:28.274 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meet1.mydomain.com/jvb2 v: null
Jicofo 2020-05-06 11:54:28.274 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet1.mydomain.com/jvb2
Jicofo 2020-05-06 11:54:28.276 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meet1.mydomain.com/jvb3 v: null
Jicofo 2020-05-06 11:54:28.276 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet1.mydomain.com/jvb3
Jicofo 2020-05-06 11:54:28.622 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.BridgeSelector.BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy
Jicofo 2020-05-06 11:54:32.068 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Removing JVB: jvbbrewery@internal.auth.meet1.mydomain.com/jvb1
Jicofo 2020-05-06 11:54:38.180 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meet1.mydomain.com/jvb1 v: null
Jicofo 2020-05-06 11:54:38.180 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet1.mydomain.com/jvb1
Jicofo 2020-05-06 11:56:17.160 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Removing JVB: jvbbrewery@internal.auth.meet1.mydomain.com/jvb1
Jicofo 2020-05-06 11:56:20.186 SEVERE: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Failed to find class for: SplitBridgeSelectionStrategy
at org.jitsi.jicofo.bridge.BridgeSelector.createBridgeSelectionStrategy(BridgeSelector.java:191)
at org.jitsi.jicofo.bridge.BridgeSelector.(BridgeSelector.java:154)
Jicofo 2020-05-06 11:56:20.187 INFO: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Using org.jitsi.jicofo.bridge.SingleBridgeSelectionStrategy
Jicofo 2020-05-06 11:56:20.187 INFO: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Bridge failure reset threshold: 300000
Jicofo 2020-05-06 11:56:20.187 INFO: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Local region: null
Jicofo 2020-05-06 11:56:20.828 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meet1.mydomain.com/jvb2 v: null
Jicofo 2020-05-06 11:56:20.828 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet1.mydomain.com/jvb2
Jicofo 2020-05-06 11:56:20.831 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meet1.mydomain.com/jvb3 v: null
Jicofo 2020-05-06 11:56:20.831 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet1.mydomain.com/jvb3
Jicofo 2020-05-06 11:56:21.165 INFO: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.BridgeSelector.BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy
Jicofo 2020-05-06 11:56:23.264 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added videobridge: jvbbrewery@internal.auth.meet1.mydomain.com/jvb1 v: null
Jicofo 2020-05-06 11:56:23.264 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet1.mydomain.com/jvb1

jicofo.txt (19.0 KB)

Tnx

can you post your prosody logs?
Were you able to register the jvbs in prosody ?
you might also want to edit your txt file. your public ip is showing.

I can, actually MUC works without issues… load can be distributed between jvbs as room based… but OCTO doesn’t seem to be working, participants join room at the same jvb.

I’m also facing the same issue like yours. Can you please have a look. Need to test that whether it's working or not after enabling OCTO . Thank you very much

Any advice?

@omesta I see this in your logs “Jicofo 2020-05-06 11:56:20.186 SEVERE: [11] org.jitsi.jicofo.bridge.BridgeSelector.log() Failed to find class for: SplitBridgeSelectionStrategy”. This makes me think you somehow don’t have this class/feature available. How old is your jicofo installation? Are you sure it is up to date?

Not the latest but march 2020 releases
dpkg -s jicofo | grep Version
Version: 1.0-539-1
dpkg -s jitsi-videobridge2 | grep Version
Version: 2.1-192-g06b3dfe5-1

Hi, your config looks very similar to ours - I noticed that both JVBs have the same region though:
org.jitsi.videobridge.REGION=region1 . If that’s actually the case then maybe giving them unique region names would help.

Also, our meet config.js doesn’t have the

    shard: "shard",
    region: "region1",
    userRegion: "region1"

Though I’m not sure if that’s relevant, or even correct, since Octo isn’t quite working right on my setup.

I set same region for both jvbs because in a weekly call it is said that it can be used for better load balancing in a single region…
also i don’t have any geo regions…

I can remove these shard, region, user region data, i am sometimes adding them sometimes removing, for test purposes.

I see, I hadn’t heard that meeting - I might try it too.
But with the SplitBridge selection strategy setup, it should randomly pick a region without needing the geo regions. That’s how I’ve been testing it at least.
In conferences it shows two servers and users get put in regions 1 or 2 randomly. the problem is that it has low-res video for some reason.

makes sense, i will try that thank you; may be it is splitting between regions not jvbs as you suggested…

and what about statistics, have you enabled colibri/stats ? can you see octo details there?

Colibri/stats is showing Octo details with our setup:

,0,0,0,0,0,0],“videostreams”:8,“jitter_aggregate”:4.839777818999097,“total_ice_succeeded_tcp”:0,“octo_endpoints”:2,“current_timestamp”:“2020-05-12 16:25:13.372”,“total_packets_dropped_octo”:2870,“conferences”:1,“participants”:4,“largest_conference”:4,“total_packets_sent”:640944,“total_data_channel_messages_sent”:0,“total_bytes_received_octo”:922637470,“octo_send_packet_rate”:724,“conferences_by_audio_senders”:[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],“total_conferences_created”:13,“total_ice_failed”:0,“threads”:55,“videochannels”:2,“total_packets_received_octo”:973546

somehow i don’t see them

I have set region1, region2, region3 to my jvbs but still unable to distribute participants… room is always on a single jvb and all participants go there… no octo related stats in colibri… the configs are quite straight forward and i can’t understand why this is not working at all…

1 Like

Hi @omesta , can you upgrade to actual versions? My octo setup is working, but I’m on latest builds from unstable repo…

Milan

Hi @bbaldino @damencho , I’ve setup octo on our installation with two jvb configured with the same region and splitbridge strategy, but have some questions:

  1. What can cause Octo droped packets? One server had jitsi_total_packets_dropped_octo: 32273
    ans second: jitsi_total_packets_dropped_octo: 58939.

  2. I see some jitsi_total_failed_conferences (15 and 13) on our servers after I’ve enabled octo, before this I had not single one ever, and testing jitsi for 2 months now. Is this to be expected with octo?

  3. I’m using /unstable repo. on my debian install, from what branch on github are those build? Master?

Thank you very much,

kind regards,

Milan

This stat is the aggregate of a few things, you can find the code for the stat here.

octoUdpTransportStats.incomingPacketsDropped happens if the UdpTransport doesn’t have a handler for incoming packets. This is unlikely to be the cause because when the bridge starts up Octo we plug the handler in immediately.

octoTransportStats.numInvalidPackets happens if the received Octo packet is invalid. There’s are warnings logged if this happens here and here.

octoTransportStats.numIncomingDroppedNoHandler happens if an Octo packet is received but no handler for Octo packets for that conference has been installed. This is the most likely case and also has a warning logged here.

If it is indeed that last one, it could be a bit of a race in signaling: packets forwarded to a bridge before the conference has been created there (or, possibly, for a bit after a conference was deleted). If so, that isn’t the biggest deal. Are you actually seeing issues with the meeting? Or just noticed the stat?

Hi @bbaldino, thank you for your analysis, we run octo for very short time now less than 24 hours, no problems were reported still by users, but it is early to say it is OK. I’m watching stats and this stands out for me. I’ll enable logging and will post here later with results.

What can cause jitsi_total_failed_conferences numbers increasing that I didn’t seen before? Is it related to potential signaling problem you mentioned?

Actual stats:

root@BackupStorage:~/scripts# ./stats.sh
JVB stats:
jitsi_bit_rate_upload: 42217
jitsi_bit_rate_download: 12299
jitsi_participants: 113
jitsi_conferences: 14
jitsi_largest_conference: 26
jitsi_endpoints_sending_video: 15
jitsi_endpoints_sending_audio: 10
jitsi_receive_only_endpoints: 40
jitsi_threads: 289
jitsi_total_ice_failed: 74
jitsi_total_failed_conferences: 33
jitsi_total_partially_failed_conferences: 6
Octo stats:
jitsi_octo_send_bitrate: 9115488
jitsi_octo_receive_bitrate: 12011910
jitsi_octo_send_packet_rate: 1619
jitsi_octo_receive_packet_rate: 2004
jitsi_octo_conferences: 14
jitsi_octo_endpoints: 57
jitsi_total_packets_sent_octo: 26360938
jitsi_total_packets_received_octo: 25135406
jitsi_total_packets_dropped_octo: 143148

Zatazenie: 09:33:52 up 32 days, 21:43, 3 users, load average: 2.75, 2.83, 2.54
Netstat: 153656 receive buffer errors 19 send buffer errors

and:

root@virt1:~/scripts# ./stats.sh
JVB stats:
jitsi_bit_rate_upload: 45910
jitsi_bit_rate_download: 13795
jitsi_participants: 116
jitsi_conferences: 14
jitsi_largest_conference: 29
jitsi_endpoints_sending_video: 15
jitsi_endpoints_sending_audio: 10
jitsi_receive_only_endpoints: 43
jitsi_threads: 321
jitsi_total_ice_failed: 75
jitsi_total_failed_conferences: 20
jitsi_total_partially_failed_conferences: 9
Octo stats:
jitsi_octo_send_bitrate: 12026028
jitsi_octo_receive_bitrate: 9052414
jitsi_octo_send_packet_rate: 2012
jitsi_octo_receive_packet_rate: 1613
jitsi_octo_conferences: 14
jitsi_octo_endpoints: 55
jitsi_total_packets_sent_octo: 25116933
jitsi_total_packets_received_octo: 26346128
jitsi_total_packets_dropped_octo: 86385

Zatazenie: 09:33:44 up 41 days, 11:39, 3 users, load average: 2.02, 1.84, 1.51
Netstat: 155661 receive buffer errors 41 send buffer errors

Thank you very much!

Milan