Working Multi Jitsi-meet / multi Videobridge setup

As mentioned over on the users discussion, we have shared our Ansible playbooks that build or https://switch.ch/meet service.

Here is the repository: https://github.com/switch-ch/jitsi-deploy

This is NOT Plug and Play - but it should be easy enough to extract a working configuration:

Features:

  • Multiple Front-end Jitsi-Meet instances (in our case, one per university)
  • Pool of shared video-bridges (all front end instances share the same pool of videobridges) using the new MUC feature
  • Nginx / Latest nightly version of Nginx
  • Shibboleth Authorisation (but this currently only works with older (August 2019) version of Jitsi/Jicofo)

Enjoy & hopefully this helps you getting a working, scaled installation going.

Pull requests & improvements more than welcome.

/jc

6 Likes

Looks great! We are a german education provider for schools. Could you please share some more information about the configuration of videobridges and the hardware? Also information about the maximal members per room would be interesting. We are looking for a good configuration to support 20 classes with 30 person in each conference.

Greetings
Benjamin Ledel

We currently have 32 videobridges with 8VCPUs and 8GB RAM each. I have seen around 1000 conferences / day with up to 40 people in them. Our setup is probably overkill right now. I would guess that one our bridges can handle 4-5 of 20 person conferences - but I haven’t got hard data on that yet.

2 Likes

We currently have 2 videobridges as dedicated servers

• Videobridge 1 (Xeon 16 Cores, 64 GB RAM), 1 Gbps
• Videobrdige 2 (Xeon 24 Cores, 196 GB RAM), 10 Gbps

Do you recommend to split the dedicated servers up?

1 Like

I really can’t say, as we don’t employ physical servers. I would assume that they are able to handle the load.

Many thanks to you switch folks!

We just released our terraform + ansible setup based on your great work. Ours is geared for deployment on AWS, not openstack.

Also we do not have the requirement of running multiple jitsi-meet instances like ya’ll did, though one certainly could.

The biggest change we’ve made is allow for JVBs running in different regions, and using octo to route participants to the closest videobridge.

  • one jitsi meet instance running nginx
  • multiple JVBs in different AWS regions
  • nginx using the geoip2 module and SSI to serve the config js with the appropriate region
  • privacy focused - logs are kept to a minimum, no ips logged, etc.

We’ve released it on gitlab:

Next steps are:

  • adding TURN/STUN to the deployment (currently it’s using the public jitsi org’s server)
  • adding monitoring via karrieretutor/jitsi-prom-exporter - we’ve no budget for callstats.io, so we’ll approximate it using grafana and prometheus
4 Likes

Great to see that this is sparking further development!

We just got STUN/TURN with our own servers to work - check out https://github.com/switch-ch/jitsi-deploy/tree/master/ansible/roles/coturn etc.

3 Likes

Hey guys, thanks for the great work and opening this up to the community.

I was so free and added your links and descriptions to the Jitsi-Meet Wiki at https://github.com/jitsi/jitsi-meet/wiki - hope that’s okay if not, let me know and I’ll remove it, of course.

2 Likes

That is awesome, thanks for putting this together!

2 Likes

Hi Jcfishcher,

I have been trying to set up multi videobridges by following the instruction but never success, would you please help.

I would like to help our school in Thailand under COVID-19 situation. we may have end up at least 10,000 students to join the vdo conferencing.

Please help

Hi kitti

I would need more information to help you. Do you have a working single server install already? The instructions assume that you know how to configure and run a simple server and don’t describe everything in detail (because that depends on your servers and your installation)

cheers
jc

Hi jcfischer

Thanks for your reply.

Yes I have single server installed and it works ok.

I follow the instruction as the following link:

JMS: https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md

JVB https://github.com/jitsi/jitsi-meet/wiki/jitsi-meet-load-balancing-installation-Ubuntu-18.04-with-MUC-and-JID

I did test by stop jitsi-vediobridge2 at JMS and leave jitsi-vediobridge2 at JVB to run but the screen will ask me to rejoin on the second participant.

I can provide the access to our JMS and JVS (test server)

Please help

Currently i have one Jitsi Meet Instance on a vServer.

What i have to do, to install a second instance (ore more) for another subdomain on this vServer?

Thank you

Hi MarkKnochen,

I have tried several times (more then a week) but without luck, I am not really know how to do it. I hope @jcfischer can help me on this and share on his experiences.

Kitti P

Sorry, I don’t have the time to troubleshoot your installation remotely.

What is /var/log/jitsi/jicofo.log and /var/log/prosody/prosody.log telling you when a second person tries to enter the conference? What are the log files on the videobridges showing. In my experience it is very clear what is wrong when you look at the log files.

From jicofo
Jicofo 2020-04-21 19:39:54.205 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.012S. Sticky failure: false
Jicofo 2020-04-21 19:40:04.203 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.008S. Sticky failure: false
Jicofo 2020-04-21 19:40:14.205 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.011S. Sticky failure: false
Jicofo 2020-04-21 19:40:24.204 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.01S. Sticky failure: false
Jicofo 2020-04-21 19:40:34.206 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.01S. Sticky failure: false
Jicofo 2020-04-21 19:40:44.207 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.012S. Sticky failure: false
Jicofo 2020-04-21 19:40:54.206 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.011S. Sticky failure: false
Jicofo 2020-04-21 19:41:04.204 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.009S. Sticky failure: false
Jicofo 2020-04-21 19:41:14.206 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.011S. Sticky failure: false
Jicofo 2020-04-21 19:41:24.210 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.015S. Sticky failure: false
Jicofo 2020-04-21 19:41:34.205 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.01S. Sticky failure: false
Jicofo 2020-04-21 19:41:43.633 INFO: [45] org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq().401 Focus request for room: test01@conference.me$
Jicofo 2020-04-21 19:41:43.913 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=Memb$
Jicofo 2020-04-21 19:41:43.919 INFO: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test01@conference.meet99.thebrightgroupthailand.com/ Jicofo 2020-04-21 19:41:43.920 WARNING: [27] org.jitsi.jicofo.bridge.BridgeSelectionStrategy.log() Failed to select initial bridge for participantRe
Jicofo 2020-04-21 19:41:43.921 SEVERE: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant – no bridge available.
Jicofo 2020-04-21 19:41:43.923 WARNING: [27] org.jitsi.jicofo.bridge.BridgeSelectionStrategy.log() Failed to select initial bridge for participantRe$
Jicofo 2020-04-21 19:41:43.923 SEVERE: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Can not invite participant – no bridge available.
Jicofo 2020-04-21 19:41:44.206 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.011S. Sticky failure: false
Jicofo 2020-04-21 19:41:44.521 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=Memb$
Jicofo 2020-04-21 19:41:44.521 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Owner has left the room !
Jicofo 2020-04-21 19:41:44.523 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Granted owner to test01@conference.meet99.thebrightgrouptha$
Jicofo 2020-04-21 19:41:44.523 INFO: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test01@conference.meet99.thebrightgroupthailand.com/ Jicofo 2020-04-21 19:41:44.523 INFO: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, test01@conference.meet99.thebrig
Jicofo 2020-04-21 19:41:44.724 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=Memb$
Jicofo 2020-04-21 19:41:44.724 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Owner has left the room !
Jicofo 2020-04-21 19:41:44.724 INFO: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test01@conference.meet99.thebrightgroupthailand.com/ Jicofo 2020-04-21 19:41:44.724 INFO: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Removed participant: true, test01@conference.meet99.thebrig
Jicofo 2020-04-21 19:41:44.725 INFO: [27] org.jitsi.jicofo.FocusManager.log() Disposed conference for room: test01@conference.meet99.thebrightgroupt$
Jicofo 2020-04-21 19:41:54.203 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.008S. Sticky failure: false
Jicofo 2020-04-21 19:42:04.205 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.01S. Sticky failure: false
Jicofo 2020-04-21 19:42:06.289 INFO: [63] org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq().401 Focus request for room: test01@conference.me$
Jicofo 2020-04-21 19:42:06.290 INFO: [63] org.jitsi.jicofo.FocusManager.log() Created new focus for test01@conference.meet99.thebrightgroupthailand. Jicofo 2020-04-21 19:42:06.290 INFO: [63] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Joining the room: test01@conference.meet99.thebrightgroupth
Jicofo 2020-04-21 19:42:06.426 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Chat room event ChatRoomMemberPresenceChangeEvent[type=Memb$
Jicofo 2020-04-21 19:42:06.429 INFO: [27] org.jitsi.jicofo.ChatRoomRoleAndPresence.log() Granted owner to test01@conference.meet99.thebrightgrouptha$
Jicofo 2020-04-21 19:42:06.429 INFO: [27] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Member test01@conference.meet99.thebrightgroupthailand.com/$
Jicofo 2020-04-21 19:42:14.209 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.013S. Sticky failure: false
Jicofo 2020-04-21 19:42:24.209 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.013S. Sticky failure: false
Jicofo 2020-04-21 19:42:34.202 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.006S. Sticky failure: false
Jicofo 2020-04-21 19:42:44.206 INFO: [32] org.jitsi.jicofo.health.Health.log() Performed a successful health check in PT0.01S. Sticky failure: false

From prosody.log

Apr 21 19:41:42 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper warn Session filters applied
Apr 21 19:41:42 mod_bosh info New BOSH session, assigned it sid ‘f92ee087-e88f-42f4-9844-cff899da7342’
Apr 21 19:41:42 boshf92ee087-e88f-42f4-9844-cff899da7342 info Authenticated as 3f993be5-24d4-4418-b106-741e0e25def6@meet99.thebrightgroupt$
Apr 21 19:41:44 bosh198b9e77-1189-42ad-a326-690e25b338e7 info BOSH client disconnected
Apr 21 19:41:47 boshf92ee087-e88f-42f4-9844-cff899da7342 info BOSH client disconnected
Apr 21 19:42:04 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper warn Session filters applied
Apr 21 19:42:04 mod_bosh info New BOSH session, assigned it sid ‘cba5dbf0-135a-4224-83b8-852604b78401’
Apr 21 19:42:04 boshcba5dbf0-135a-4224-83b8-852604b78401 info Authenticated as e0963c11-bec7-41e7-82f8-f860dcd05233@meet99.thebrightgroupt$

Please help

Here is your problem.

Do you see the bridge register in prosody / jicofo when it starts?

Where can I find the bridge register? Please guide me.

Sorry that I have so many questions because I am not really 100% technical person

when prosody starts

Apr 21 20:11:57 general info Hello and welcome to Prosody version 0.10.0
Apr 21 20:11:57 general info Prosody is using the select backend for connection handling
Apr 21 20:11:57 portmanager info Activated service ‘s2s’ on [::]:5269, []:5269
Apr 21 20:11:57 portmanager info Activated service ‘c2s’ on [::]:5222, [
]:5222
Apr 21 20:11:57 portmanager info Activated service ‘legacy_ssl’ on no ports
Apr 21 20:11:57 mod_posix info Prosody is about to detach from the console, disabling further console output
Apr 21 20:11:57 mod_posix info Successfully daemonized to PID 27109
Apr 21 20:11:57 portmanager info Activated service ‘http’ on [::]:5280, :5280
Apr 21 20:11:57 portmanager error Error binding encrypted port for https: No key present in SSL/TLS configuration for https port 5281
Apr 21 20:11:57 portmanager error Error binding encrypted port for https: No key present in SSL/TLS configuration for https port 5281
Apr 21 20:11:57 portmanager info Activated service ‘https’ on no ports
Apr 21 20:11:57 conferenceduration.meet99.thebrightgroupthailand.com:conference_duration_component warn conference duration will not work wi$
Apr 21 20:11:57 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host internal.auth.meet99. Apr 21 20:11:57 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host meet99.thebrightgroup
Apr 21 20:11:57 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host conferenceduration.me$
Apr 21 20:11:57 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host auth.meet99.thebright$
Apr 21 20:11:57 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host conference.meet99.the$
Apr 21 20:11:58 portmanager info Activated service ‘component’ on [127.0.0.1]:5347, [::1]:5347
Apr 21 20:11:58 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host focus.meet99.thebrigh$
Apr 21 20:11:58 speakerstats.meet99.thebrightgroupthailand.com:speakerstats_component warn speaker stats will not work with Prosody version 0.1$
Apr 21 20:11:58 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host speakerstats.meet99.t$
Apr 21 20:11:58 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper info Loading mod_muc_domain_mapper for host localhost!
Apr 21 20:12:08 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper warn Session filters applied
Apr 21 20:12:08 c2s5598fdf83170 info Client connected
Apr 21 20:12:09 c2s5598fdf83170 info Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Apr 21 20:12:10 c2s5598fdf83170 info Authenticated as focus@auth.meet99.thebrightgroupthailand.com
Apr 21 20:12:10 focus.meet99.thebrightgroupthailand.com:component warn Component not connected, bouncing error for: <iq id=‘tUINU-41’ type= Apr 21 20:12:12 jcp5598fdfe4d20 info Incoming Jabber component connection Apr 21 20:12:12 focus.meet99.thebrightgroupthailand.com:component info External component successfully authenticated Apr 21 20:12:12 conference.meet99.thebrightgroupthailand.com:muc_domain_mapper warn Session filters applied Apr 21 20:12:12 mod_bosh info New BOSH session, assigned it sid 'd76fbcc0-1992-4a90-845b-fd6d63994a5f' Apr 21 20:12:13 boshd76fbcc0-1992-4a90-845b-fd6d63994a5f info Authenticated as 6d1411c9-e7b3-41f9-9376-c16f7d56b172@meet99.thebrightgroupt

Check your videobridge log. It should read something like:

2020-04-21 15:34:34.369 WARNING: [32] MucClient.createXMPPTCPConnectionConfiguration#115: Disabling certificate verification!
2020-04-21 15:34:34.544 INFO: [32] [hostname=meet.example.com id=shard-1] MucClient$1.connected#266: Connected.
2020-04-21 15:34:34.545 INFO: [32] [hostname=meet.example.com id=shard-1] MucClient.lambda$getConnectAndLoginCallable$7#648: Logging in.
2020-04-21 15:34:34.680 INFO: [32] [hostname=meet.example.com id=shard-1] MucClient$MucWrapper.join#751: Joined MUC: jvbbrewery@internal.auth.meet.example.com

and on prosody:

Apr 21 15:34:34 c2s56039c6d0150 info    Client connected
Apr 21 15:34:34 c2s56039c6d0150 info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Apr 21 15:34:34 c2s56039c6d0150 info    Authenticated as jvb@auth.test.meet.example.com

and jicofo.log:

Jicofo 2020-04-21 15:34:38.310 INFO: [28] org.jitsi.jicofo.xmpp.BaseBrewery.processInstanceStatusChanged().329 Added brewery instance: jvbbrewery@internal.auth.meet.example.com/test-v1.meet.example.com
Jicofo 2020-04-21 15:34:38.311 WARNING: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() No pub-sub node mapped for jvbbrewery@internal.auth.meet.example.comh/test-v1.meet.example.com
Jicofo 2020-04-21 15:34:38.312 INFO: [28] org.jitsi.jicofo.bridge.BridgeSelector.log() Added new videobridge: Bridge[jid=jvbbrewery@internal.auth.meet.example.com/test-v1.meet.example.com, relayId=null, region=null, stress=0.00]