Jitsi for swiss higher education - success story

Hi all

SWITCH is the “National Research & Education Network - NREN” for Switzerland. We provide various IT related services to the higher education institutions (universities, universities of applied science, university hospitals etc) sector - including LAN (we run our own 100GBs backbone), Identity Services (based on Shibboleth/ AAI), Cloud IaaS and a lot more. Two weeks ago we saw a huge rise in interest in video conferencing. We use Vidyo and Adobe Connect with a couple of hundred simultaneous licenses each.

We decided to start a Jitsi based service on March 12th and had a running prototype of the service on March 13th. Since then my team has worked hard to scale the system.

Our current setup uses 22 jitsi-meet frontend servers (for various insitutions). The plan was to have them authenticate/authorize via Shibboleth - which worked find in the first prototype (using the August 2019 verision of Jitisi due to a problem in Jicofo with Shibboleth: https://github.com/jitsi/jicofo/issues/404 - I will file a new issue shortly, so that it gets tracked again).

We also soon realized that we had to work on scaling our video bridge infrastructure. In order to have simplified deployment of video bridges, we now work with the nightly version of Jitsi (without Shibboleth Authentication) using a pool of currently 32 videobridges (8 VCPUs, 8 GB RAM each) that connect to all front end instances.

We learnt many things during the last two weeks, and I’m busy sanitizing our deployment scripts (Ansible) to share with others. I hope to make them available this weekend or early next week so others can benefit.

Things we learned and configured (in no particular order):

  • Enable LastN (currently 3)
  • Start with Audio/Video muted after third particpant
  • Enable UDP on videobridge (and in the firewalls :wink: )
  • Callstasts.io is expensive but very worthwhile to diagnose problems
  • Jicofo / Prosody Certificates are a pain, disable the checks

We also built a simple “view” app, that wraps the Jitsi frame (using the external-js API) and removes most of the UI chrome (for example enabling audio/video) except for chat and hangup so that lecturers can use that to lecture and provide a “one way” video channel. This has been a mixed success, but it works.

We ran into the 75 users hard limit, but in practice the conference became unusable after around 25-30 users (mostly due to the browsers having problems handling the UI). We regularly have internal meetings with 15 people and that works very well. I have seen meetings with up to 50 participants, but am not sure, just how well that went.

Our immediate next plans:

  • Upgrade to the latest nightly version this weekend
  • Enable epoll on Prosody
  • Build current versions of https://github.com/jitsi/jitsi-meet-electron and make that available to users
  • Build a custome version of Jibri and stream to an internal RTMP server

All in all this project has been challenging (the documentation is all over the place and many things are not documented well, finding stuff on the discussion forums has been difficult, but the level of knowledge from people like @damencho is great) but so far a great success. We have received a lot of good feedback from our (university) community. Me, I am impressed by the design of Jitsi and how well it works.

Thanks for that & onwards!
Jens-Christian

24 Likes

Is there any code available for the view app? We would like to have the same ‘one way’ video for a school. Many thanks

We also built a simple “view” app, that wraps the Jitsi frame (using the external-js API) and removes most of the UI chrome (for example enabling audio/video) except for chat and hangup so that lecturers can use that to lecture and provide a “one way” video channel. This has been a mixed success, but it works.

Excellent write-up. Thank you for sharing your experiences with such a large scale deployment.

Congratulations on the success!

just awesome and great !

Im a noob user with my single server, as many here i think in this trouble context…

I think the Jitsi team is spending a bunch of time and effort to answer all … and its great,

May you can help buy sharing here each doc & trick you solved ? Pushing some “how to” in this forum ?

Congratulations

Yes, that will be part of the full repository

thx for your quick reply
Is there any time line. We have about 1 week to get things prepared for classes. Any source dump will be very much appreciated.

BTW, what happens if students access jitsi with android app? Does your tool block it also - so that the classes can operate like 1 video stream from teacher - and students cannot mess about. like a video -only app? or do you force them to use broweser?

This only works in the browser. As soon as they use the iOS or the Android Apps, it’s normal measures. We mute Audio/Video after the third participant, but of course they can still wreck havoc :slight_smile:

May I suggest moving some documentation into the Wiki? After getting lost searching the forum and finding lots of people with similar issues, but no real solution, I just added my findings of enabling LDAP authentication on there. My hope is that it’s easier to find for people since it is a little more “static”.

The moderator can always change the password and kick the offending user if necessary. Unless he figures out the new password he shouldn’t be able to join back?

1 Like

Thanks for the pointer to the Wiki, @Balu - I didn’t even realize that it existed. Will put together some docs and lessons learnt.

3 Likes

Here is our Ansible playbook and the necessary roles to deploy our Jitsi infrastructure.

This “works for us” ™ but there surely are things that could be improved. We are looking forward to pull requests :slight_smile:

2 Likes

Hi Jens-Christian,

from what I understand right now, the Jitsi app doesn’t support sibboleth authentication. How do you handle that case?

Ciao
Marcus

We currently choose performance over authentication

Thanks a lot @jcfischer for sharing this story.
We are trying to accomplish the same in Italy with https://iorestoacasa.work

1 Like

I saw what you are doing already last week @tapion . Great work!

The current version in unstable had the bug fixed that prevented the Shibboleth authentication dance to finish. We are now enabling AAI authentication on selected frontend instances again.

Dear team,
I’m Maurizio
I tried to install jitsi following the video tutorial, but in the connection I do not see the video and I cannot hear the audio while giving consent to the camera and microphone. I was trying to make a local server to do lecture in the classroom given the period. A cross appears in my microphone and cameras and cannot be enabled
Thanks
Sorry for my english

Hello Maurizio,

have you tried if they work on https://meet.jit.si/?

Is your server behind a NAT router?

Thank you very much I await instructions to resolve the problem, I noticed that the system has locked my account. I look forward thanks

I am doing a test by creating a virutal server with a shared card and belonging to the same network, it connects but I cannot enable the microphone or the camera, and in reality I do not see any user connected to be able to write in chat, as if in reality I enter the web but I don’t see anyone connected

Good morning,
I am attaching the copy of the logs that my server generates, in my opinion it does not connect at all, in effect it disconnects and tries to reconnect.
Thanks
2020-04-01 09:12:16.803 INFORMAZIONI: [17] Videobridge.createConference#326: create_conf, id=ae8ba105089bdbc3 gid=null logging=false

2020-04-01 09:12:16.806 INFORMAZIONI: [17] Health.doRun#294: Performed a successful health check in 3ms. Sticky failure: false

2020-04-01 09:12:25.837 INFORMAZIONI: [16] VideobridgeExpireThread.expire#144: Running expire()

2020-04-01 09:12:26.808 INFORMAZIONI: [17] Videobridge.createConference#326: create_conf, id=e765e2b8ff4e7b4b gid=null logging=false

2020-04-01 09:12:26.812 INFORMAZIONI: [17] Health.doRun#294: Performed a successful health check in 4ms. Sticky failure: false

2020-04-01 09:12:36.813 INFORMAZIONI: [17] Videobridge.createConference#326: create_conf, id=be282a3fae133bef gid=null logging=false

2020-04-01 09:12:36.816 INFORMAZIONI: [17] Health.doRun#294: Performed a successful health check in 3ms. Sticky failure: false

2020-04-01 09:12:46.817 INFORMAZIONI: [17] Videobridge.createConference#326: create_conf, id=9626391da1d70cfd gid=null logging=false

MPP provider Jabber:focus@auth.test.miodominio.locla/focus16658684513@localhost connected (JID: focus@auth.test.miodominio.locla/focus16658684513)

Jicofo 2020-03-31 18:42:28.433 INFORMAZIONI: [10] org.eclipse.jetty.server.handler.ContextHandler.doStart() Started o.e.j.s.ServletContextHandler@7b32224{/,null,AVAILABLE}

Jicofo 2020-03-31 18:42:28.563 INFORMAZIONI: [10] org.eclipse.jetty.server.AbstractConnector.doStart() Started MuxServerConnector@55848e58{HTTP/1.1,[http/1.1]}{0.0.0.0:8888}

Jicofo 2020-03-31 18:42:28.563 INFORMAZIONI: [10] org.eclipse.jetty.server.Server.doStart() Started @8845ms

Jicofo 2020-03-31 18:42:28.566 INFORMAZIONI: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() net.java.sip.communicator.impl.protocol.jabber.extensions.caps.EntityCapsManager.CAPS.http://prosody.im#ZBWApSGFMsTZkuVThHtyU5xv1Mk==

Jicofo 2020-03-31 18:42:28.567 INFORMAZIONI: [1] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.jitsi.jicofo.BRIDGE_MUC=JvbBrewery@internal.auth.test.miodominio.locla

Jicofo 2020-03-31 18:42:28.594 INFORMAZIONI: [1] org.jitsi.xmpp.component.ComponentBase.log() Component org.jitsi.jicofo. config:

Jicofo 2020-03-31 18:42:28.594 INFORMAZIONI: [1] org.jitsi.xmpp.component.ComponentBase.log() ping interval: 10000 ms

Jicofo 2020-03-31 18:42:28.594 INFORMAZIONI: [1] org.jitsi.xmpp.component.ComponentBase.log() ping timeout: 5000 ms

Jicofo 2020-03-31 18:42:28.594 INFORMAZIONI: [1] org.jitsi.xmpp.component.ComponentBase.log() ping threshold: