Enterprise self-hosted in k8s

Hello everyone!

I need to install selfhosted jitsi core stack (prosody, jvb, web, jicofo) in big company for multiuser conferences with all modern enterprise requirements. First of all run in kubernetes cluster in docker containers as non-root user.
Non-root requirement I fixed with some chmod magic in official images but can’t setup it for multiuser conferences.

My problems is:

  1. k8s cluster have its own internal network, clients outside of it inside other network in office and I absolutely don’t understand how it all should work at all (in best case it should work from external internet).
  2. I get many errors in many layers and I find few pages and no solutions in my searches.

Few errors with 3 participants:

  1. from chrome devtools console
2022-08-03T06:21:54.878Z [modules/statistics/AvgRTPStatsReporter.js] <Dd.addNext>:  bandwidth_upload - invalid value for idx: 1 undefined
r @ Logger.js:154
addNext @ AvgRTPStatsReporter.js:44
_calculateAvgStats @ AvgRTPStatsReporter.js:638
Md.t._onLocalStatsUpdated @ AvgRTPStatsReporter.js:534
r.emit @ events.js:158
_updateLocalStats @ ConnectionQuality.js:489
r.emit @ events.js:158
nn._processAndEmitReport @ RTPStatsCollector.js:429
nn.processStatsReport @ RTPStatsCollector.js:703
(anonymous) @ RTPStatsCollector.js:248
Promise.then (async)
t @ RTPStatsCollector.js:242
setInterval (async)
nn.start @ RTPStatsCollector.js:259
cn.startRemoteStats @ statistics.js:232
bl._acceptJvbIncomingCall @ JitsiConference.js:2327
bl.onIncomingCall @ JitsiConference.js:2238
r.emit @ events.js:153
onJingle @ strophe.jingle.js:218
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onMessage @ strophe.umd.js:5836
Logger.js:154 
2022-08-03T06:21:54.879Z [modules/statistics/AvgRTPStatsReporter.js] <Dd.addNext>:  bandwidth_download - invalid value for idx: 0 undefined
r @ Logger.js:154
addNext @ AvgRTPStatsReporter.js:44
_calculateAvgStats @ AvgRTPStatsReporter.js:639
Md.t._onLocalStatsUpdated @ AvgRTPStatsReporter.js:534
r.emit @ events.js:158
_updateLocalStats @ ConnectionQuality.js:489
r.emit @ events.js:158
nn._processAndEmitReport @ RTPStatsCollector.js:429
nn.processStatsReport @ RTPStatsCollector.js:703
(anonymous) @ RTPStatsCollector.js:248
Promise.then (async)
t @ RTPStatsCollector.js:242
setInterval (async)
nn.start @ RTPStatsCollector.js:259
cn.startRemoteStats @ statistics.js:232
bl._acceptJvbIncomingCall @ JitsiConference.js:2327
bl.onIncomingCall @ JitsiConference.js:2238
r.emit @ events.js:153
onJingle @ strophe.jingle.js:218
run @ strophe.umd.js:1875
(anonymous) @ strophe.umd.js:3157
forEachChild @ strophe.umd.js:830
_dataRecv @ strophe.umd.js:3146
_onMessage @ strophe.umd.js:5836
Logger.js:154
  1. from JVB logs (also have it with 2 participants, but in this case all works fine)
Aug 03, 2022 6:24:46 AM org.jitsi.utils.logging2.LoggerImpl log                                         
INFO: Pair failed: 10.239.6.83:10000/udp/host -> 10.7.33.125:56388/udp/host (stream-815eceb1.RTP)       
Aug 03, 2022 6:24:46 AM org.jitsi.utils.logging2.LoggerImpl log                                         
WARNING: Failed to send BINDING-REQUEST(0x1)[attrib.count=6 len=92 tranID=0x7ACB606282014210A8C4BA15]   
java.lang.IllegalArgumentException: No socket found for 10.239.6.83:10000/udp->10.7.34.6:54581/udp      
    at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:631)                          
    at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:581)                          
    at org.ice4j.stack.StunClientTransaction.sendRequest0(StunClientTransaction.java:267)               
    at org.ice4j.stack.StunClientTransaction.sendRequest(StunClientTransaction.java:245)                
    at org.ice4j.stack.StunStack.sendRequest(StunStack.java:680)                                        
    at org.ice4j.ice.ConnectivityCheckClient.startCheckForPair(ConnectivityCheckClient.java:335)        
    at org.ice4j.ice.ConnectivityCheckClient.startCheckForPair(ConnectivityCheckClient.java:231)        
    at org.ice4j.ice.ConnectivityCheckClient$PaceMaker.run(ConnectivityCheckClient.java:938)            
    at org.ice4j.util.PeriodicRunnable.executeRun(PeriodicRunnable.java:206)                            
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)                          
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)                                         
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)                  
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)                  
    at java.lang.Thread.run(Thread.java:748)                                                            

Aug 03, 2022 6:24:46 AM org.jitsi.utils.logging2.LoggerImpl log                                         
INFO: Pair failed: 10.239.6.83:10000/udp/host -> 10.7.34.6:54581/udp/host (stream-815eceb1.RTP)         
Aug 03, 2022 6:24:47 AM org.jitsi.utils.logging2.LoggerImpl log                                         
INFO: Expiring.                                                                                         
Aug 03, 2022 6:24:47 AM org.jitsi.utils.logging2.LoggerImpl log                                         
INFO: Spent 0 seconds oversending                                                                       
Aug 03, 2022 6:24:47 AM org.jitsi.utils.logging2.LoggerImpl log                                         

What should I do? Any ideas or solutions that I don’t know about? (JaaS in not an option - we need selfhosted product)

Do you have any more browser console logs? There is a backtrace but the original error is mising.

Yes. Here it is.
jitsi-1659580321553.log (730.0 KB)

Looks like I fixed it. Sorry for your time Saghul(
There was some problems with service in k8s - for some reasons selector didn’t work with a labels from my helm installation (I checked syntax after discovered problem and all was good). Fix it with by adding myself labels.

Now I need some advice about STUN/TURN servers: I don’t fully understand how it should work in my case, where I have k8s subnet inside office subnet and wanna have multiuser conferences with users from global internet. Can you explain it please? Because I am afraid that a articles on the wiki are not entirely clear for me.

No problem, I’m happy you got it working!

You don’t need them, but they help with an optimization for calls involving just 2 people (the call path would be direct, instead of through the JVB) and for “hostile” network environments.

A simple way to get started would be to deploy the TURN server separately to the k8s setup and just configure the k8s setup to use it. Then you can consider adding a coturn container to your setup.

Thanks a lot!
So if call going through JVB I don’t need a STUN/TUNR servers at all?
Am I understand right?

And due to a recent update of JIBRI with PulseAudio (which now can launch in k8s without audio devices) we want to know if you plan to optimize call recording cause now it looks like very indirect solution? I read about that in other topic.
I mean, I don’t think that MS Teams, for example, follows the same roundabout way with launching Chrome GUI.
Hope your product will keep evolve)

You may still have problems in restrictive networks.

We have no plans to change this. The resulting recording is the most true to the meeting experience. I don’t know what MS Teams but I don’t think speculating is a productive use of time.

Thanks again.
MS Teams = Microsoft Teams
Now I think all clear for me.

I know MS Teams, I somehow botched the sentence though :slight_smile: I meant that I don’t know what they do, so we can only speculate.