Jicofo creating nonstop new threads

Hello there,

I’m hosting Jitsi on a VPS with 8GB RAM, 4Cores and 100 GB Storage.
Config is default, except own turnserver and Auth enabled for Rooms.

My problem is this:
Over time my VPS “freezes” because of the providers Proc/Thread Limit (400), which stops jitsi working and controling the server via shh, so i need to reboot it via provider page.

By stopping services one by one i could identify jicofo creating nonstop new threads, approximately 5 every 10 seconds (by watching count via htop).

I have checked the logs, but no Warning or Error giving me any hint what is going on.

Has someone a glue for me what i should check (in logs or config) to find the problem causing this?

Cheers,
Blockmen

1 Like

I too have this problem. count of threads jicofo on my server ˜290

hi @damencho can you help us?

Update: It’s actually not endless. It stops around 401 ish threads, but that exceeds the limit of my VPS resulting in deadlock.

However, not using Jitsi anymore, there are alternatives which are much more performant and aswell open source.

Topic closed for me.

Hi there,
I try to use Jitsi Meet on a Strato VServer (looks like the system OP uses).
Initial setup and conferences work, but JICOFO creates numerous threads even when the system is idle. After some minutes (approx. 30-60 min.) the server is locked up.

If needed, I could prove output of “jstack -l”.

Please help!
Thank you


In this Case (Strato) the value of “user_beancounters” helps detecting the limit:

cat /proc/user_beancounters| grep -E “uid|numproc”

       uid  resource                     held              maxheld              barrier                limit              failcnt
            numproc                       124                  124                  400                  400                    0

Versions used:

dpkg --list| grep -E “jitsi|jicofo”

ii  jicofo                        1.0-644-1                         all          JItsi Meet COnference FOcus
ii  jitsi-meet                    2.0.5142-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody            1.0.4466-1                        all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-web                1.0.4466-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config         1.0.4466-1                        all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2            2.1-376-g9f12bfe2-1               all          WebRTC compatible Selective Forwarding Unit (SFU)

that’s how it is coded. You have better to pick another provider, 124 threads is a incredibly low value and has to be selected to block too powerful software to work on these systems - they would exhaust the limited resources available for them.

Hello GPatel, thank you for your resonse (and contributions to this great tool).
You are right, obviously the hypervisor (Strato) avoids customers from opening to much processes and threads.

BUT:

  1. when I startup JitsiMeet (esp. JICOFO) I am able to create a single conference with up to 10 users (as much as I espect for my private use).
  2. Strato’s overall thread limit (“numproc”) is 400 (not 124).
  3. No matter if there are conferences or not, the number of threads increases (see the attached JProfiler Screenshot). Every 10s a new thread is started (and never terminated).
  4. If this coded as designed: Is there a chance to modify (or configure) JitsiMeet to avoid creating more and more threads?

Thank you very much

I’m not a contributor to Jitsi-meet

maybe it’s what is remaining then. No matter, 400 is not enough anyway.

try to modify Jicofo code and recompile.

main/java/org/jitsi/jicofo/FocusBundleActivator.java: private static final int SHARED_SCHEDULED_POOL_SIZE = 200;

not a great long term idea, obviously. and if your hoster is limiting so much an essential parameter like threads, chances are that other parameters like bandwidth are very limited too. And you can’t recompile for better bandwidth.

Are you using prosody 0.10?

No, I am using the Debian9-based version of prosody:

dpkg --list| grep -E “jitsi|jicofo|prosody”

ii jicofo 1.0-644-1 all JItsi Meet COnference FOcus
ii jitsi-meet 2.0.5142-1 all WebRTC JavaScript video conferences
ii jitsi-meet-prosody 1.0.4466-1 all Prosody configuration for Jitsi Meet
ii jitsi-meet-web 1.0.4466-1 all WebRTC JavaScript video conferences
ii jitsi-meet-web-config 1.0.4466-1 all Configuration for web serving of Jitsi Meet
ii jitsi-videobridge2 2.1-376-g9f12bfe2-1 all WebRTC compatible Selective Forwarding Unit (SFU)
ii prosody 0.9.12-2+deb9u2 amd64 Lightweight Jabber/XMPP server

cat /etc/apt/sources.list.d/jitsimeet.list

deb https://download.jitsi.org stable/

cat /etc/debian_version

9.13

uname -svomr

Linux 4.9.0 #1 SMP Tue Jun 9 12:58:54 MSK 2020 x86_64 GNU/Linux

So I suspect prosody is leaking the conferences and maybe that may case some of this. But I agree with @gpatel-fr that these numbers seem restrictive and you may expect all sorts of issues.

Is it worth to try an update?
e.g. Debian -- Informationen über Paket prosody in buster
I don’t know if this package is compatible with Deb9.

Hi all!

Same here now - Using a Strato vServer, one category higher and having a limit of 700, which just gets filled on time, while doing nothing.
@weHyggl did you achieve any result or solution here?

Are you using prosody 0.10?

@damencho was there any special thought behind this question? We are running on prosody 0.10.

If now 400/700 is a low or high number as restriction, i can’t tell, as my experience is not high enough. However, is it really a normal behaviour, that jicofo permanently creates threads without destroying the old ones?

Prosody 0.10 is known to leak participants, which will lead to leaking conferences in jicofo.

@jotoeri, too bad you hitting the same problem. I didn’t manage to run Jitsi Meet on the small Strato Vserver (4VCPU, 8GB RAM). Instead, I moved to a similar product at Hetzner (“cx21”, with even fewer cores and RAM: 2 VCPU, 4 GB). The price is comparable. Jitsi Meet runs like a charm for my setup (single conferences with a limited number of users, I haven’t tested more than 10 users).

I just counted the threads on the Hetzner server. With an idle Jitsi server I find
# ps aux -T| wc -l
445

Counting only Jitsi process/threads:
# ps aux -T | grep -E “jicofo|jvb|prosody”| wc -l
341

I come to the following conclusions:

  • The smaller system avoids a high number of threads (I read something that the number of threads is related to the number of CPU in the Java VM, can anyone confirm this?).
  • Used virtualisation of Strato and Hetzner differs, especially in respect of limiting the number of threads
  • I guess there must be some leaking, I don’t see the rationale creating more and more threads on an idle system.
  • @gpatel-fr: Looking a the smaller server, I find a resulting max. number of threads. Guess it should be possible to control this limit.
  • I am happy with my setup. Thanks to the developers!

Ok, so - thank you both for your responses! :relieved:

Seems like we finally got it to work:

  • Just updating prosody seems to have messed up configuration a bit, the number of errors increased, i had some configurations, which were different on a clean installation afterwards.
  • A clean installation with having the prosody-source included (now using prosody v0.11.7) solved our problem of standing right in the limit of threads, even that the number of threads was still very close to the limit. So i’m not sure, if this solved all of the initial problem.
  • Disabling the healthcheck of jicofo, which permanently created new threads, has reduced the number of threads massively.

All in all - We are now at ~500 processes in idle state, having a conference with 7 participants, this increased to 550. However, this absolute number of threads includes a nextcloud instance and a onlyoffice-documentserver. So there’s some more stuff running already.
For now, we’ll see the next weeks how many participants or parallel conferences we can handle before we hit the limit of 700 processes. :thinking:

Greets!
Jonas