Jigasi high CPU usage with few calls

Hi there,

I’m stumbling on an issue with jigasi and I can’t figure out what I’m doing wrong.

The issue is that jigasi seems to consume high amounts of CPU, I have the following setup:

1 server with jitsi web + jicofo + prosody
1 server with JVB
2 servers with an IPBX and Jigasi

I can successfully join a conference room using my browser and have one IPBX joining the call or have the two IPBX to join the same conference room and talk to each other.

But it seems that this cannot scale at all, with 5 concurrent calls (1 call = 1 conference room), Jigasi is using between 15/20% of the CPU.

I tried 5 calls to 5 conference rooms between my two IPBX&Jigasi and 5 calls to 5 conference room between 1 IPBX&Jigasi and 5 browsers sessions and the results are the same, 15/20% CPU is used by jigasi itself. (other servers are fine no cpu usage to report).

Any thoughts? Jigasi is configured to use PCMA/8000, my IPBX too so there should be no transcoding and i’m pretty sure that my servers can handle more than a few calls (those are dual Xeon Platinum 8275@3Ghz so pretty strong).

Here is the Jigasi setup file sip-communicator.properties:

net.java.sip.communicator.impl.protocol.SingleCallInProgressPolicy.enabled=false

# Adjust opus encoder complexity
net.java.sip.communicator.impl.neomedia.codec.audio.opus.encoder.COMPLEXITY=10

# Disables packet logging
net.java.sip.communicator.packetlogging.PACKET_LOGGING_ENABLED=false

# SIP account
net.java.sip.communicator.impl.protocol.sip.acc1=acc1

net.java.sip.communicator.impl.protocol.sip.acc1.PROXY_ADDRESS=127.0.0.1
net.java.sip.communicator.impl.protocol.sip.acc1.PROXY_AUTO_CONFIG=false
net.java.sip.communicator.impl.protocol.sip.acc1.PROXY_PORT=5060
net.java.sip.communicator.impl.protocol.sip.acc1.PREFERRED_TRANSPORT=UDP


net.java.sip.communicator.impl.protocol.sip.acc1.ACCOUNT_UID=SIP\:redacted@127.0.0.1
net.java.sip.communicator.impl.protocol.sip.acc1.PASSWORD=redacted
net.java.sip.communicator.impl.protocol.sip.acc1.PROTOCOL_NAME=SIP
net.java.sip.communicator.impl.protocol.sip.acc1.SERVER_ADDRESS=127.0.0.1
net.java.sip.communicator.impl.protocol.sip.acc1.USER_ID=redacted@127.0.0.1
net.java.sip.communicator.impl.protocol.sip.acc1.KEEP_ALIVE_INTERVAL=25
net.java.sip.communicator.impl.protocol.sip.acc1.KEEP_ALIVE_METHOD=OPTIONS
net.java.sip.communicator.impl.protocol.sip.acc1.VOICEMAIL_ENABLED=false
net.java.sip.communicator.impl.protocol.sip.acc1.JITSI_MEET_ROOM_HEADER_NAME=X-Room-Name
net.java.sip.communicator.impl.protocol.sip.acc1.JITSI_MEET_DOMAIN_BASE_HEADER_NAME=X-Domain-Base
net.java.sip.communicator.impl.protocol.sip.acc1.Encodings.PCMA/8000=600
net.java.sip.communicator.impl.protocol.sip.acc1.Encodings.PCMU/8000=650
net.java.sip.communicator.impl.protocol.sip.acc1.Encodings.telephone-event/8000=1
net.java.sip.communicator.impl.protocol.sip.acc1.Encodings.ulpfec/90000=0
net.java.sip.communicator.impl.protocol.sip.acc1.OVERRIDE_ENCODINGS=false
net.java.sip.communicator.impl.protocol.sip.acc1.DOMAIN_BASE=meet.jitsi

# XMPP account used for control
net.java.sip.communicator.impl.protocol.jabber.acc1=acc1
net.java.sip.communicator.impl.protocol.jabber.acc1.ACCOUNT_UID=Jabber:jigasi@auth.meet.jitsi
net.java.sip.communicator.impl.protocol.jabber.acc1.USER_ID=jigasi@auth.meet.jitsi
net.java.sip.communicator.impl.protocol.jabber.acc1.IS_SERVER_OVERRIDDEN=true
net.java.sip.communicator.impl.protocol.jabber.acc1.SERVER_ADDRESS=redacted
net.java.sip.communicator.impl.protocol.jabber.acc1.PASSWORD=redacted
net.java.sip.communicator.impl.protocol.jabber.acc1.AUTO_GENERATE_RESOURCE=true
net.java.sip.communicator.impl.protocol.jabber.acc1.RESOURCE_PRIORITY=30
net.java.sip.communicator.impl.protocol.jabber.acc1.IS_CARBON_DISABLED=true
net.java.sip.communicator.impl.protocol.jabber.acc1.DEFAULT_ENCRYPTION=true
net.java.sip.communicator.impl.protocol.jabber.acc1.IS_USE_ICE=true
net.java.sip.communicator.impl.protocol.jabber.acc1.IS_ACCOUNT_DISABLED=false
net.java.sip.communicator.impl.protocol.jabber.acc1.IS_PREFERRED_PROTOCOL=false
net.java.sip.communicator.impl.protocol.jabber.acc1.AUTO_DISCOVER_JINGLE_NODES=false
net.java.sip.communicator.impl.protocol.jabber.acc1.PROTOCOL=Jabber
net.java.sip.communicator.impl.protocol.jabber.acc1.IS_USE_UPNP=false
net.java.sip.communicator.impl.protocol.jabber.acc1.USE_DEFAULT_STUN_SERVER=true
net.java.sip.communicator.impl.protocol.jabber.acc1.ENCRYPTION_PROTOCOL.DTLS-SRTP=0
net.java.sip.communicator.impl.protocol.jabber.acc1.ENCRYPTION_PROTOCOL_STATUS.DTLS-SRTP=true
net.java.sip.communicator.impl.protocol.jabber.acc1.VIDEO_CALLING_DISABLED=true
net.java.sip.communicator.impl.protocol.jabber.acc1.OVERRIDE_ENCODINGS=false
net.java.sip.communicator.impl.protocol.jabber.acc1.Encodings.PCMA/8000=0
net.java.sip.communicator.impl.protocol.jabber.acc1.Encodings.PCMU/8000=0
net.java.sip.communicator.impl.protocol.jabber.acc1.Encodings.telephone-event/8000=0
net.java.sip.communicator.impl.protocol.jabber.acc1.BREWERY=jigasibrewery@internal-muc.meet.jitsi
net.java.sip.communicator.impl.protocol.jabber.acc1.DOMAIN_BASE=meet.jitsi

org.jitsi.jigasi.BREWERY_ENABLED=true

org.jitsi.jigasi.HEALTH_CHECK_SIP_URI=
org.jitsi.jigasi.HEALTH_CHECK_INTERVAL=300000
org.jitsi.jigasi.HEALTH_CHECK_TIMEOUT=600000

org.jitsi.jigasi.xmpp.acc.IS_SERVER_OVERRIDDEN=true
org.jitsi.jigasi.xmpp.acc.SERVER_ADDRESS=redacted
org.jitsi.jigasi.xmpp.acc.VIDEO_CALLING_DISABLED=true
org.jitsi.jigasi.xmpp.acc.JINGLE_NODES_ENABLED=false
org.jitsi.jigasi.xmpp.acc.AUTO_DISCOVER_STUN=false
org.jitsi.jigasi.xmpp.acc.IM_DISABLED=true
org.jitsi.jigasi.xmpp.acc.SERVER_STORED_INFO_DISABLED=true
org.jitsi.jigasi.xmpp.acc.IS_FILE_TRANSFER_DISABLED=true


# Activate this property if you are using self-signed certificates or other
# type of non-trusted certicates. In this mode your service trust in the
# remote certificates always.
net.java.sip.communicator.service.gui.ALWAYS_TRUST_MODE_ENABLED=true


net.java.sip.communicator.packetlogging.PACKET_LOGGING_ENABLED=false

Thanks!

Mixing audio is intensive so I don’t expect it to use less CPU. The CPU will grow based on the participants in the meeting…

We are offloading jigasi in production by using translator mode and leaving it to the sip side doing the mixing. Guide for setting up Jigasi with Voximplant

Hi Damencho,

Thanks for the hint,

Since we have our own ipbx with our own numbers, I managed to enable the translator mode on jigasi.

I had to change my IPBX configuration to enforce usage of OPUS codec, is it possible to still use alaw/ulaw with translator mode or opus is the only supported codec?

Then I tested again, CPU usage is lower but still high, for example with 10 users per server (1 user = 1 conference room) cpu usage from jigasi is around 10-15%.
I managed to squeeze 50 simultaneous calls and Jigasi is almost maxing out one CPU leaving not much for my IPBX.

So there was 50 conference rooms, each conference room had 1 user from my server A and 1 user from my server B, nothing very fancy with hundreds of users inside one conference room.
The two servers with the ipbx+jigasi showed the same CPU usage.
The JVB and Jitsi server were fine with low cpu usage.

I was expecting some performance penalty for using jigasi to allow my IPBX to join and handle jitsi meetings but not at this scale, is there something that I am missing and I can try?

Opus, as this is what is coming from clients, the idea is to leave decoding and mixing on the sip side. Does your ipbx support multistreaming? You need to test with two participants from the web side and see whether your dial-in hears it well.

Well, that’s why yo have a pool of jigasis that can scale up and down.

  • Yes, you’re right!
    That’s why I changed my Ipbx (I am using Asterisk) configuration so it handle the PCMA to Opus conversion and I was hoping that jigasi would just act as a signaling layer and use very little CPU.

A single asterisk box without jigasi can handle more than 150-200 concurrent calls easily, but then even with jigasi in translator mode I’m in trouble since 50 simultaneous calls works… but it is putting too much pressure on the server.

  • I have already a pool of asterisk servers, I was planning to have my ipbx+jigasi on the same host to leave all the sip related operations at the same place (and thus, preventing network back and forth between multiple hosts since I have a cluster of JVB too).
    Building an additional cluster for jigasi instances will be costly and complicated since the ipbx cluster has to work well with the jigasi cluster (more moving parts and more logic to put in place).

Hello,

I’m trying multiple approaches regarding codec usage/translator trying to reduce the CPU load of jigasi, so far without success.

It seems that Jitsi meet uses opus by default, is it possible to force jitsi meet/libjitsi to use only PCMA/PCMU (G711) ?

Anything else I can try?

Edit: Just in case, I want to specify there are no video at all since the beginning of this topic, all involved calls are audio only (that’s mainly why I’m blown away by the amount of system resources that are used)

Are you asking about jigasi or jitsi-meet. Using g711 from client to the bridge is not a good idea … and I’m not sure whether it is possible at all.
If you use g711 in jigasi you cannot use translator mode as it needs decoding, mixing and encoding before sending.

Yes I was asking about Jitsi-meet,

I was thinking of using jitsi-meet with g711 (if possible), jigasi in translator mode with g711, asterisk with g711, so the full chain is G711 and no transcoding has to be done anywhere in the chain, in the hope of reducing system load.

My use case implies many small rooms with 1 to 3 people in each room, mainly for audio calls (video usage will be marginal and I understand that using video will consume more system ressources and I’m OK with that :slight_smile: )

But for 1 to 1 or 1 to 2 audio calls, the performance of jigasi is really an issue and I need SIP since some users will only be reachable by phone.

I made the change to have asterisk feeding jigasi directly with opus (like your example with Voximplant), it seems better (uses less CPU) but still, I can’t manage to handle more than 50-60 calls on my server that has 2x Xeon Platinum 8275@3Ghz)

With 50 simultaneous calls jigasi is taking 1 full CPU (the other is taken by asterisk)

@damencho I was wondering, are you willing to share with me some specs of your current infrastructure?
What kind of hardware are you using?
How many concurrent calls one of your instance of jigasi is able to manage on your hardware?
What would be your recomandations for handling about 300 calls with jigasi per server?

Thanks :slight_smile:

We do not handle 300 participants per jigasi. We autoscale to keep jigasi as low as 250 participants avg per instance, I think, these are participants counted in the conference. So when 1 jigasi enters a conference with 100 participants this is counted as 100. We use c5.xlarge if I remember correctly.

Ok, I use C5.Large instances for my tests so only 2 cpu available.

I tried 50 participants inside one conference room using browsers, then added one sip call using jitsi, indeed it works and the load is almost 10%CPU for 1 jigasi so that’s ok.
I added more sip calls until the server maxed out it’s CPU with 7 active calls (same number of browsers connected so there was 50 browsers + 7 sip calls using jigasi).

Those numbers are okay but only for the use case you described.
You have one sip call using jigasi with 100 participants with browsers inside one room.
So I understand that your c5.xlarge will handle ~250 participants with 1 or 2 jigasi inside the conference.

I have 100 participants but only with phones, 2 participants per conference room for a total of 50 conference rooms.
In my case my c5.large can handle only 50 participants after that jigasi is taking too much cpu.
I will upgrade my instance size to match yours but I highly doubt that I can manage to get as close as your figures.

Any recommandations?
Is there another way to allow sip calls to join conferences without jigasi?

Thanks for your advices

7 jigasi calls in a 50 participant meeting, this are total of 350 participants.

One more thing is that voximplant is using rfc6464 marking the audio levels in rtp headers so this reduces a lot of re-sending of streams as those are dropped early in the pipeline. I think, here maybe the big difference.

Nope.

Yes this confirms what you said earlier, but how about 100-250 participants, everyone with phones ?
This is here that things are not working well with jigasi

Ok I doubt that asterisk supports the rfc6464 I will check, but I think it won’t solve the issue I’m facing since I have multiple conference rooms with only 2 users/room, rfc6464 is meant for large conferences with lot of streams, not my case.

Yeah, that’s right.

Hum, I don’t think we do such number, I will check do we have some stats for meet.jit.si about it.

Hum, now I see on meet.jit.si we have one jigasi with 18 calls totalling 160 participants and this is on aws t3.large instance.
So apparently I messed up the numbers above. And for c5.xlarge I see 20 calls and going up to 260 participants in the last few hours. But yeah, what I see we don’t go over 25 calls on an instance and we do up to 500 participants per instance for the c5 … for the last week.
So maybe more calls with lower number of participants is more expensive … How is the RAM going on, what is the usage when you do 50 calls? What about the jigasi threads? You can see the threads value using curl http://localhost:8788/about/stats.

Ok so here are some feedback:

Test run specs:
100 participants
50 rooms
50 participants from server A C5.L
50 participants from server B C5.L
2 participants/room (1 participant per ec2 instance)

RAM usage (server A) :

  • Before : ~584M of system memory used, jigasi only is using about 154M of ram (fresh start)
  • During : ~1.42G of system memory used, jigasi only is using about 690M of ram

CPU usage (server A):

  • Before : <1% (jigasi process only, load avg 1 min 0.01)
  • During: 145% (jigasi process only, load avg 1min = ~50!)

Stats (server A) :

  • During:

    {
    “total_calls_with_connection_failed”: 0,
    “conference_sizes”: [
    0,
    1,
    48,
    1,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0,
    0
    ],
    “threads”: 2168,
    “graceful_shutdown”: false,
    “total_conference_seconds”: 0,
    “total_calls_with_sip_call_waiting”: 0,
    “total_conferences_completed”: 0,
    “total_calls_with_sip_call_reconnected”: 0,
    “current_timestamp”: “2021-03-11 17:56:46.442”,
    “total_participants”: 0,
    “conferences”: 50,
    “participants”: 100,
    “total_calls_with_dropped_media”: 3
    }

Server B stats are almost the same so I think it doesn’t need to be posted.

Conferences names were from 0000 to 0049, checked some randomly with a browser and there was 2 participants in each as expected, sound almost ok (some lags due do the server being full)

I can provide more insights and do more testing if needed!

This is with mixing or as a translator?

Yeah, we have seen that jigasi going above 2500 threads is no good. So seems that jigasi is close to its limits.

This run was with mixing, I can try with translator if you want (it is slightly less CPU-intensive but it won’t make a very big difference)

I would suspect jigasi will use less CPU in this situation which will allow you to squeeze more calls per instance and have a pool of jigasi instances that can serve a large number of calls.

I don’t see anything that can be improved, any PRs improving performance are welcome.

I tried with translator mode

Indeed CPU usage is back to about 80-88% and load average about 40, so indeed this is slightly better but the number of threads is still high:

  • Translator mode with 50 concurrent calls:

{
“total_calls_with_connection_failed”: 0,
“conference_sizes”: [
0,
0,
49,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
“threads”: 2002,
“graceful_shutdown”: false,
“total_conference_seconds”: 0,
“total_calls_with_sip_call_waiting”: 0,
“total_conferences_completed”: 0,
“total_calls_with_sip_call_reconnected”: 0,
“current_timestamp”: “2021-03-11 18:24:55.042”,
“total_participants”: 0,
“conferences”: 50,
“participants”: 101,
“total_calls_with_dropped_media”: 0
}

With translator mode I can squeeze ~55 calls, after that it starts to drop media too because the cpu utilization gets to 100% on each core.

Sadly i’m not competent in Java programming, otherwise it would have been a pleasure to help.