Jigasi memmory release issue after disconnect conference

I have setup jigasi on one server and freeSWITCH media server on other server, audio call dial in working fine with freeswitch conncetivity also with voice.
but problem i am facing that when some join into conference through phone by dialing DID then it consumes RAM and does not release memory even after terminate call result is high memory utilization due to this jigasi service crashes.
If anybody has get succes in this then please help me.

Thanks & regards

What is the jigasi version used and what is the memory amount of the server?

Jigasi by default requires 3GB of memory, if its less it can crash.

Version: 1.1-220-g9b1fb11-1
Stable

Memory amount: 3072MB RAM

Thanks

Not enough of there are other stuff running. Increase RAM or instruct jigasi that its max memory is 2 or 1.5.

I have increased memory till 5 GB but still getting same issue.

Thanks

You see it grow to 3Gb and then it throws OutOfMemory exception?
Growing memory to the max point is normal, if you see out of memory exception that is indication for a problem.

I restarted the jigasi service and check the RAM consumption is 161.7 Mb(normal) of jigasi service. I made a call for 5 minutes and RAM consumption increases to 304.5 Mb. I disconnected the phone call and was hoping jigasi RAM consumption will decrease 161.7 Mb(approx.) but this did not happen. If I make a few more calls then RAM consumption will shoot up to 7 GB and jigasi will got killed by another service. This makes jigasi unavailable until jigasi service is again restarted.

Uses sudo systemd-cgtop command for RAM stats.
There is only one call running during testing.

@mukeshstartelelogic and I are from same team.

Growing memory to the max point is normal

When it will starts releasing memory?.
Thanks.

To not go back down to 161 is normal.

You have reproduced that? If it tries to go beyond the max that is configured and there is out of memory error this means there is a problem.

When there is no activity and garbage collection has been running for several cycles several times. If you want to preserve memory you can switch to more aggressive garbage collection, but for real-time applications like jigasi and jitsi-videobridge this will hit performance and you will experience regular freezes due to garbage collection running.

Jigasi max memory value = 3072 Mb
Java version for jigasi = 8

Before killing of jigasi service, I had noted down the PID of jigasi service which is 3563.

Below is kern.log file output when jigasi is killed by oom-killer service. Around 1h 30 minutes before the killing of service, the RAM consumption was 3.1GB.

Mar 30 05:33:18 username kernel: [4288002.251419] VM Periodic Tas invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Mar 30 05:33:18 username kernel: [4288002.251421] VM Periodic Tas cpuset=/ mems_allowed=0
Mar 30 05:33:18 username kernel: [4288002.251425] CPU: 1 PID: 13681 Comm: VM Periodic Tas Not tainted 4.15.0-166-generic #174-Ubuntu
Mar 30 05:33:18 username kernel: [4288002.251426] Hardware name: Amazon EC2 c5.xlarge/, BIOS 1.0 10/16/2017
Mar 30 05:33:18 username kernel: [4288002.251427] Call Trace:
Mar 30 05:33:18 username kernel: [4288002.251434]  dump_stack+0x6d/0x8b
Mar 30 05:33:18 username kernel: [4288002.251436]  dump_header+0x71/0x282
Mar 30 05:33:18 username kernel: [4288002.251439]  ? security_capable+0x51/0x70
Mar 30 05:33:18 username kernel: [4288002.251442]  oom_kill_process+0x21f/0x420
Mar 30 05:33:18 username kernel: [4288002.251444]  out_of_memory+0x116/0x4e0
Mar 30 05:33:18 username kernel: [4288002.251446]  __alloc_pages_slowpath+0xa53/0xe00
Mar 30 05:33:18 username kernel: [4288002.251448]  __alloc_pages_nodemask+0x29a/0x2c0
Mar 30 05:33:18 username kernel: [4288002.251452]  alloc_pages_current+0x6a/0xe0
Mar 30 05:33:18 username kernel: [4288002.251455]  __page_cache_alloc+0x81/0xa0
Mar 30 05:33:18 username kernel: [4288002.251457]  filemap_fault+0x42f/0x750
Mar 30 05:33:18 username kernel: [4288002.251459]  ? set_page_dirty+0x5e/0xb0
Mar 30 05:33:18 username kernel: [4288002.251460]  ? filemap_map_pages+0x181/0x390
Mar 30 05:33:18 username kernel: [4288002.251464]  ext4_filemap_fault+0x31/0x50
Mar 30 05:33:18 username kernel: [4288002.251466]  __do_fault+0x34/0xe0
Mar 30 05:33:18 username kernel: [4288002.251468]  __handle_mm_fault+0x982/0xc50
Mar 30 05:33:18 username kernel: [4288002.251470]  handle_mm_fault+0xe7/0x260
Mar 30 05:33:18 username kernel: [4288002.251474]  __do_page_fault+0x281/0x4b0
Mar 30 05:33:18 username kernel: [4288002.251476]  do_page_fault+0x2e/0xe0
Mar 30 05:33:18 username kernel: [4288002.251479]  ? async_page_fault+0x2f/0x50
Mar 30 05:33:18 username kernel: [4288002.251482]  do_async_page_fault+0x51/0x80
Mar 30 05:33:18 username kernel: [4288002.251483]  async_page_fault+0x45/0x50
Mar 30 05:33:18 username kernel: [4288002.251485] RIP: 0033:0x7f58c54fc6fc
Mar 30 05:33:18 username kernel: [4288002.251486] RSP: 002b:00007f5873ffedb0 EFLAGS: 00010206
Mar 30 05:33:18 username kernel: [4288002.251487] RAX: 00007f58c00294e0 RBX: 00000000000000c2 RCX: 0000000000000017
Mar 30 05:33:18 username kernel: [4288002.251488] RDX: 0000000000000000 RSI: 00007f5873ffed30 RDI: 00007f5873ffecbc
Mar 30 05:33:18 username kernel: [4288002.251489] RBP: 00007f5873ffedf0 R08: 00007ffe0a3e3000 R09: 00058e53d0d40d1b
Mar 30 05:33:18 username kernel: [4288002.251490] R10: 00007f5873ffed10 R11: 0000000000000001 R12: 00007f58c5a664e0
Mar 30 05:33:18 username kernel: [4288002.251490] R13: 0000000000000001 R14: 0000000000000002 R15: 0000000000000001
Mar 30 05:33:18 username kernel: [4288002.251492] Mem-Info:
Mar 30 05:33:18 username kernel: [4288002.251496] active_anon:1736541 inactive_anon:222 isolated_anon:0
Mar 30 05:33:18 username kernel: [4288002.251496]  active_file:32 inactive_file:41 isolated_file:1
Mar 30 05:33:18 username kernel: [4288002.251496]  unevictable:0 dirty:0 writeback:0 unstable:0
Mar 30 05:33:18 username kernel: [4288002.251496]  slab_reclaimable:16774 slab_unreclaimable:64247
Mar 30 05:33:18 username kernel: [4288002.251496]  mapped:161 shmem:578 pagetables:14326 bounce:0
Mar 30 05:33:18 username kernel: [4288002.251496]  free:25300 free_pcp:621 free_cma:0
Mar 30 05:33:18 username kernel: [4288002.251498] Node 0 active_anon:6946164kB inactive_anon:888kB active_file:128kB inactive_file:164kB unevictable:0kB isolated(anon):0kB isolated(file):4kB mapped:644kB dirty:0kB writeback:0kB shmem:2312kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Mar 30 05:33:18 username kernel: [4288002.251499] Node 0 DMA free:15908kB min:136kB low:168kB high:200kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 30 05:33:18 username kernel: [4288002.251502] lowmem_reserve[]: 0 2933 7544 7544 7544
Mar 30 05:33:18 username kernel: [4288002.251504] Node 0 DMA32 free:44344kB min:26224kB low:32780kB high:39336kB active_anon:2698340kB inactive_anon:8kB active_file:12kB inactive_file:248kB unevictable:0kB writepending:0kB present:3129256kB managed:3063688kB mlocked:0kB kernel_stack:138308kB pagetables:23520kB bounce:0kB free_pcp:748kB local_pcp:144kB free_cma:0kB
Mar 30 05:33:18 username kernel: [4288002.251507] lowmem_reserve[]: 0 0 4610 4610 4610
Mar 30 05:33:18 username kernel: [4288002.251509] Node 0 Normal free:40948kB min:41216kB low:51520kB high:61824kB active_anon:4247824kB inactive_anon:880kB active_file:724kB inactive_file:284kB unevictable:0kB writepending:0kB present:4878336kB managed:4727520kB mlocked:0kB kernel_stack:122348kB pagetables:33784kB bounce:0kB free_pcp:1736kB local_pcp:128kB free_cma:0kB
Mar 30 05:33:18 username kernel: [4288002.251511] lowmem_reserve[]: 0 0 0 0 0
Mar 30 05:33:18 username kernel: [4288002.251513] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Mar 30 05:33:18 username kernel: [4288002.251519] Node 0 DMA32: 1014*4kB (UME) 575*8kB (UME) 318*16kB (UE) 168*32kB (UME) 66*64kB (UME) 47*128kB (UE) 25*256kB (ME) 15*512kB (ME) 1*1024kB (M) 0*2048kB 0*4096kB = 44464kB
Mar 30 05:33:18 username kernel: [4288002.251525] Node 0 Normal: 1725*4kB (UMEH) 445*8kB (UMEH) 577*16kB (UMEH) 437*32kB (UMEH) 65*64kB (UME) 23*128kB (UME) 3*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41548kB
Mar 30 05:33:18 username kernel: [4288002.251531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 30 05:33:18 username kernel: [4288002.251532] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 30 05:33:18 username kernel: [4288002.251533] 670 total pagecache pages
Mar 30 05:33:18 username kernel: [4288002.251534] 0 pages in swap cache
Mar 30 05:33:18 username kernel: [4288002.251535] Swap cache stats: add 0, delete 0, find 0/0
Mar 30 05:33:18 username kernel: [4288002.251536] Free swap  = 0kB
Mar 30 05:33:18 username kernel: [4288002.251536] Total swap = 0kB
Mar 30 05:33:18 username kernel: [4288002.251536] 2005896 pages RAM
Mar 30 05:33:18 username kernel: [4288002.251537] 0 pages HighMem/MovableOnly
Mar 30 05:33:18 username kernel: [4288002.251537] 54117 pages reserved
Mar 30 05:33:18 username kernel: [4288002.251538] 0 pages cma reserved
Mar 30 05:33:18 username kernel: [4288002.251538] 0 pages hwpoisoned
Mar 30 05:33:18 username kernel: [4288002.251539] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Mar 30 05:33:18 username kernel: [4288002.251545] [  443]     0   443    59192     2964   466944        0             0 systemd-journal
Mar 30 05:33:18 username kernel: [4288002.251547] [  453]     0   453    26478       58   102400        0             0 lvmetad
Mar 30 05:33:18 username kernel: [4288002.251549] [  465]     0   465    11403      337   114688        0         -1000 systemd-udevd
Mar 30 05:33:18 username kernel: [4288002.251551] [  622] 62583   622    33860      115   163840        0             0 systemd-timesyn
Mar 30 05:33:18 username kernel: [4288002.251553] [  692]   100   692    20012      167   184320        0             0 systemd-network
Mar 30 05:33:18 username kernel: [4288002.251554] [  694]   101   694    17684      187   180224        0             0 systemd-resolve
Mar 30 05:33:18 username kernel: [4288002.251556] [  789]     0   789     1140       15    57344        0             0 acpid
Mar 30 05:33:18 username kernel: [4288002.251557] [  801]     0   801    72188      447   208896        0             0 accounts-daemon
Mar 30 05:33:18 username kernel: [4288002.251558] [  803]     0   803    42773     1997   233472        0             0 networkd-dispat
Mar 30 05:33:18 username kernel: [4288002.251560] [  806]     0   806     7939       75   106496        0             0 cron
Mar 30 05:33:18 username kernel: [4288002.251561] [  811]     0   811    27623       83   122880        0             0 irqbalance
Mar 30 05:33:18 username kernel: [4288002.251563] [  814]   103   814    12638      300   143360        0          -900 dbus-daemon
Mar 30 05:33:18 username kernel: [4288002.251564] [  838]     0   838     7085       55   102400        0             0 atd
Mar 30 05:33:18 username kernel: [4288002.251566] [  851]   102   851    65760      458   155648        0             0 rsyslogd
Mar 30 05:33:18 username kernel: [4288002.251567] [  852]     0   852   169540      242   147456        0             0 lxcfs
Mar 30 05:33:18 username kernel: [4288002.251569] [  857]     0   857    17879      423   176128        0             0 systemd-logind
Mar 30 05:33:18 username kernel: [4288002.251570] [  858]     0   858   374717     3532   307200        0          -999 containerd
Mar 30 05:33:18 username kernel: [4288002.251572] [  870]     0   870    72867      317   208896        0             0 polkitd
Mar 30 05:33:18 username kernel: [4288002.251573] [  872]     0   872     4105       37    73728        0             0 agetty
Mar 30 05:33:18 username kernel: [4288002.251574] [  875]     0   875     3724       33    69632        0             0 agetty
Mar 30 05:33:18 username kernel: [4288002.251576] [  882]     0   882    46920     1979   262144        0             0 unattended-upgr
Mar 30 05:33:18 username kernel: [4288002.251578] [  906]     0   906    18077      188   176128        0         -1000 sshd
Mar 30 05:33:18 username kernel: [4288002.251579] [ 1113]     0  1113   412249     7597   516096        0          -500 dockerd
Mar 30 05:33:18 username kernel: [4288002.251580] [ 6389]  1000  6389    19172      293   196608        0             0 systemd
Mar 30 05:33:18 username kernel: [4288002.251582] [ 6390]  1000  6390    64803      612   262144        0             0 (sd-pam)
Mar 30 05:33:18 username kernel: [4288002.251583] [ 6945]     0  6945   151794     4294  1015808        0             0 node
Mar 30 05:33:18 username kernel: [4288002.251585] [23814]     0 23814   151728     4179   999424        0             0 node
Mar 30 05:33:18 username kernel: [4288002.251586] [25027]     0 25027   151968     4422  1036288        0             0 node
Mar 30 05:33:18 username kernel: [4288002.251588] [25150]     0 25150   152617     5475  1077248        0             0 node
Mar 30 05:33:18 username kernel: [4288002.251589] [25474]     0 25474   151704     4156   987136        0             0 node
Mar 30 05:33:18 username kernel: [4288002.251590] [25634]     0 25634   151762     4040  1024000        0             0 node
Mar 30 05:33:18 username kernel: [4288002.251591] [31331]     0 31331   382625     1871   225280        0             0 amazon-ssm-agen
Mar 30 05:33:18 username kernel: [4288002.251593] [31375]     0 31375   385677     2783   241664        0             0 ssm-agent-worke
Mar 30 05:33:18 username kernel: [4288002.251594] [ 8682]     0  8682   181664     2735   344064        0             0 turnserver
Mar 30 05:33:18 username kernel: [4288002.251595] [20416]     0 20416   296357     4231   335872        0          -900 snapd
Mar 30 05:33:18 username kernel: [4288002.251596] [27705]     0 27705    25914      624   241664        0             0 apache2
Mar 30 05:33:18 username kernel: [4288002.251598] [26765]   996 26765   277083    12235   606208        0             0 Xorg
Mar 30 05:33:18 username kernel: [4288002.251599] [26766]   996 26766     3517       36    77824        0             0 icewm-session
Mar 30 05:33:18 username kernel: [4288002.251600] [26769]   996 26769    19131      501   192512        0             0 icewm
Mar 30 05:33:18 username kernel: [4288002.251602] [22212]   996 22212  1389258    92795  1331200        0             0 java
Mar 30 05:33:18 username kernel: [4288002.251603] [23563]   998 23563  1749649   212172  2822144        0             0 java
Mar 30 05:33:18 username kernel: [4288002.251605] [11153]   111 11153    36716    20124   331776        0             0 lua5.2
Mar 30 05:33:18 username kernel: [4288002.251606] [ 3101]     0  3101   229099     6857  1294336        0             0 node
Mar 30 05:33:18 username kernel: [4288002.251608] [13584]   999 13584  2490407   412436 10186752        0             0 java
Mar 30 05:33:18 username kernel: [4288002.251609] [26638]    33 26638   525804     2827   495616        0             0 apache2
Mar 30 05:33:18 username kernel: [4288002.251611] [26639]    33 26639   525864     2980   491520        0             0 apache2
Mar 30 05:33:18 username kernel: [4288002.251612] [ 3563]   997  3563  4966131   741045 32833536        0             0 java
Mar 30 05:33:18 username kernel: [4288002.251613] [21478]     0 21478    26998      254   258048        0             0 sshd
Mar 30 05:33:18 username kernel: [4288002.251615] [21575]  1000 21575    26998      252   249856        0             0 sshd
Mar 30 05:33:18 username kernel: [4288002.251616] [21576]  1000 21576     5806      432    86016        0             0 bash
Mar 30 05:33:18 username kernel: [4288002.251618] [32002]     0 32002    18077      188   167936        0         -1000 sshd
Mar 30 05:33:18 username kernel: [4288002.251619] Out of memory: Kill process 3563 (java) score 383 or sacrifice child
Mar 30 05:33:18 username kernel: [4288002.282673] Killed process 3563 (java) total-vm:19864524kB, anon-rss:2964180kB, file-rss:0kB, shmem-rss:0kB
Mar 30 05:33:18 username kernel: [4288002.624488] oom_reaper: reaped process 3563 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Below is the start and stop time of jigasi service.

Mar 29 11:04:18 dummy.domain.com systemd[1]: Starting Jitsi Gateway for SIP...
Mar 29 11:04:18 dummy.domain.com systemd[1]: Started Jitsi Gateway for SIP.
Mar 30 05:33:18 dummy.domain.com systemd[1]: jigasi.service: Main process exited, code=killed, status=9/KILL
Mar 30 05:33:18 dummy.domain.com systemd[1]: jigasi.service: Failed with result 'signal'.

When there is no activity and garbage collection has been running for several cycles several times.

Before Going to sleep at night, I checked the RAM consumption of jigasi is 1GB,
When i wake in the morning, the RAM consumption increases to 2GB. I was hoping jigasi will release memory because server was idle at night and garbage collection will do the job, but that doesn’t happen.

Thank you very much for your valuable input.

Jigasi logs, please?
Then there is something leaking … When there are no calls create a heapdump. There is a script doing that in usr share jigasi
And send it to me personally, to take a look. You can message me here privately or use damencho at jitsi dot org.

I have sent you a mail.

In your log file there are 13193 incoming sip calls and 137 were hanged up and 1449 failed.

The heap dump looks normal 3 JvbConferences, 3 XMPP calls and 3 SIP calls. But you took the heapdump after restarting jigasi. I don’t see any problem with the jigasi at that time.

I can see is that you are using adoptopenjdk, this is something we haven’t tested with … not sure whether there will be any difference if you run openjdk-8.

Maybe give more context what are you trying to do? You are using the same room over and over again? How many participants do you expect to have in one room … ? How many are there when you see it failing.

Are you monitoring the prosody CPU usage on the shard where is the room you are testing with?
There are a lot of “Did not received session invite (30000 ms)” … which means something else is wrong … jigasi is not receiving invite from jicofo …
And a lot of failed sip calls is this expected?

I can see is that you are using adoptopenjdk, this is something we haven’t tested with … not sure whether there will be any difference if you run openjdk-8

If Jitsi team tested jigasi with openjdk-8 ??

Maybe give more context what are you trying to do

User can create a conference by default lobby turned on. Users who created the create the conference are authenticated using jwt token. Each conference have have dial in and recording extra feature. Remaining users who joined the conference are joined as guest user. Multiple users can dial in conference via phone in a given conference after pin verification from freeswitch side.

You are using the same room over and over again?

Most of the time, we are using new room but some time we are using old room.

How many are there when you see it failing.

It started even with one user from browser and one user from conference in a given room. I don’t think , this issue has to do with users count.

Are you monitoring the prosody CPU usage on the shard where is the room you are testing with?

No

There are a lot of “Did not received session invite (30000 ms)” … which means something else is wrong … jigasi is not receiving invite from jicofo …

I have raised issue to voip team why so many unwanted sip invite are coming to jigasi server which does not have below sip header.

X-Room-Name: room@conference.domain
Jitsi-Conference-Room: room@conference.domain
Jitsi-Conference-Room-Pass: pass

And a lot of failed sip calls is this expected?

I have raised the issue to voip team, why so many unwanted session invite are coming to jitsi and those invites are not related to jitsi.

I have again sent you a mail with jigasi dump.