Prosody leaking memory

We were running sessions as usual and realized that prosody spiked memory usage without any reason.Usually the memory usage is around 2 gb for 56 participants

> JVB 2020-03-03 20:00:10.718 INFO: [26] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=80b90048dec252e3 conf_name=null,logging=false,conf_count=4,ch_count=168,v_streams=78

But we saw a dramatic increase in prosody memory consumption
Please find snapshot

Also these are prosody logs

Mar 03 19:59:02 bosh0be78c67-5153-4a14-8707-29c317257ee7        info    Authenticated as 707e6269-9e33-4523-a78b-dc0fae52138f@mediaserver.portalelevate.com
Mar 03 19:59:02 boshcc488e8a-eb91-4bc2-9762-2090f97b94bc        info    Authenticated as f63240fc-b680-4cde-bfc0-21b7815e2879@mediaserver.portalelevate.com
Mar 03 19:59:02 mod_bosh        info    New BOSH session, assigned it sid '73e79e6c-e8d3-4cee-a47e-f41e395c2e8a'
Mar 03 19:59:02 mod_bosh        info    New BOSH session, assigned it sid '1074f6d6-cf3e-4be6-bada-24957893d377'
Mar 03 19:59:02 mod_bosh        info    New BOSH session, assigned it sid 'f1ed43cc-c719-47ef-991c-ddd97de462fa'
Mar 03 19:59:03 mod_bosh        info    New BOSH session, assigned it sid 'adf98217-6563-44ea-b8cf-14a63e59bb8e'
Mar 03 19:59:03 bosh73e79e6c-e8d3-4cee-a47e-f41e395c2e8a        info    Authenticated as ee388e48-05d4-436a-bf93-adb400f6a52e@mediaserver.portalelevate.com
Mar 03 19:59:03 mod_bosh        info    New BOSH session, assigned it sid '1f0fb93c-59a2-4ba8-8dc7-63156c723c8f'
Mar 03 19:59:03 boshf1ed43cc-c719-47ef-991c-ddd97de462fa        info    Authenticated as 54fb659f-e66d-47d3-8620-43f83167e27e@mediaserver.portalelevate.com
Mar 03 19:59:03 mod_bosh        info    New BOSH session, assigned it sid 'bc50bc57-5572-42bf-96c5-5f24d16b3a4f'
Mar 03 19:59:03 boshadf98217-6563-44ea-b8cf-14a63e59bb8e        info    Authenticated as 4fa9a743-8f89-46cd-ac54-101f2d838d4f@mediaserver.portalelevate.com
Mar 03 19:59:04 mod_bosh        info    New BOSH session, assigned it sid '6806a915-3667-4fab-aa6c-bed7b2f63840'
Mar 03 19:59:04 mod_bosh        info    New BOSH session, assigned it sid 'c2b0cc91-afc1-4b7a-8b8e-3de0726be0ba'
Mar 03 19:59:04 bosh6806a915-3667-4fab-aa6c-bed7b2f63840        info    Authenticated as 5c7a0360-6143-46b0-ac57-d541fe8d39d7@mediaserver.portalelevate.com
Mar 03 19:59:04 boshc2b0cc91-afc1-4b7a-8b8e-3de0726be0ba        info    Authenticated as fbe05523-6a4b-455e-84ae-27544030c646@mediaserver.portalelevate.com
Mar 03 19:59:04 mod_bosh        info    New BOSH session, assigned it sid '220153fa-bd8d-4f12-b2b0-0a438268c1a2'
Mar 03 19:59:05 bosh220153fa-bd8d-4f12-b2b0-0a438268c1a2        info    Authenticated as 5b972570-a57c-4536-a823-59750b462fb8@mediaserver.portalelevate.com
Mar 03 19:59:05 mod_bosh        info    New BOSH session, assigned it sid '9efbe5b2-700c-4128-8869-0c7955fff361'
Mar 03 19:59:06 bosh9efbe5b2-700c-4128-8869-0c7955fff361        info    Authenticated as 20397d28-203c-40a9-9148-efd6cc94be5d@mediaserver.portalelevate.com
Mar 03 19:59:06 mod_bosh        info    New BOSH session, assigned it sid '77991dd8-0745-418a-9c80-2204c609148c'
Mar 03 19:59:06 mod_bosh        info    New BOSH session, assigned it sid '31121d1d-5aaf-48bd-9390-ecaf202b3b18'
Mar 03 19:59:06 bosh31121d1d-5aaf-48bd-9390-ecaf202b3b18        info    Authenticated as 628e73e9-8c7b-4360-806f-1fbf0c15b9dd@mediaserver.portalelevate.com
Mar 03 19:59:07 mod_bosh        info    New BOSH session, assigned it sid '398227cf-f2cb-44b4-b45e-184d273f818b'
Mar 03 19:59:07 bosh77991dd8-0745-418a-9c80-2204c609148c        info    Authenticated as 09ab0946-611b-4d00-8cd0-ceb8b3492e53@mediaserver.portalelevate.com
Mar 03 19:59:07 bosh398227cf-f2cb-44b4-b45e-184d273f818b        info    Authenticated as 74b77cde-4c4f-4796-9732-16fa1cbee921@mediaserver.portalelevate.com
Mar 03 19:59:07 mod_bosh        info    New BOSH session, assigned it sid 'e5a8b1d4-d7f4-4a06-9933-288c9d879e67'
Mar 03 19:59:07 boshe5a8b1d4-d7f4-4a06-9933-288c9d879e67        info    Authenticated as 9a6705db-7a66-4887-8e86-16f09c9c6d85@mediaserver.portalelevate.com
Mar 03 19:59:08 mod_bosh        info    New BOSH session, assigned it sid 'f28dec49-c349-4aa3-af83-0337711a1f78'
Mar 03 19:59:08 boshf28dec49-c349-4aa3-af83-0337711a1f78        info    Authenticated as e3cef153-7f8e-4da1-bdd4-c4308c127ec3@mediaserver.portalelevate.com
Mar 03 19:59:08 mod_bosh        info    New BOSH session, assigned it sid '9e9a8497-8fc3-4fcb-ab41-14dfba89f8cb'
Mar 03 19:59:09 bosh9e9a8497-8fc3-4fcb-ab41-14dfba89f8cb        info    Authenticated as 39781526-8ba0-42b9-91d8-d060462f4219@mediaserver.portalelevate.com
Mar 03 19:59:09 boshbc50bc57-5572-42bf-96c5-5f24d16b3a4f        info    Authenticated as 8c14b9c9-b6b0-451d-b8b8-8c32da5c6ac1@mediaserver.portalelevate.com
Mar 03 19:59:12 mod_bosh        info    Client tried to use sid 'f40eb7f7-29df-4403-ae07-60bb2ffbd825' which we don't know about
Mar 03 19:59:14 mod_bosh        info    New BOSH session, assigned it sid '599843a1-37c2-4b0e-9a89-0cbe275895d4'
Mar 03 19:59:14 bosh599843a1-37c2-4b0e-9a89-0cbe275895d4        info    Authenticated as 7deef8e0-47f9-40f6-a866-93a3c46499f5@mediaserver.portalelevate.com
Mar 03 19:59:15 mod_bosh        info    New BOSH session, assigned it sid '1be39a88-2d30-4126-a418-3c7c93f2fa72'
Mar 03 19:59:16 bosh1be39a88-2d30-4126-a418-3c7c93f2fa72        info    Authenticated as db81a69a-eae4-46a2-9b95-de88670f1f8d@mediaserver.portalelevate.com
Mar 03 19:59:23 mod_bosh        info    New BOSH session, assigned it sid 'd4f1fc64-09b8-4368-95f4-49f04fd5d0b7'
Mar 03 19:59:23 boshd4f1fc64-09b8-4368-95f4-49f04fd5d0b7        info    Authenticated as e38c74e4-705a-428c-8b0b-7dbe70d1ecd2@mediaserver.portalelevate.com
Mar 03 19:59:28 bosh1f0fb93c-59a2-4ba8-8dc7-63156c723c8f        info    Authenticated as 8a89c1db-cb67-4917-949b-5bb8fa2a3811@mediaserver.portalelevate.com
Mar 03 19:59:31 mod_bosh        info    New BOSH session, assigned it sid 'a928915c-72d8-45d5-ac3b-6c6201678e3a'
Mar 03 19:59:31 bosha928915c-72d8-45d5-ac3b-6c6201678e3a        info    Authenticated as ce1ca9f4-bf00-42c1-a526-a4deb986080c@mediaserver.portalelevate.com
Mar 03 19:59:34 bosh24a227b2-8672-49f0-ad73-4d85e0ff9f88        info    Authenticated as 7ad42c51-1805-4817-b250-a7177e701014@mediaserver.portalelevate.com
Mar 03 19:59:37 mod_bosh        info    New BOSH session, assigned it sid 'd41398dd-ed34-4809-a5cb-9d726926f5ef'
Mar 03 19:59:37 boshd41398dd-ed34-4809-a5cb-9d726926f5ef        info    Authenticated as 708fd6b1-9d85-46f3-9841-ebf428de2cdf@mediaserver.portalelevate.com
Mar 03 19:59:40 mod_bosh        info    New BOSH session, assigned it sid '74eb82e4-2a80-4722-8bd4-c138d60cc411'
Mar 03 19:59:40 mod_bosh        info    New BOSH session, assigned it sid '73be81a8-9390-4f1f-9505-43e13e01171b'
Mar 03 19:59:40 bosh74eb82e4-2a80-4722-8bd4-c138d60cc411        info    Authenticated as 9f6f28d8-c988-4a86-bf97-d136bf99129c@mediaserver.portalelevate.com
Mar 03 19:59:40 bosh73be81a8-9390-4f1f-9505-43e13e01171b        info    Authenticated as dc6c2ba9-2a81-4057-a19c-50f26de90c80@mediaserver.portalelevate.com
Mar 03 19:59:49 mod_bosh        info    New BOSH session, assigned it sid '8fc29426-2a0c-41fe-a433-faf96e13bb3b'
Mar 03 19:59:49 bosh8fc29426-2a0c-41fe-a433-faf96e13bb3b        info    Authenticated as 6c93fbdf-a980-434d-91b9-59d36b666289@mediaserver.portalelevate.com
Mar 03 19:59:52 mod_bosh        info    New BOSH session, assigned it sid '2dead054-cc23-4a53-92eb-ad92535d554c'
Mar 03 19:59:52 bosh2dead054-cc23-4a53-92eb-ad92535d554c        info    Authenticated as b1a04a7f-8627-4925-9ad0-d87cc95e4edd@mediaserver.portalelevate.com
Mar 03 19:59:58 mod_bosh        info    New BOSH session, assigned it sid '3b71ddc7-97ce-4000-ae53-706a51a9d379'
Mar 03 19:59:58 bosh3b71ddc7-97ce-4000-ae53-706a51a9d379        info    Authenticated as cd59d67e-1e17-4dea-961c-65ca2fb8d42c@mediaserver.portalelevate.com
Mar 03 19:59:59 bosh1074f6d6-cf3e-4be6-bada-24957893d377        info    Authenticated as d9868235-1588-4bac-9e9f-ce71e771e5e7@mediaserver.portalelevate.com
Mar 03 20:00:01 mod_bosh        info    New BOSH session, assigned it sid '4e6850c0-7fed-4d55-a510-51ba5e0acb98'
Mar 03 20:00:01 bosh4e6850c0-7fed-4d55-a510-51ba5e0acb98        info    Authenticated as 591448c3-87d0-440b-b8e2-42ce57b29f4d@mediaserver.portalelevate.com
Mar 03 20:00:12 mod_bosh        info    New BOSH session, assigned it sid '75a0408a-4eb2-4038-9fe1-23fae3f06e47'
Mar 03 20:00:12 bosh75a0408a-4eb2-4038-9fe1-23fae3f06e47        info    Authenticated as 55300c7c-b06f-4017-8823-c64ea3df7d6b@mediaserver.portalelevate.com
Mar 03 20:00:39 mod_bosh        info    New BOSH session, assigned it sid '1dadd46a-84d0-4e99-b858-0a4ea4a6f8dc'
Mar 03 20:00:40 bosh1dadd46a-84d0-4e99-b858-0a4ea4a6f8dc        info    Authenticated as 2bdc3769-4dd7-4be5-82ad-ace5669b29e1@mediaserver.portalelevate.com
Mar 03 20:00:47 mod_bosh        info    New BOSH session, assigned it sid 'd0596eba-90ff-4889-a7cf-43a5983133fe'
Mar 03 20:00:48 boshd0596eba-90ff-4889-a7cf-43a5983133fe        info    Authenticated as fd1eb60d-c3e5-4b44-83ee-d27cc6bef471@mediaserver.portalelevate.com
Mar 03 20:00:51 mod_bosh        info    New BOSH session, assigned it sid '04d2ffec-4ca7-47cc-8bd9-7ae2bf9dccbd'
Mar 03 20:00:51 bosh04d2ffec-4ca7-47cc-8bd9-7ae2bf9dccbd        info    Authenticated as 920a654e-b2da-4241-b54a-574d5f5a5e2d@mediaserver.portalelevate.com
Mar 03 20:00:52 mod_bosh        info    New BOSH session, assigned it sid 'd14f4eff-aaa8-43b5-84b1-79d2a5427c5c'
Mar 03 20:00:52 boshd14f4eff-aaa8-43b5-84b1-79d2a5427c5c        info    Authenticated as 392c6779-97bd-4697-9d7c-498ed3b205db@mediaserver.portalelevate.com
Mar 03 20:00:53 mod_bosh        info    New BOSH session, assigned it sid '47bd0192-23a9-4ca6-a626-4d93513ae0fe'
Mar 03 20:00:54 bosh47bd0192-23a9-4ca6-a626-4d93513ae0fe        info    Authenticated as 0f396361-45fa-40b7-81a5-d2d7818b390d@mediaserver.portalelevate.com
Mar 03 20:01:01 mod_bosh        info    New BOSH session, assigned it sid '986418ea-4c7f-4897-964c-513a99a513a9'
Mar 03 20:01:01 bosh986418ea-4c7f-4897-964c-513a99a513a9        info    Authenticated as 0584af04-00e7-4512-890a-287ed48eecde@mediaserver.portalelevate.com
Mar 03 20:01:03 mod_bosh        info    New BOSH session, assigned it sid '38bbb5ff-1aa5-49f0-b8ef-54d6e7bbcc43'
Mar 03 20:01:03 bosh38bbb5ff-1aa5-49f0-b8ef-54d6e7bbcc43        info    Authenticated as 088a7137-8284-457d-a5b8-7724c3b07a67@mediaserver.portalelevate.com
Mar 03 20:01:08 mod_bosh        info    Client tried to use sid 'bc50bc57-5572-42bf-96c5-5f24d16b3a4f' which we don't know about
Mar 03 20:01:12 mod_bosh        info    Client tried to use sid '398227cf-f2cb-44b4-b45e-184d273f818b' which we don't know about
Mar 03 20:01:15 mod_bosh        info    New BOSH session, assigned it sid '0dc0e338-95d3-452f-93f2-1154232eaf54'
Mar 03 20:01:15 bosh0dc0e338-95d3-452f-93f2-1154232eaf54        info    Authenticated as 60e5d835-2a68-4ca6-a695-4ba39045af4c@mediaserver.portalelevate.com
Mar 03 20:01:20 mod_bosh        info    New BOSH session, assigned it sid '7202de6b-9fc4-4371-97b5-1715917219ed'
Mar 03 20:01:22 bosh7202de6b-9fc4-4371-97b5-1715917219ed        info    Authenticated as 1df39573-687d-4196-9ddb-b0b22e3a8b23@mediaserver.portalelevate.com
Mar 03 20:01:28 mod_bosh        info    New BOSH session, assigned it sid '94d17b47-8134-4ca1-9287-16eb11625cfc'
Mar 03 20:01:33 mod_bosh        info    New BOSH session, assigned it sid '749f4337-8fbb-441a-b40f-fb1cc9971fcf'
Mar 03 20:01:33 bosh94d17b47-8134-4ca1-9287-16eb11625cfc        info    Authenticated as d5a70c2a-3a10-4628-8eb1-cec0dd9661d4@mediaserver.portalelevate.com
Mar 03 20:01:36 mod_bosh        info    New BOSH session, assigned it sid '9b0fb901-6f87-44f9-b94a-88ff002cada7'
Mar 03 20:01:36 bosh9b0fb901-6f87-44f9-b94a-88ff002cada7        info    Authenticated as aba279cd-4f60-4b17-bc65-a58e31c049a2@mediaserver.portalelevate.com
Mar 03 20:01:52 mod_bosh        info    New BOSH session, assigned it sid '359cca14-d0fa-4aeb-9b68-63875e47950e'
Mar 03 20:01:52 bosh359cca14-d0fa-4aeb-9b68-63875e47950e        info    Authenticated as 5501082c-336c-435e-9355-e1e750c7cd29@mediaserver.portalelevate.com
Mar 03 20:02:08 mod_bosh        info    New BOSH session, assigned it sid '661848ec-af52-4a64-9978-27c4cd501c90'
Mar 03 20:02:09 bosh661848ec-af52-4a64-9978-27c4cd501c90        info    Authenticated as ca18f914-59b3-480d-b2bf-f7284e610c0c@mediaserver.portalelevate.com
Mar 03 20:02:14 bosh749f4337-8fbb-441a-b40f-fb1cc9971fcf        info    Authenticated as 82ee7714-e9f0-41d9-a03e-0a97dd87c760@mediaserver.portalelevate.com
Mar 03 20:02:56 mod_bosh        info    New BOSH session, assigned it sid 'e21b818d-60b1-44ed-a3c2-569e53c861b4'
Mar 03 20:02:57 boshe21b818d-60b1-44ed-a3c2-569e53c861b4        info    Authenticated as bf986549-f765-4267-bb68-5dda93fe8ee4@mediaserver.portalelevate.com
Mar 03 20:03:09 mod_bosh        info    New BOSH session, assigned it sid '022c9b7d-55b2-4a9f-99ad-d2f3c816c786'
Mar 03 20:03:09 bosh022c9b7d-55b2-4a9f-99ad-d2f3c816c786        info    Authenticated as 41a5073b-0c1a-470a-b213-45267c550769@mediaserver.portalelevate.com
Mar 03 20:03:53 mod_bosh        info    New BOSH session, assigned it sid '4ac649c1-348a-41df-8d8e-9d68f5f88726'
Mar 03 20:03:53 bosh4ac649c1-348a-41df-8d8e-9d68f5f88726        info    Authenticated as 6bad3b49-10c1-4759-ad81-f8e88ce6c69d@mediaserver.portalelevate.com
Mar 03 20:04:23 mod_bosh        info    New BOSH session, assigned it sid 'e43913b4-e218-461c-9566-51a2df62542d'
Mar 03 20:04:23 boshe43913b4-e218-461c-9566-51a2df62542d        info    Authenticated as 88640e55-4794-4a84-9d3f-1a01422f67f8@mediaserver.portalelevate.com
Mar 03 20:04:35 mod_bosh        info    New BOSH session, assigned it sid '337e5a5f-bab9-459b-bd1d-3f760894c18e'
Mar 03 20:04:35 bosh337e5a5f-bab9-459b-bd1d-3f760894c18e        info    Authenticated as aee3b5a9-1978-4d1a-b33c-5fabc0f5b963@mediaserver.portalelevate.com
1 Like

We also experience the same behavior today and we google everywhere didn’t seem to find any fix. So workaround might be restart prosody at low traffic for now. ;((((

1 Like

Maybe related:
https://issues.prosody.im/1312

We discussed this on prosody mailing list:
https://groups.google.com/forum/#!topic/prosody-users/qb3bynuJJVc

I will try using prosody 11.05 with jemalloc.

Did using jemalloc help? I’m using 0.11.10 and seeing lua eating a lot of memory.
Haven’t figured out how to use jemalloc with prosody yet.

I still see this on multiple installations. At least on prosody version 0.11.10 and 0.11.13

Have tried to add rate limits and configure garbage collector to use incremental mode as documented here:
https://prosody.im/doc/advanced_gc

Anyone else seeing this or is my configuration somehow wrong?

I have the same memory problem with prosody (0.11.13) - and eventually it would trigger oomkiller on jicofo.

I have tried using jemalloc but it doesn’t help. I’m guessing some native code is keeping things from lua’s gc.