Prosody leaking memory

We were running sessions as usual and realized that prosody spiked memory usage without any reason.Usually the memory usage is around 2 gb for 56 participants

> JVB 2020-03-03 20:00:10.718 INFO: [26] org.jitsi.videobridge.Videobridge.log() CAT=stat create_conf,conf_id=80b90048dec252e3 conf_name=null,logging=false,conf_count=4,ch_count=168,v_streams=78

But we saw a dramatic increase in prosody memory consumption
Please find snapshot

Also these are prosody logs

Mar 03 19:59:02 bosh0be78c67-5153-4a14-8707-29c317257ee7        info    Authenticated as 707e6269-9e33-4523-a78b-dc0fae52138f@mediaserver.portalelevate.com
Mar 03 19:59:02 boshcc488e8a-eb91-4bc2-9762-2090f97b94bc        info    Authenticated as f63240fc-b680-4cde-bfc0-21b7815e2879@mediaserver.portalelevate.com
Mar 03 19:59:02 mod_bosh        info    New BOSH session, assigned it sid '73e79e6c-e8d3-4cee-a47e-f41e395c2e8a'
Mar 03 19:59:02 mod_bosh        info    New BOSH session, assigned it sid '1074f6d6-cf3e-4be6-bada-24957893d377'
Mar 03 19:59:02 mod_bosh        info    New BOSH session, assigned it sid 'f1ed43cc-c719-47ef-991c-ddd97de462fa'
Mar 03 19:59:03 mod_bosh        info    New BOSH session, assigned it sid 'adf98217-6563-44ea-b8cf-14a63e59bb8e'
Mar 03 19:59:03 bosh73e79e6c-e8d3-4cee-a47e-f41e395c2e8a        info    Authenticated as ee388e48-05d4-436a-bf93-adb400f6a52e@mediaserver.portalelevate.com
Mar 03 19:59:03 mod_bosh        info    New BOSH session, assigned it sid '1f0fb93c-59a2-4ba8-8dc7-63156c723c8f'
Mar 03 19:59:03 boshf1ed43cc-c719-47ef-991c-ddd97de462fa        info    Authenticated as 54fb659f-e66d-47d3-8620-43f83167e27e@mediaserver.portalelevate.com
Mar 03 19:59:03 mod_bosh        info    New BOSH session, assigned it sid 'bc50bc57-5572-42bf-96c5-5f24d16b3a4f'
Mar 03 19:59:03 boshadf98217-6563-44ea-b8cf-14a63e59bb8e        info    Authenticated as 4fa9a743-8f89-46cd-ac54-101f2d838d4f@mediaserver.portalelevate.com
Mar 03 19:59:04 mod_bosh        info    New BOSH session, assigned it sid '6806a915-3667-4fab-aa6c-bed7b2f63840'
Mar 03 19:59:04 mod_bosh        info    New BOSH session, assigned it sid 'c2b0cc91-afc1-4b7a-8b8e-3de0726be0ba'
Mar 03 19:59:04 bosh6806a915-3667-4fab-aa6c-bed7b2f63840        info    Authenticated as 5c7a0360-6143-46b0-ac57-d541fe8d39d7@mediaserver.portalelevate.com
Mar 03 19:59:04 boshc2b0cc91-afc1-4b7a-8b8e-3de0726be0ba        info    Authenticated as fbe05523-6a4b-455e-84ae-27544030c646@mediaserver.portalelevate.com
Mar 03 19:59:04 mod_bosh        info    New BOSH session, assigned it sid '220153fa-bd8d-4f12-b2b0-0a438268c1a2'
Mar 03 19:59:05 bosh220153fa-bd8d-4f12-b2b0-0a438268c1a2        info    Authenticated as 5b972570-a57c-4536-a823-59750b462fb8@mediaserver.portalelevate.com
Mar 03 19:59:05 mod_bosh        info    New BOSH session, assigned it sid '9efbe5b2-700c-4128-8869-0c7955fff361'
Mar 03 19:59:06 bosh9efbe5b2-700c-4128-8869-0c7955fff361        info    Authenticated as 20397d28-203c-40a9-9148-efd6cc94be5d@mediaserver.portalelevate.com
Mar 03 19:59:06 mod_bosh        info    New BOSH session, assigned it sid '77991dd8-0745-418a-9c80-2204c609148c'
Mar 03 19:59:06 mod_bosh        info    New BOSH session, assigned it sid '31121d1d-5aaf-48bd-9390-ecaf202b3b18'
Mar 03 19:59:06 bosh31121d1d-5aaf-48bd-9390-ecaf202b3b18        info    Authenticated as 628e73e9-8c7b-4360-806f-1fbf0c15b9dd@mediaserver.portalelevate.com
Mar 03 19:59:07 mod_bosh        info    New BOSH session, assigned it sid '398227cf-f2cb-44b4-b45e-184d273f818b'
Mar 03 19:59:07 bosh77991dd8-0745-418a-9c80-2204c609148c        info    Authenticated as 09ab0946-611b-4d00-8cd0-ceb8b3492e53@mediaserver.portalelevate.com
Mar 03 19:59:07 bosh398227cf-f2cb-44b4-b45e-184d273f818b        info    Authenticated as 74b77cde-4c4f-4796-9732-16fa1cbee921@mediaserver.portalelevate.com
Mar 03 19:59:07 mod_bosh        info    New BOSH session, assigned it sid 'e5a8b1d4-d7f4-4a06-9933-288c9d879e67'
Mar 03 19:59:07 boshe5a8b1d4-d7f4-4a06-9933-288c9d879e67        info    Authenticated as 9a6705db-7a66-4887-8e86-16f09c9c6d85@mediaserver.portalelevate.com
Mar 03 19:59:08 mod_bosh        info    New BOSH session, assigned it sid 'f28dec49-c349-4aa3-af83-0337711a1f78'
Mar 03 19:59:08 boshf28dec49-c349-4aa3-af83-0337711a1f78        info    Authenticated as e3cef153-7f8e-4da1-bdd4-c4308c127ec3@mediaserver.portalelevate.com
Mar 03 19:59:08 mod_bosh        info    New BOSH session, assigned it sid '9e9a8497-8fc3-4fcb-ab41-14dfba89f8cb'
Mar 03 19:59:09 bosh9e9a8497-8fc3-4fcb-ab41-14dfba89f8cb        info    Authenticated as 39781526-8ba0-42b9-91d8-d060462f4219@mediaserver.portalelevate.com
Mar 03 19:59:09 boshbc50bc57-5572-42bf-96c5-5f24d16b3a4f        info    Authenticated as 8c14b9c9-b6b0-451d-b8b8-8c32da5c6ac1@mediaserver.portalelevate.com
Mar 03 19:59:12 mod_bosh        info    Client tried to use sid 'f40eb7f7-29df-4403-ae07-60bb2ffbd825' which we don't know about
Mar 03 19:59:14 mod_bosh        info    New BOSH session, assigned it sid '599843a1-37c2-4b0e-9a89-0cbe275895d4'
Mar 03 19:59:14 bosh599843a1-37c2-4b0e-9a89-0cbe275895d4        info    Authenticated as 7deef8e0-47f9-40f6-a866-93a3c46499f5@mediaserver.portalelevate.com
Mar 03 19:59:15 mod_bosh        info    New BOSH session, assigned it sid '1be39a88-2d30-4126-a418-3c7c93f2fa72'
Mar 03 19:59:16 bosh1be39a88-2d30-4126-a418-3c7c93f2fa72        info    Authenticated as db81a69a-eae4-46a2-9b95-de88670f1f8d@mediaserver.portalelevate.com
Mar 03 19:59:23 mod_bosh        info    New BOSH session, assigned it sid 'd4f1fc64-09b8-4368-95f4-49f04fd5d0b7'
Mar 03 19:59:23 boshd4f1fc64-09b8-4368-95f4-49f04fd5d0b7        info    Authenticated as e38c74e4-705a-428c-8b0b-7dbe70d1ecd2@mediaserver.portalelevate.com
Mar 03 19:59:28 bosh1f0fb93c-59a2-4ba8-8dc7-63156c723c8f        info    Authenticated as 8a89c1db-cb67-4917-949b-5bb8fa2a3811@mediaserver.portalelevate.com
Mar 03 19:59:31 mod_bosh        info    New BOSH session, assigned it sid 'a928915c-72d8-45d5-ac3b-6c6201678e3a'
Mar 03 19:59:31 bosha928915c-72d8-45d5-ac3b-6c6201678e3a        info    Authenticated as ce1ca9f4-bf00-42c1-a526-a4deb986080c@mediaserver.portalelevate.com
Mar 03 19:59:34 bosh24a227b2-8672-49f0-ad73-4d85e0ff9f88        info    Authenticated as 7ad42c51-1805-4817-b250-a7177e701014@mediaserver.portalelevate.com
Mar 03 19:59:37 mod_bosh        info    New BOSH session, assigned it sid 'd41398dd-ed34-4809-a5cb-9d726926f5ef'
Mar 03 19:59:37 boshd41398dd-ed34-4809-a5cb-9d726926f5ef        info    Authenticated as 708fd6b1-9d85-46f3-9841-ebf428de2cdf@mediaserver.portalelevate.com
Mar 03 19:59:40 mod_bosh        info    New BOSH session, assigned it sid '74eb82e4-2a80-4722-8bd4-c138d60cc411'
Mar 03 19:59:40 mod_bosh        info    New BOSH session, assigned it sid '73be81a8-9390-4f1f-9505-43e13e01171b'
Mar 03 19:59:40 bosh74eb82e4-2a80-4722-8bd4-c138d60cc411        info    Authenticated as 9f6f28d8-c988-4a86-bf97-d136bf99129c@mediaserver.portalelevate.com
Mar 03 19:59:40 bosh73be81a8-9390-4f1f-9505-43e13e01171b        info    Authenticated as dc6c2ba9-2a81-4057-a19c-50f26de90c80@mediaserver.portalelevate.com
Mar 03 19:59:49 mod_bosh        info    New BOSH session, assigned it sid '8fc29426-2a0c-41fe-a433-faf96e13bb3b'
Mar 03 19:59:49 bosh8fc29426-2a0c-41fe-a433-faf96e13bb3b        info    Authenticated as 6c93fbdf-a980-434d-91b9-59d36b666289@mediaserver.portalelevate.com
Mar 03 19:59:52 mod_bosh        info    New BOSH session, assigned it sid '2dead054-cc23-4a53-92eb-ad92535d554c'
Mar 03 19:59:52 bosh2dead054-cc23-4a53-92eb-ad92535d554c        info    Authenticated as b1a04a7f-8627-4925-9ad0-d87cc95e4edd@mediaserver.portalelevate.com
Mar 03 19:59:58 mod_bosh        info    New BOSH session, assigned it sid '3b71ddc7-97ce-4000-ae53-706a51a9d379'
Mar 03 19:59:58 bosh3b71ddc7-97ce-4000-ae53-706a51a9d379        info    Authenticated as cd59d67e-1e17-4dea-961c-65ca2fb8d42c@mediaserver.portalelevate.com
Mar 03 19:59:59 bosh1074f6d6-cf3e-4be6-bada-24957893d377        info    Authenticated as d9868235-1588-4bac-9e9f-ce71e771e5e7@mediaserver.portalelevate.com
Mar 03 20:00:01 mod_bosh        info    New BOSH session, assigned it sid '4e6850c0-7fed-4d55-a510-51ba5e0acb98'
Mar 03 20:00:01 bosh4e6850c0-7fed-4d55-a510-51ba5e0acb98        info    Authenticated as 591448c3-87d0-440b-b8e2-42ce57b29f4d@mediaserver.portalelevate.com
Mar 03 20:00:12 mod_bosh        info    New BOSH session, assigned it sid '75a0408a-4eb2-4038-9fe1-23fae3f06e47'
Mar 03 20:00:12 bosh75a0408a-4eb2-4038-9fe1-23fae3f06e47        info    Authenticated as 55300c7c-b06f-4017-8823-c64ea3df7d6b@mediaserver.portalelevate.com
Mar 03 20:00:39 mod_bosh        info    New BOSH session, assigned it sid '1dadd46a-84d0-4e99-b858-0a4ea4a6f8dc'
Mar 03 20:00:40 bosh1dadd46a-84d0-4e99-b858-0a4ea4a6f8dc        info    Authenticated as 2bdc3769-4dd7-4be5-82ad-ace5669b29e1@mediaserver.portalelevate.com
Mar 03 20:00:47 mod_bosh        info    New BOSH session, assigned it sid 'd0596eba-90ff-4889-a7cf-43a5983133fe'
Mar 03 20:00:48 boshd0596eba-90ff-4889-a7cf-43a5983133fe        info    Authenticated as fd1eb60d-c3e5-4b44-83ee-d27cc6bef471@mediaserver.portalelevate.com
Mar 03 20:00:51 mod_bosh        info    New BOSH session, assigned it sid '04d2ffec-4ca7-47cc-8bd9-7ae2bf9dccbd'
Mar 03 20:00:51 bosh04d2ffec-4ca7-47cc-8bd9-7ae2bf9dccbd        info    Authenticated as 920a654e-b2da-4241-b54a-574d5f5a5e2d@mediaserver.portalelevate.com
Mar 03 20:00:52 mod_bosh        info    New BOSH session, assigned it sid 'd14f4eff-aaa8-43b5-84b1-79d2a5427c5c'
Mar 03 20:00:52 boshd14f4eff-aaa8-43b5-84b1-79d2a5427c5c        info    Authenticated as 392c6779-97bd-4697-9d7c-498ed3b205db@mediaserver.portalelevate.com
Mar 03 20:00:53 mod_bosh        info    New BOSH session, assigned it sid '47bd0192-23a9-4ca6-a626-4d93513ae0fe'
Mar 03 20:00:54 bosh47bd0192-23a9-4ca6-a626-4d93513ae0fe        info    Authenticated as 0f396361-45fa-40b7-81a5-d2d7818b390d@mediaserver.portalelevate.com
Mar 03 20:01:01 mod_bosh        info    New BOSH session, assigned it sid '986418ea-4c7f-4897-964c-513a99a513a9'
Mar 03 20:01:01 bosh986418ea-4c7f-4897-964c-513a99a513a9        info    Authenticated as 0584af04-00e7-4512-890a-287ed48eecde@mediaserver.portalelevate.com
Mar 03 20:01:03 mod_bosh        info    New BOSH session, assigned it sid '38bbb5ff-1aa5-49f0-b8ef-54d6e7bbcc43'
Mar 03 20:01:03 bosh38bbb5ff-1aa5-49f0-b8ef-54d6e7bbcc43        info    Authenticated as 088a7137-8284-457d-a5b8-7724c3b07a67@mediaserver.portalelevate.com
Mar 03 20:01:08 mod_bosh        info    Client tried to use sid 'bc50bc57-5572-42bf-96c5-5f24d16b3a4f' which we don't know about
Mar 03 20:01:12 mod_bosh        info    Client tried to use sid '398227cf-f2cb-44b4-b45e-184d273f818b' which we don't know about
Mar 03 20:01:15 mod_bosh        info    New BOSH session, assigned it sid '0dc0e338-95d3-452f-93f2-1154232eaf54'
Mar 03 20:01:15 bosh0dc0e338-95d3-452f-93f2-1154232eaf54        info    Authenticated as 60e5d835-2a68-4ca6-a695-4ba39045af4c@mediaserver.portalelevate.com
Mar 03 20:01:20 mod_bosh        info    New BOSH session, assigned it sid '7202de6b-9fc4-4371-97b5-1715917219ed'
Mar 03 20:01:22 bosh7202de6b-9fc4-4371-97b5-1715917219ed        info    Authenticated as 1df39573-687d-4196-9ddb-b0b22e3a8b23@mediaserver.portalelevate.com
Mar 03 20:01:28 mod_bosh        info    New BOSH session, assigned it sid '94d17b47-8134-4ca1-9287-16eb11625cfc'
Mar 03 20:01:33 mod_bosh        info    New BOSH session, assigned it sid '749f4337-8fbb-441a-b40f-fb1cc9971fcf'
Mar 03 20:01:33 bosh94d17b47-8134-4ca1-9287-16eb11625cfc        info    Authenticated as d5a70c2a-3a10-4628-8eb1-cec0dd9661d4@mediaserver.portalelevate.com
Mar 03 20:01:36 mod_bosh        info    New BOSH session, assigned it sid '9b0fb901-6f87-44f9-b94a-88ff002cada7'
Mar 03 20:01:36 bosh9b0fb901-6f87-44f9-b94a-88ff002cada7        info    Authenticated as aba279cd-4f60-4b17-bc65-a58e31c049a2@mediaserver.portalelevate.com
Mar 03 20:01:52 mod_bosh        info    New BOSH session, assigned it sid '359cca14-d0fa-4aeb-9b68-63875e47950e'
Mar 03 20:01:52 bosh359cca14-d0fa-4aeb-9b68-63875e47950e        info    Authenticated as 5501082c-336c-435e-9355-e1e750c7cd29@mediaserver.portalelevate.com
Mar 03 20:02:08 mod_bosh        info    New BOSH session, assigned it sid '661848ec-af52-4a64-9978-27c4cd501c90'
Mar 03 20:02:09 bosh661848ec-af52-4a64-9978-27c4cd501c90        info    Authenticated as ca18f914-59b3-480d-b2bf-f7284e610c0c@mediaserver.portalelevate.com
Mar 03 20:02:14 bosh749f4337-8fbb-441a-b40f-fb1cc9971fcf        info    Authenticated as 82ee7714-e9f0-41d9-a03e-0a97dd87c760@mediaserver.portalelevate.com
Mar 03 20:02:56 mod_bosh        info    New BOSH session, assigned it sid 'e21b818d-60b1-44ed-a3c2-569e53c861b4'
Mar 03 20:02:57 boshe21b818d-60b1-44ed-a3c2-569e53c861b4        info    Authenticated as bf986549-f765-4267-bb68-5dda93fe8ee4@mediaserver.portalelevate.com
Mar 03 20:03:09 mod_bosh        info    New BOSH session, assigned it sid '022c9b7d-55b2-4a9f-99ad-d2f3c816c786'
Mar 03 20:03:09 bosh022c9b7d-55b2-4a9f-99ad-d2f3c816c786        info    Authenticated as 41a5073b-0c1a-470a-b213-45267c550769@mediaserver.portalelevate.com
Mar 03 20:03:53 mod_bosh        info    New BOSH session, assigned it sid '4ac649c1-348a-41df-8d8e-9d68f5f88726'
Mar 03 20:03:53 bosh4ac649c1-348a-41df-8d8e-9d68f5f88726        info    Authenticated as 6bad3b49-10c1-4759-ad81-f8e88ce6c69d@mediaserver.portalelevate.com
Mar 03 20:04:23 mod_bosh        info    New BOSH session, assigned it sid 'e43913b4-e218-461c-9566-51a2df62542d'
Mar 03 20:04:23 boshe43913b4-e218-461c-9566-51a2df62542d        info    Authenticated as 88640e55-4794-4a84-9d3f-1a01422f67f8@mediaserver.portalelevate.com
Mar 03 20:04:35 mod_bosh        info    New BOSH session, assigned it sid '337e5a5f-bab9-459b-bd1d-3f760894c18e'
Mar 03 20:04:35 bosh337e5a5f-bab9-459b-bd1d-3f760894c18e        info    Authenticated as aee3b5a9-1978-4d1a-b33c-5fabc0f5b963@mediaserver.portalelevate.com
1 Like

We also experience the same behavior today and we google everywhere didn’t seem to find any fix. So workaround might be restart prosody at low traffic for now. ;((((

1 Like

Maybe related:
https://issues.prosody.im/1312

We discussed this on prosody mailing list:
https://groups.google.com/forum/#!topic/prosody-users/qb3bynuJJVc

I will try using prosody 11.05 with jemalloc.

Did using jemalloc help? I’m using 0.11.10 and seeing lua eating a lot of memory.
Haven’t figured out how to use jemalloc with prosody yet.