Prosody 0.11.4 crashes having 100+ confs

Hello, guys.
I’m am struggling with a constant prosody error leading to having “Ooops, something went wrong” message on the client-side.

Under a certain load around 100 conferences per server prosody starts sending the following message

    Oct 11 12:04:37 c2s562f389460a0 error   Traceback[c2s]:         /usr/lib/prosody/util/serialization.lua:38: Can't serialize table: table has multiple references
stack traceback:
        [C]: in function 'error'
        /usr/lib/prosody/util/serialization.lua:38: in function 'fallback'
        /usr/lib/prosody/util/serialization.lua:135: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        ...
        /usr/lib/prosody/modules/muc/mod_muc.lua:158: in function 'on_evict'
        /usr/lib/prosody/util/cache.lua:57: in function 'set'
        /usr/lib/prosody/modules/muc/mod_muc.lua:176: in function 'track_room'
        /usr/lib/prosody/modules/muc/mod_muc.lua:346: in function '?'
        /usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
        (...tail calls...)
        /usr/lib/prosody/modules/muc/muc.lib.lua:442: in function </usr/lib/prosody/modules/muc/muc.lib.lua:431>
        (...tail calls...)
        /usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
        (...tail calls...)
        /usr/lib/prosody/core/stanza_router.lua:180: in function 'core_post_stanza'
        /usr/lib/prosody/core/stanza_router.lua:127: in function 'core_process_stanza'
        /usr/lib/prosody/modules/mod_c2s.lua:275: in function 'func'
        /usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125>

At the same time Jicofo has following in the log:

Jicofo 2020-10-11 12:04:26.800 WARNING: [10375] org.jitsi.xmpp.component.ComponentBase.log() PROCESSING TIME LIMIT EXCEEDED - it took 15003ms to process: <iq type="set" from="xxx@thedomain.com" id="1a8ff2df-6ff0-4551-bcca-5389c94810af:sendIQ" to="focus.thedomain.com"><conference xmlns="http://jitsi.org/protocol/focus" machine-uid="1233434212

Restarting prosody using service prosody restart takes long (1-2 minutes) in that case and helps for a while, but doesn’t solve the problem. After reducing load on the server it generally recovers.

Prosody version: 0.11.4, luaevent:cing 0.4.6
JVB uses octo

At the same time server doesn’t show critical RAM or CPU usage.
How can I solve that? And are there any guidelines for scaling prosody and jicofo? What are the best practices there?

Any luck figuring this out? We just started running into exactly the same problem.

Doesn’t appear to be directly correlated to the # of conferences, we recently had 5 simultaneous shards running with similar load numbers on each, and only started seeing this error on one of the 5.

Our prosody error is slightly different, as well:

Nov 07 00:33:57 mod_bosh        error   Traceback[bosh]: /usr/lib/prosody/util/serialization.lua:38: Can't serialize function
stack traceback:
        [C]: in function 'error'
        /usr/lib/prosody/util/serialization.lua:38: in function </usr/lib/prosody/util/serialization.lua:37>
        (...tail calls...)
        /usr/lib/prosody/util/serialization.lua:196: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'
        /usr/lib/prosody/util/serialization.lua:220: in function </usr/lib/prosody/util/serialization.lua:218>
        (...tail calls...)
        /usr/lib/prosody/modules/mod_storage_memory.lua:39: in function </usr/lib/prosody/modules/mod_storage_memory.lua:37>
        (...tail calls...)
        /usr/lib/prosody/modules/muc/mod_muc.lua:159: in function 'on_evict'
        /usr/lib/prosody/util/cache.lua:57: in function 'set'
        /usr/lib/prosody/modules/muc/mod_muc.lua:177: in function 'track_room'
        /usr/lib/prosody/modules/muc/mod_muc.lua:205: in function </usr/lib/prosody/modules/muc/mod_muc.lua:192>
        (...tail calls...)
        ...share/jitsi-meet/prosody-plugins/mod_muc_lobby_rooms.lua:225: in function '?'
        /usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
        (...tail calls...)
        /usr/lib/prosody/modules/mod_disco.lua:104: in function '?'
        /usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
        (...tail calls...)
        /usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
        (...tail calls...)
        /usr/lib/prosody/core/stanza_router.lua:180: in function 'core_post_stanza'
        /usr/lib/prosody/core/stanza_router.lua:127: in function 'dispatch_stanza'
        /usr/lib/prosody/modules/mod_bosh.lua:305: in function 'func'
        /usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125>
stack traceback:
        /usr/lib/prosody/util/async.lua:211: in function 'run'
        /usr/lib/prosody/modules/mod_bosh.lua:447: in function 'cb_handlestanza'
        /usr/lib/prosody/util/xmppstream.lua:182: in function </usr/lib/prosody/util/xmppstream.lua:162>
        [C]: in function 'parse'
        /usr/lib/prosody/util/xmppstream.lua:282: in function 'feed'
        /usr/lib/prosody/modules/mod_bosh.lua:133: in function '?'
        /usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
        (...tail calls...)
        /usr/lib/prosody/net/http/server.lua:229: in function </usr/lib/prosody/net/http/server.lua:176>
        [C]: in function 'xpcall'
        /usr/lib/prosody/net/http/server.lua:108: in function 'process_next'
        /usr/lib/prosody/net/http/server.lua:124: in function 'success_cb'
        /usr/lib/prosody/net/http/parser.lua:177: in function 'feed'
        /usr/lib/prosody/net/http/server.lua:155: in function </usr/lib/prosody/net/http/server.lua:154>
        [C]: in function 'pcall'
        /usr/lib/prosody/net/server_epoll.lua:159: in function 'on'
        /usr/lib/prosody/net/server_epoll.lua:348: in function 'onreadable'
        /usr/lib/prosody/net/server_epoll.lua:744: in function </usr/lib/prosody/net/server_epoll.lua:736>
        [C]: in function 'xpcall'
        /usr/bin/prosody:76: in function 'loop'
        /usr/bin/prosody:86: in main chunk
        [C]: in ?