Prosody 0.11.13 (lua5.2) vs 0.12.1 (lua 5.4) performance difference

Hello,

We are using docker-jitsi-meet prosody images with our custom plugins and cfg files.

With the same prosody.cfg.lua, jitsi.meet.cfg.lua, prosody-plugins and custom plugins folder (only difference are lua and prosody version) files I can join 400 participant in prosody 0.11.13 (7001 image) but while using prosody 0.12.1 (7868-3 image) I can only join 280 participant. After these numbers the room crashes.

Have jitsi enabled additional modules in 0.12.1 for better performance which I forget to add? What can be the cause for difference?

Do you have any recommendation?

Thank you in advance.
Best

Interesting!

Are you keeping the rest of the images consistent? Or are you running the whole stack in 2 different versions?

We only use prosody image, the other applications are installed on our server natively. the plugins are from version 7287, the other applications are in version 7287, we only use the prosody image for prosody and lua, and change the other files.

FROM jitsi/prosody:stable-7001

FROM jitsi/prosody:stable-7648-3

some → prosody cfg lua


modules_enabled = {

        -- Generally required
                "roster"; -- Allow users to have a roster. Recommended ;)
                "saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
                "tls"; -- Add support for secure TLS on c2s/s2s connections
                "dialback"; -- s2s dialback support
                "disco"; -- Service discovery

        -- Not essential, but recommended
                "private"; -- Private XML storage (for room bookmarks, etc.)
                --"vcard"; -- Allow users to set vCards
                "limits"; -- Enable bandwidth limiting for XMPP connections
                "vcard4"; -- User profiles (stored in PEP)
                "vcard_legacy"; -- Conversion between legacy vCard and PEP Avatar, vcard
                "blocklist"; -- Allow users to block communications with other users
                "carbons"; -- Keep multiple clients in sync



        -- These are commented by default as they have a performance impact
                --"privacy"; -- Support privacy lists
                --"compression"; -- Stream compression (Debian: requires lua-zlib module to work)

        -- Nice to have
                "version"; -- Replies to server version requests
                "uptime"; -- Report how long server has been running
                "time"; -- Let others know the time here on this server
                "ping"; -- Replies to XMPP pings with pongs
                "pep"; -- Enables users to publish their mood, activity, playing music and more
                "register"; -- Allow users to register on this server using a client and change passwords

        -- Admin interfaces
                "admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
                "admin_telnet"; -- Opens telnet console interface on localhost port 5582

        -- HTTP modules
                --"bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
                --"http_files"; -- Serve static files from a directory over HTTP

        -- Other specific functionality
                --"posix"; -- POSIX functionality, sends server to background, enables syslog, etc.
                --"groups"; -- Shared roster support
                --"announce"; -- Send announcement to all online users
                --"welcome"; -- Welcome users who register accounts
                --"watchregistrations"; -- Alert admins of registrations
                --"motd"; -- Send a message to users when they log in
                --"legacyauth"; -- Legacy authentication. Only used by some old clients and bots.


        "websocket";
        "smacks"; -- XEP-0198: Stream Management



};



allow_registration = false;

-- Enable rate limits for incoming client and server connections
limits = {
  c2s = {
    rate = "10kb/s";
  };
  s2sin = {
    rate = "30kb/s";
  };
}
network_backend = "epoll";
-- Set the TCP backlog to 511 since the kernel rounds it up to the next power of 2: 512.
network_settings = {
  tcp_backlog = 511;
}
unbound = {
    resolvconf = true
}

gc = {
    mode = "incremental";
    threshold = 105;
    speed = 250;
}

some jitsi cfg lua →


Component "conference.jitsi.com" "muc"
    storage = "memory"
    modules_enabled = {
        "muc_meeting_id";
"muc_persistent_password";
"muc_close_room";
"muc_room_subject";
"muc_package_properties";
"muc_recording";
"muc_room_subject";
"muc_mute_upon_entry";
"muc_cohost";
"muc_password_whitelist";
"muc_max_occupants";
        "token_verification";

        "muc_domain_mapper";
        }
    muc_room_cache_size = 1000
    muc_room_locking = false
    muc_room_default_public_jids = true
    muc_max_occupants=500

--

--smacks
smacks_max_unacked_stanzas = 3;
smacks_hibernation_time = 60;
smacks_max_hibernated_sessions = 1;
smacks_max_old_sessions = 1;

Is prosody taking 100% cpu when it crashes? Any errors in prosody logs? Are you using websockets?
You should not mix the versions that way.
We haven’t seen any performance issues between those versions, the contrary we have seen improvements with 0.12 especially for closing connections when using websockets.

yes for both cases it has 100% cpu usage when crashes and we use websocket,
we dont have any error in error level however

the first error message is this and others, main breaking points there are 107-108 unacked stansaz

Aug 25 11:46:43 http.server	debug	Firing event: GET /xmpp-websocket
Aug 25 11:46:43 c2s55df841e4840	info	Client connected
Aug 25 11:46:43 c2s55df841e4840	debug	Sending WebSocket handshake
Aug 25 11:46:43 mod_websocket	debug	Websocket received frame: opcode=8, 2 bytes
Aug 25 11:46:43 connIZVVYGZHNvWr	debug	Close after writing remaining buffered data
Aug 25 11:46:43 connIZVVYGZHNvWr	debug	Read error, closing (closed)
Aug 25 11:46:43 c2s55df81cb7960	info	Client disconnected: closed
Aug 25 11:46:43 c2s55df81cb7960	debug	Destroying session for bec74a1e-1864-469d-95a2-dba86b903a39@guest.meet.jitsi.com/dhxynyNH0UH5 (bec74a1e-1864-469d-95a2-dba86b903a39@guest.meet.jitsi.com): closed
Aug 25 11:46:43 c2s55df81cb7960	debug	mod_smacks hibernating session for up to 60 seconds
 Aug 25 11:46:44 c2s55df81800310	debug	Received[c2s]: <presence to='ali@conference.preprodtenant.meet.jitsi.com/56c83ebf' xml:lang='en' type='unavailable'>
Aug 25 11:46:44 conference.meet.jitsi.com:muc	debug	session 81f958f2-6c9f-45a8-9f78-7d58f00e301d@guest.meet.jitsi.com/vkz_EyQqZv2S is leaving occupant [preprodtenant]ali@conference.meet.jitsi.com/56c83ebf
Aug 25 11:46:44 c2s55df82cb0060	debug	Sending[c2s]: <presence to='d4cdc0b1-e6d3-4489-aa0c-aec7b32e3307@guest.meet.jitsi.com/jN1VRNZ93l6u' type='unavailable' xml:lang='en' from='[preprodtenant]ali@conference.meet.jitsi.com/56c83ebf'>
Aug 25 11:46:44 c2s55df81954620	debug	Sending[c2s]: <presence to='43306ead-dfab-4f1f-b9c0-576e3b97e669@guest.meet.jitsi.com/PaON8TwkBU9V' type='unavailable' xml:lang='en' from='ali@conference.preprodtenant.meet.jitsi.com/56c83ebf'>
Aug 25 11:46:44 c2s55df81bd5ab0	debug	Sending[c2s]: <presence to='e4121bb0-2c78-4c88-b7b3-5c5505d99758@guest.meet.jitsi.com/VICR_R5SS7ui' type='unavailable' xml:lang='en' from='ali@conference.preprodtenant.meet.jitsi.com/56c83ebf'>
Aug 25 11:46:50 c2s55df8420b9b0	debug	Sending[c2s]: <a xmlns='urn:xmpp:sm:3' h='5'>
Aug 25 11:46:50 c2s55df8420b9b0	debug	Handled 6 incoming stanzas
Aug 25 11:46:50 c2s55df8420b9b0	debug	Received[c2s]: <presence type='unavailable' xml:lang='en'>
Aug 25 11:46:50 c2s55df8420b9b0	debug	Sending[c2s]: <presence type='unavailable' xml:lang='en' from='9a675492-6256-41f5-b467-f2ea2d18c4ee@guest.meet.jitsi.com/OwWiTDN1xLNb'>
Aug 25 11:46:50 c2s55df8420b9b0	debug	Received </stream:stream>
Aug 25 11:46:50 c2s55df8420b9b0	debug	Sending[c2s]: <a xmlns='urn:xmpp:sm:3' h='6'>
Aug 25 11:46:50 c2s55df8420b9b0	debug	Sending[c2s]: <close xmlns='urn:ietf:params:xml:ns:xmpp-framing'>
Aug 25 11:46:50 c2s55df8420b9b0	debug	c2s stream for 9a675492-6256-41f5-b467-f2ea2d18c4ee@guest.meet.jitsi.com/OwWiTDN1xLNb closed: session closed
Aug 25 11:46:50 c2s55df8420b9b0	debug	Destroying session for 9a675492-6256-41f5-b467-f2ea2d18c4ee@guest.meet.jitsi.com/OwWiTDN1xLNb (9a675492-6256-41f5-b467-f2ea2d18c4ee@guest.meet.jitsi.com)
Aug 25 11:46:50 c2s55df8420b9b0	debug	Destroying session with 107 unacked stanzas
Aug 25 11:46:50 c2s55df8420b9b0	debug	Received[c2s]: <presence to='ali@conference.preprodtenant.meet.jitsi.com/focus' type='error' from='9a675492-6256-41f5-b467-f2ea2d18c4ee@guest.meet.jitsi.com/OwWiTDN1xLNb'>
Aug 25 11:46:50 stanzarouter	debug	Discarding unhandled error presence (cancel, recipient-unavailable) from c2s: <presence to='[preprodtenant]ali@conference.meet.jitsi.com/focus' type='error' from='9a675492-6256-41f5-b467-f2ea2d18c4ee@guest.meet.jitsi.com/OwWiTDN1xLNb'>
Aug 25 11:46:50 c2s55df80f11790	debug	Sending[c2s]: <a xmlns='urn:xmpp:sm:3' h='10'>
Aug 25 11:46:50 c2s55df80f11790	debug	Sending[c2s]: <close xmlns='urn:ietf:params:xml:ns:xmpp-framing'>
Aug 25 11:46:50 c2s55df80f11790	debug	c2s stream for 1b062c1d-1fa8-417a-8dc3-936d0ae31be2@guest.meet.jitsi.com/bSxIwLaYOLla closed: session closed
Aug 25 11:46:50 c2s55df80f11790	debug	Destroying session for 1b062c1d-1fa8-417a-8dc3-936d0ae31be2@guest.meet.jitsi.com/bSxIwLaYOLla (1b062c1d-1fa8-417a-8dc3-936d0ae31be2@guest.meet.jitsi.com)
Aug 25 11:46:50 c2s55df80f11790	debug	Destroying session with 108 unacked stanzas
Aug 25 11:46:50 c2s55df80f11790	debug	Received[c2s]: <presence to='ali@conference.preprodtenant.meet.jitsi.com/focus' type='error' from='1b062c1d-1fa8-417a-8dc3-936d0ae31be2@guest.meet.jitsi.com/bSxIwLaYOLla'>
Aug 25 11:46:50 stanzarouter	debug	Discarding unhandled error presence (cancel, recipient-unavailable) from c2s: <presence to='[preprodtenant]ali@conference.meet.jitsi.com/focus' type='error' from='1b062c1d-1fa8-417a-8dc3-936d0ae31be2@guest.meet.jitsi.com/bSxIwLaYOLla'>
Aug 25 11:46:50 c2s55df80f11790	debug	Received[c2s]: <presence to='ali@conference.preprodtenant.meet.jitsi.com/focus' type='error' from='1b062c1d-1fa8-417a-8dc3-936d0ae31be2@guest.meet.jitsi.com/bSxIwLaYOLla'>

Aug 25 11:46:52 guest.meet.jitsi.com:smacks	debug	Received ack request, acking for 6
Aug 25 11:46:52 c2s55df823c7c40	debug	Sending[c2s]: <a xmlns='urn:xmpp:sm:3' h='6'>
Aug 25 11:46:52 c2s55df823c7c40	debug	Handled 7 incoming stanzas
Aug 25 11:46:52 c2s55df823c7c40	debug	Received[c2s]: <presence type='unavailable' xml:lang='en'>
Aug 25 11:46:52 c2s55df823c7c40	debug	Sending[c2s]: <presence type='unavailable' xml:lang='en' from='7829fae0-8e0d-4a2d-bc24-57d7e1c60219@guest.meet.jitsi.com/nJ5aztMTgQQV'>
Aug 25 11:46:52 c2s55df823c7c40	debug	Received </stream:stream>
Aug 25 11:46:52 c2s55df823c7c40	debug	Sending[c2s]: <a xmlns='urn:xmpp:sm:3' h='7'>
Aug 25 11:46:52 c2s55df823c7c40	debug	Sending[c2s]: <close xmlns='urn:ietf:params:xml:ns:xmpp-framing'>
Aug 25 11:46:52 c2s55df823c7c40	debug	c2s stream for 7829fae0-8e0d-4a2d-bc24-57d7e1c60219@guest.meet.jitsi.com/nJ5aztMTgQQV closed: session closed
Aug 25 11:46:52 c2s55df823c7c40	debug	Destroying session for 7829fae0-8e0d-4a2d-bc24-57d7e1c60219@guest.meet.jitsi.com/nJ5aztMTgQQV (7829fae0-8e0d-4a2d-bc24-57d7e1c60219@guest.meet.jitsi.com)
Aug 25 11:46:52 c2s55df823c7c40	debug	Destroying session with 130 unacked stanzas
Aug 25 11:46:52 c2s55df823c7c40	debug	Received[c2s]: <presence to='ali@conference.preprodtenant.meet.jitsi.com/focus' type='error' from='7829fae0-8e0d-4a2d-bc24-57d7e1c60219@guest.meet.jitsi.com/nJ5aztMTgQQV'>

Aug 25 11:46:57 c2s55df82ada660	debug	Sending[c2s_unbound]: <stream:features>
Aug 25 11:46:57 mod_websocket	debug	Websocket received frame: opcode=1, 108 bytes
Aug 25 11:46:57 c2s55df8a5a7190	debug	Received[c2s_unbound]: <iq type='set' id='_bind_auth_2' xml:lang='en'>
Aug 25 11:46:57 rostermanager	debug	load_roster: asked for: 88c32d63-8254-4205-a7d2-c9be1839a431@guest.meet.jitsi.com
Aug 25 11:46:57 rostermanager	debug	load_roster: loading for new user: 88c32d63-8254-4205-a7d2-c9be1839a431@guest.meet.jitsi.com
Aug 25 11:46:57 rostermanager	debug	load_roster: cache miss, loading from storage
Aug 25 11:46:57 c2s55df8a5a7190	debug	No legacy vCard to migrate or already migrated
Aug 25 11:46:59 c2s55df84511520	debug	Received[c2s]: <iq to='ali@conference.preprodtenant.meet.jitsi.com' id='93a83d9e-f229-4726-9cda-15280dcf32bf:sendIQ' xml:lang='en' type='get'>
Aug 25 11:46:59 conference.meet.jitsi.com:muc_package_properties	debug	jitsi_room_info.package_properties: {table: 0x55df83380c30, table: 0x55df83380c70, table: 0x55df83380cf0}
Aug 25 11:46:59 c2s55df84511520	debug	Sending[c2s]: <iq to='33ad190a-dc37-4325-80ed-65956ad3c627@guest.meet.jitsi.com/0ZEKANGWXtHQ' id='93a83d9e-f229-4726-9cda-15280dcf32bf:sendIQ' type='result' from='[preprodtenant]ali@conference.meet.jitsi.com'>
Aug 25 11:46:59 c2s55df8181fba0	debug	Received[c2s]: <iq to='focus.meet.jitsi.com/1ee7e3ed-fb8e-4afa-805a-38d355af7ec0@guest.meet.jitsi.com/9lg2bh14GV2h' id='2b6b31f5-1962-4842-8a74-52626ee7aca2:sendIQ' xml:lang='en-US' type='error'>
Aug 25 11:46:59 focus.meet.jitsi.com:client_proxy	debug	received stanza from c2s session
Aug 25 11:46:59 focus.meet.jitsi.com:client_proxy	debug	non-presence stanza from target: name = iq, type = error
Aug 25 11:46:59 focus.meet.jitsi.com:client_proxy	debug	de-NAT-ed stanza: from: focus@auth.meet.jitsi.com/focus -> focus.meet.jitsi.com, to: focus.meet.jitsi.com/1ee7e3ed-fb8e-4afa-805a-38d355af7ec0@guest.meet.jitsi.com/9lg2bh14GV2h -> 1ee7e3ed-fb8e-4afa-805a-38d355af7ec0@guest.meet.jitsi.com/9lg2bh14GV2h
Aug 25 11:46:59 c2s55df82ada660	debug	Sending[c2s]: <iq type='error' from='focus.meet.jitsi.com' to='1ee7e3ed-fb8e-4afa-805a-38d355af7ec0@guest.meet.jitsi.com/9lg2bh14GV2h' id='2b6b31f5-1962-4842-8a74-52626ee7aca2:sendIQ' xml:lang='en-US'>
Aug 25 11:46:59 c2s55df8181fba0	debug	Received[c2s]: <iq to='focus.meet.jitsi.com/d9c57134-a75a-4e78-a31b-f7e99c1c5bb8@guest.meet.jitsi.com/1SErIc4vtqQj' id='57f6958c-a55c-4248-9e8c-fb6434ebcd2d:sendIQ' xml:lang='en-US' type='error'>
Aug 25 11:46:59 focus.meet.jitsi.com:client_proxy	debug	received stanza from c2s session
Aug 25 11:47:55 c2s55df86432c70	debug	Destroying session for hibernating too long
Aug 25 11:47:55 c2s55df86432c70	debug	Destroying session for d367ba1e-5914-4926-9144-e41207b5befc@guest.meet.jitsi.com/C7gf-G_bMW3r (d367ba1e-5914-4926-9144-e41207b5befc@guest.meet.jitsi.com)
Aug 25 11:47:55 c2s55df86432c70	debug	Destroying session with 2 unacked stanzas
Aug 25 11:47:55 c2s55df86432c70	debug	Received[c2s]: <iq to='meet.jitsi.com' id='086c5c70-7e0d-42d5-8557-82f6b2c76dc4:sendIQ' type='error' from='d367ba1e-5914-4926-9144-e41207b5befc@guest.meet.jitsi.com/C7gf-G_bMW3r'>
Aug 25 11:47:55 stanzarouter	debug	Discarding unhandled error iq (cancel, recipient-unavailable) from c2s: <iq to='meet.jitsi.com' id='086c5c70-7e0d-42d5-8557-82f6b2c76dc4:sendIQ' type='error' from='d367ba1e-5914-4926-9144-e41207b5befc@guest.meet.jitsi.com/C7gf-G_bMW3r'>
Aug 25 11:47:55 c2s55df86432c70	debug	Received[c2s]: <iq to='focus.meet.jitsi.com' id='7c9a0756-fed2-45b8-9597-59d497ae8dce:sendIQ' type='error' from='d367ba1e-5914-4926-9144-e41207b5befc@guest.meet.jitsi.com/C7gf-G_bMW3r'>
Aug 25 11:47:55 focus.meet.jitsi.com:client_proxy	debug	received stanza from c2s session
Aug 25 11:47:55 focus.meet.jitsi.com:client_proxy	debug	stanza to target: name = iq, type = error
Aug 25 11:47:55 sessionmanager	debug	All resources of d367ba1e-5914-4926-9144-e41207b5befc are now offline
Aug 25 11:47:55 c2s55df84482280	debug	mod_smacks hibernation timeout reached...
Aug 25 11:47:55 c2s55df84482280	debug	Destroying session for hibernating too long

When using mod_smacks with prosody 0.12 try with deleting the version from jitsi-meet and use the version that comes with prosody 0.12, does that change a thing?

Thank you for recommendation however I have changed the mod_smack in prosody-plugins to mod_smacks in prosody12.1 binary , same result crashed at 281 user

Can you rebuild that image reverting it to lua 5.2 and try again?

1 Like

Hello thank you, I have built the image with prosody12.1 with lua5.2 .

Also the luajwtjitsi.lib.lua I used is from luajwtjitsi version 2.0.0 from luarocks because the one in latest prosody-plugins is not compatible to 5.2 only worked with 5.4. I had changed it previously while using 5.4 to one from jitsi (and the line in token/util.lib added the module:require line, I reversed it while building 5.2).

I have reached to 441 participant and couldn’t make it to crash the bots are finished. What makes the difference do you have an idea?

Thanks for doing this work!

That doesn’t look too good for Lua 5.4 :-/

So, I’d say the enxt step would be to reach out to the Prosody folks and see if they are aware of anything specific.

1 Like

Hello,

I opened a thread in prosody group.

https://groups.google.com/g/prosody-users/c/dgvcxPH82x8/m/tpcJSZDNBAAJ

We chatted about the situation, and results pointed to in lua 5.4 the gc settings are updated (Advanced GC configuration – Prosody IM ) and I was using same gc settings in both lua 5.2 and 5.4 versions.

In lua 5.2 and 5.4 I was using

gc = {
    mode = "incremental";
    threshold = 105;
    speed = 250;
}

I have deleted the gc settings for defaults values in lua 5.4 I have reached to 321 participant before the room crash.

After that I have increased the threshold and written the default step_size (new in lua 5.4)value.

gc = {
    mode = "incremental";
    threshold = 150;
    speed = 250;
    step_size=13;
}

I have reached to 441 participants in the room couldn’t make the room crash in prosody12.1-lua 5.4 with my updated gc settings (same as prosody 12.1-lua5.2 with my initial gc settings).

I will tweak the gc settings more.

Thank you for steering me to prosody community and your suggestions.

Best.

4 Likes

This is very valuable information, thanks for sharing!

1 Like

@krebya If you are up for it, a PR adding env variables for the gc parameters would be most welcome.

1 Like

Hello,

I created a PR.

I have tried both incremental and the new generational mode. I have made incremental mode default and added option for generational.

Best

1 Like

Awesome!