Hello,
I have an issue with ghosts users and tried everything that I found on the community forum.
I upgraded prosody to 0.11.5 but it is still happening.
The configuration:
Server #1 / 16 CPU, 32G RAM, 4Gb/s
debian 9.12
nginx 1.10.3
jitsi-meet 2.0.4548
jitsi-meet-prosody 1.0.4074
jitsi-meet-web 1.0.4074
jitsi-meet-web-config 1.0.4074
jicofo 1.0-567
openjdk-8-jre-headless
Servers #2 #3 #4 #5 / 16 CPU, 32G RAM, 4Gb/s
debian 9.12
nginx 1.10.3
jitsi-videobridge2 2.1-197-g38256192
openjdk-8-jre-headless
I have configured Websockets in nginx, in the Jitsi meet config.js and in prosody.
The prosody storage is set to « memory », the secure domain is set.
/etc/systemd/system.conf set with:
DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000
DefaultTasksMax=65000
Prosody config has:
cross_domain_bosh = false;
consider_bosh_secure = true;
cross_domain_websocket = true;
consider_websocket_secure = true;
The issue:
When we have a lot of users using the service (around 1000 at the same minute on 4 videobridges - 250 users per videobridge), some rooms are never shutting down because of some « ghosts » users remaining.
In those rooms, when refreshing the page, another user appears.
Writing a message into the Chat refreshes the connections and ghosts disappear. It doesn’t solve the problem because all of the users that are still in the room will be ghosts when they leave.
More information:
I think that the videobridge has made the disconnection (the rooms occupation in colibri stats seems to be OK) but jicofo / prosody still try to use the disconnected JID.
Restarting prosody solves the issue but it starts over each day.
These are the logs that I can see in /var/log/prosody/prosody.err. We have the same logs in /var/log/prosody/prosody.log. They continue periodically and never end:
May 14 14:41:22 c2s55598c5bbe90 error Traceback[c2s]: …itsi-meet/prosody-plugins/mod_speakerstats_component.lua:193: attempt to index field ‘speakerStats’ (a nil value)
stack traceback:
…itsi-meet/prosody-plugins/mod_speakerstats_component.lua:193: in function ‘?’
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/modules/muc/muc.lib.lua:561: in function </usr/lib/prosody/modules/muc/muc.lib.lua:498>
(…tail calls…)
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/core/stanza_router.lua:180: in function ‘core_post_stanza’
/usr/lib/prosody/core/stanza_router.lua:127: in function ‘core_process_stanza’
/usr/lib/prosody/modules/mod_c2s.lua:275: in function ‘func’
/usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125>
May 14 14:41:26 c2s55598403cde0 error Traceback[c2s]: …itsi-meet/prosody-plugins/mod_speakerstats_component.lua:193: attempt to index field ‘speakerStats’ (a nil value)
stack traceback:
…itsi-meet/prosody-plugins/mod_speakerstats_component.lua:193: in function ‘?’
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/modules/muc/muc.lib.lua:561: in function </usr/lib/prosody/modules/muc/muc.lib.lua:498>
(…tail calls…)
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/core/stanza_router.lua:180: in function ‘core_post_stanza’
/usr/lib/prosody/modules/mod_presence.lua:109: in function ‘handle_normal_presence’
/usr/lib/prosody/modules/mod_presence.lua:304: in function ‘?’
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/core/stanza_router.lua:180: in function ‘core_post_stanza’
/usr/lib/prosody/core/stanza_router.lua:127: in function ‘core_process_stanza’
/usr/lib/prosody/modules/mod_c2s.lua:275: in function ‘func’
/usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125>
May 14 14:41:27 mod_bosh error Traceback[bosh]: …itsi-meet/prosody-plugins/mod_speakerstats_component.lua:193: attempt to index field ‘speakerStats’ (a nil value)
stack traceback:
…itsi-meet/prosody-plugins/mod_speakerstats_component.lua:193: in function ‘?’
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/modules/muc/muc.lib.lua:561: in function </usr/lib/prosody/modules/muc/muc.lib.lua:498>
(…tail calls…)
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/core/stanza_router.lua:180: in function ‘core_post_stanza’
/usr/lib/prosody/core/stanza_router.lua:127: in function ‘dispatch_stanza’
/usr/lib/prosody/modules/mod_bosh.lua:305: in function ‘func’
/usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125>
stack traceback:
/usr/lib/prosody/util/async.lua:211: in function ‘run’
/usr/lib/prosody/modules/mod_bosh.lua:447: in function ‘cb_handlestanza’
/usr/lib/prosody/util/xmppstream.lua:182: in function </usr/lib/prosody/util/xmppstream.lua:162>
[C]: in function ‘parse’
/usr/lib/prosody/util/xmppstream.lua:282: in function ‘feed’
/usr/lib/prosody/modules/mod_bosh.lua:133: in function ‘?’
/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>
(…tail calls…)
/usr/lib/prosody/net/http/server.lua:228: in function </usr/lib/prosody/net/http/server.lua:176>
[C]: in function ‘xpcall’
/usr/lib/prosody/net/http/server.lua:108: in function ‘process_next’
/usr/lib/prosody/net/http/server.lua:124: in function ‘success_cb’
/usr/lib/prosody/net/http/parser.lua:177: in function ‘feed’
/usr/lib/prosody/net/http/server.lua:155: in function </usr/lib/prosody/net/http/server.lua:154>
(…tail calls…)
/usr/lib/prosody/net/server_select.lua:915: in function </usr/lib/prosody/net/server_select.lua:899>
[C]: in function ‘xpcall’
/usr/bin/prosody:76: in function ‘loop’
/usr/bin/prosody:86: in main chunk
[C]: in ?
The line quoted in /usr/share/jitsi-meet/prosody-plugins/mod_speakerstats_component.lua is:
local speakerStatsForOccupant = room.speakerStats [occupant.jid];
It seems that the occupant exists but the jid is not in room.speakerStats anymore (because he has disconnected, but somehow, he is still kept somewhere).
Do you have some clues about how to solve this issue and make the ghosts disappear?
Can I use something to force jicofo/prosody to purge these connections without restarting prosody?
Thank you for any help you can give. I can send more info / complete files if needed. Everything works fine for us except this issue.