CONNECTION_FAILED event handling and connection quality status range

Hi,

We have deployed Jitsi meet which has latest installation.
And using lib-jitsi-meet to build custom UI.

  1. Does Jitsi recover even after sending CONNECTION_FAILED event?
    Because what we are doing is refreshing window as soon as we get this error(we could do better like set timeout), and we are getting feedback from users that refreshing window is bad UX and again there is 60sec timeout for which prev abnormally left user window stays on UI.

  2. I am getting 0-100 value for connection quality, it would be helpful if I get any info about setting range for each status (poor-fair-good)

Thanks in advance.

Depends on the error, CONNECTION_FAILED is because of an authentication error we do not reload, but I think rest of the cases jitsi-meet relaods, but this should not happen often so it is acceptable.

Thanks,

We have been observing this many times lately.

Previously observed when connection is poor, but it seems like we are getting this error when connection is good too.

Have you looked why is this happening?
Any correlation between regions of client and signalling server, is it happening when the rtt is big.

Most of the clients are USA based, and Jitsi meet server + 5 JVBs (without octo configuration) are installed in us-central region.
We(developers team) connect from India in our tech sync-ups.

The thing is, in our meetings, client from USA also gets this error and drops.

I checked the Jicofo logs, but couldn’t find anything.

I am monitoring Jicofo, prosody logs though.

@damencho Hi,

I got the following logs at prosody side,

Jan 21 12:32:12 mod_bosh info New BOSH session, assigned it sid ‘2505c0bc-9570-4af5-8453-27c0e20fba6a’
Jan 21 12:32:12 bosh2505c0bc-9570-4af5-8453-27c0e20fba6a info Authenticated as 77fded41-8821-403e-854e-c32c428bad44@subdomain.domain.com
Jan 21 12:33:29 mod_bosh info Client tried to use sid ‘2505c0bc-9570-4af5-8453-27c0e20fba6a’ which we don’t know about
Jan 21 12:33:29 mod_bosh info Client tried to use sid ‘2505c0bc-9570-4af5-8453-27c0e20fba6a’ which we don’t know about

Is there any issue at prosody configuration?

No everything seems fine, it is just that after Jan 21 12:32:12 there was on communication between the client and the server for at least 60 seconds and the bosh connection expired and when next request came with that sessionId it is unknown.
That is strange normally there should be an xmpp ping every 10 seconds and the client will notice earlier that there is no network and do not try to re-use the sid of the connection…

Do you see in the client pings every 10 seconds, you should see those on meet.jit.si.

Yes, there’s http-bind ping every 10 seconds from client.

Hi @damencho I have reinstalled the Jitsi following quick install repo readme.md, kept everything default, and still users are dropping, on Jitsi UI, it throws to rejoin prompt, can you please help me out here?

Blockquote
VM408 backend.js:6 2020-01-24T09:44:27.414Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Error: Ping timeout
at VM401 lib-jitsi-meet.min.js:22
at r.TimedHandler.handler (VM401 lib-jitsi-meet.min.js:6)
at r.TimedHandler.run (VM401 lib-jitsi-meet.min.js:6)
at r.Connection._onIdle (VM401 lib-jitsi-meet.min.js:6)
at r.Connection. (VM401 lib-jitsi-meet.min.js:6)
r @ VM408 backend.js:6
o @ VM401 lib-jitsi-meet.min.js:6
getGlobalOnErrorHandler @ VM401 lib-jitsi-meet.min.js:22
window.onerror @ VM402 app.bundle.min.js:1
callErrorHandler @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:22
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
run @ VM401 lib-jitsi-meet.min.js:6
_onIdle @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
setTimeout (async)
_onIdle @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
VM408 backend.js:6 2020-01-24T09:44:27.418Z [modules/xmpp/strophe.ping.js] Ping timeout null
r @ VM408 backend.js:6
o @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:22
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
run @ VM401 lib-jitsi-meet.min.js:6
VM401 lib-jitsi-meet.min.js:6 2020-01-24T09:44:32.539Z [modules/xmpp/JingleSessionPC.js] <w.peerconnection.oniceconnectionstatechange>: (TIME) ICE disconnected P2P? false: 120320.81000006292
VM401 lib-jitsi-meet.min.js:6 2020-01-24T09:44:33.106Z [modules/xmpp/JingleSessionPC.js] <w.peerconnection.oniceconnectionstatechange>: (TIME) ICE connected P2P? false: 120888.11499997973
VM408 backend.js:6 2020-01-24T09:44:37.445Z [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Error: Ping timeout
at VM401 lib-jitsi-meet.min.js:22
at r.TimedHandler.handler (VM401 lib-jitsi-meet.min.js:6)
at r.TimedHandler.run (VM401 lib-jitsi-meet.min.js:6)
at r.Connection._onIdle (VM401 lib-jitsi-meet.min.js:6)
at r.Connection. (VM401 lib-jitsi-meet.min.js:6)
r @ VM408 backend.js:6
o @ VM401 lib-jitsi-meet.min.js:6
getGlobalOnErrorHandler @ VM401 lib-jitsi-meet.min.js:22
window.onerror @ VM402 app.bundle.min.js:1
callErrorHandler @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:22
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
run @ VM401 lib-jitsi-meet.min.js:6
_onIdle @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
setTimeout (async)

VM408 backend.js:6 2020-01-24T09:44:37.451Z [modules/xmpp/strophe.ping.js] Ping timeout null
r @ VM408 backend.js:6
o @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:22
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
run @ VM401 lib-jitsi-meet.min.js:6
_onIdle @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
setTimeout (async)

VM408 backend.js:6 2020-01-24T09:44:39.389Z [modules/xmpp/strophe.util.js] <Object.i.Strophe.log>: Strophe: Request 47 timed out, over 66 seconds since last activity
r @ VM408 backend.js:6
o @ VM401 lib-jitsi-meet.min.js:6
i.Strophe.log @ VM401 lib-jitsi-meet.min.js:22
warn @ VM401 lib-jitsi-meet.min.js:6
_onIdle @ VM401 lib-jitsi-meet.min.js:6
_onIdle @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
setTimeout (async)
_onIdle @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
setTimeout (async)

_onIdle @ VM401 lib-jitsi-meet.min.js:6
(anonymous) @ VM401 lib-jitsi-meet.min.js:6
VM401 lib-jitsi-meet.min.js:6 2020-01-24T09:44:40.478Z [modules/xmpp/xmpp.js] <t.value>: (TIME) Strophe connfail[item-not-found]: 128260.08999999613
VM401 lib-jitsi-meet.min.js:6 2020-01-24T09:44:40.484Z [modules/xmpp/xmpp.js] <t.value>: (TIME) Strophe disconnected[item-not-found]: 128266.01999998093
VM401 lib-jitsi-meet.min.js:6 2020-01-24T09:44:40.486Z [modules/xmpp/strophe.ping.js] <s.value>: Ping interval cleared
VM401 lib-jitsi-meet.min.js:6 2020-01-24T09:44:40.490Z [modules/statistics/statistics.js] <Function.S.sendAnalyticsAndLog>: {“type”:“operational”,“action”:“connection.failed”,“attributes”:{“error_type”:“connection.otherError”,“error_message”:“item-not-found”,“suspend_time”:9,“time_since_last_success”:68454}}

Is http-bind address working?
Share the prosody version you use and the prosody config you have.
Do you see any errors in prosody logs?

Got log while installing : jitsi-meet-prosody (1.0.3729-1)
Prosody config :
VirtualHost “subdomain.domain.com
– enabled = false – Remove this line to enable this host
authentication = “anonymous”
– Properties below are modified by jitsi-meet-tokens package config
– and authentication above is switched to “token”
–app_id=“example_app_id”
–app_secret=“example_app_secret”
– Assign this host a certificate for TLS, otherwise it would use the one
– set in the global section (if any).
– Note that old-style SSL on port 5223 only supports one certificate, and will always
– use the global one.
ssl = {
key = “/etc/prosody/certs/subdomain.domain.com.key”;
certificate = “/etc/prosody/certs/subdomain.domain.com.crt”;
}
– we need bosh
modules_enabled = {
“bosh”;
“pubsub”;
“ping”; – Enable mod_ping
}

    c2s_require_encryption = false

Component “conference.subdomain.domain.com” “muc”
storage = “none”
–modules_enabled = { “token_verification” }
admins = { “focus@auth.subdomain.domain.com” }

Component “jitsi-videobridge.subdomain.domain.com
component_secret = “MQGafWL0”

VirtualHost “auth.subdomain.domain.com
ssl = {
key = “/etc/prosody/certs/auth.subdomain.domain.com.key”;
certificate = “/etc/prosody/certs/auth.subdomain.domain.com.crt”;
}
authentication = “internal_plain”

Component “focus.subdomain.domain.com
component_secret = “SnT@IHtp”

prosody logs :
Jan 24 11:33:22 bosh5b4f65ae-0143-4163-8bfc-01de667d428f info Authenticated as bc48c641-5880-4824-93bb-73875470e43e@subdomain.domain.com
Jan 24 11:33:32 mod_bosh info New BOSH session, assigned it sid ‘06361bc3-240f-42ec-b3e9-a0a7be500cc1’
Jan 24 11:33:33 bosh06361bc3-240f-42ec-b3e9-a0a7be500cc1 info Authenticated as 975ba9d2-4558-48f5-9e7a-db4ae1515002@subdomain.domain.com
Jan 24 11:33:44 mod_bosh info New BOSH session, assigned it sid ‘b03658c1-db01-4390-8a26-3872658beba9’
Jan 24 11:33:45 boshb03658c1-db01-4390-8a26-3872658beba9 info Authenticated as 936f1fb1-edfe-4e88-8c13-88eed236e91a@subdomain.domain.com
Jan 24 11:35:18 mod_bosh info New BOSH session, assigned it sid ‘2204b85a-44ad-473b-be5a-9069af3f21e5’
Jan 24 11:35:18 bosh2204b85a-44ad-473b-be5a-9069af3f21e5 info Authenticated as 904834c8-e2ba-41f4-bd26-e54178ae7fe8@subdomain.domain.com
Jan 24 11:35:44 mod_bosh info New BOSH session, assigned it sid ‘a41555e6-3a64-47e3-abc8-bc2f13fa3de3’
Jan 24 11:35:45 bosha41555e6-3a64-47e3-abc8-bc2f13fa3de3 info Authenticated as 690396ee-cc14-44dc-9d8b-80c899c76464@subdomain.domain.com
Jan 24 11:35:46 mod_bosh info New BOSH session, assigned it sid ‘dc1b9135-3a15-42c6-b851-959965654046’
Jan 24 11:35:46 boshdc1b9135-3a15-42c6-b851-959965654046 info Authenticated as df2db61d-fb33-4177-a9d7-64c39a07a2b1@subdomain.domain.com
Jan 24 11:37:19 mod_bosh info Client tried to use sid ‘2204b85a-44ad-473b-be5a-9069af3f21e5’ which we don’t know about
Jan 24 11:37:19 mod_bosh info Client tried to use sid ‘2204b85a-44ad-473b-be5a-9069af3f21e5’ which we don’t know about
Jan 24 11:37:23 mod_bosh info Client tried to use sid ‘a41555e6-3a64-47e3-abc8-bc2f13fa3de3’ which we don’t know about
Jan 24 11:37:23 mod_bosh info Client tried to use sid ‘a41555e6-3a64-47e3-abc8-bc2f13fa3de3’ which we don’t know about
Jan 24 11:38:01 mod_bosh info New BOSH session, assigned it sid ‘dfb0a564-fd68-47bd-bbbe-6b12fee99024’
Jan 24 11:38:02 boshdfb0a564-fd68-47bd-bbbe-6b12fee99024 info Authenticated as 18a39dee-d76a-46f5-ac2d-1d42a9db7379@subdomain.domain.com
Jan 24 11:38:08 mod_bosh info New BOSH session, assigned it sid ‘c5205e88-9be1-4378-8359-cf2f66cde774’
Jan 24 11:38:08 boshc5205e88-9be1-4378-8359-cf2f66cde774 info Authenticated as 0c151afc-891c-4e2f-aaf6-6c1a75225c6c@subdomain.domain.com
Jan 24 11:40:19 mod_bosh info Client tried to use sid ‘dfb0a564-fd68-47bd-bbbe-6b12fee99024’ which we don’t know about
Jan 24 11:40:19 mod_bosh info Client tried to use sid ‘dfb0a564-fd68-47bd-bbbe-6b12fee99024’ which we don’t know about

When user about to get Connection_failed error, http-bind calls seems to stop.

Normal as there are minute and the half between requests.

You have ping enabled, it is even in the client logs that ping timeouts … well seems either client loosing connection or server loosing connection and bosh requests timeout … look at the VM stats, network stats, usage and load … if running on some cloud provider, can it be to have stolen cpu time from a noisy neighbour …

Running on GCP n1-standard-4 (4 vCPUs, 15 GB memory)
Only Jitsi-server is running on this VM and nothing else (no other server)

Please find the CPU stats

Hi @damencho, I have been monitoring the meetings, I can see difference between meet.jit.si and our server is huge RTT(sometimes over 25k+) and when RTT spikes and stays same, CONNECTION_FAILED event occurs.
Even tried downgrading Jitsi to 1.0.3936-1, It seems less drops on this version but still no sure.

So that’s definitely the network in front of your server, in case that when you see big RTT between your client and your server and the RTT to meet.jit.si servers is low.

Hi @damencho I’ve got the proper browser logs for this error.
I can see CORS error for first http-bind request, and then user drops from call.
Sometime first http-bind call takes huge time to get response resulting user connection delay.

Btw I had already added following lines in prosody.cfg.lua way back, and this happens intermittently,
cross_domain_bosh = true;
consider_bosh_secure = true;

Is there any missing while doing CORS configuration for Jitsi-meet backend?

So the address you use to access your website and the bosh connection address configured in config.js are different, and you haven’t allow this situation in your nginx config. You can try adding:

location = /http-bind {
    ....
    add_header 'Access-Control-Allow-Origin' '*';
    ....
}

Thanks for the response @damencho .

Our Jitsi server installation is done using quick install readme.md documentation so I think default server is jetty (and not nginx).

Btw error mentioned comes intermittently in http-bind call. For all other http-bind, I can see Access-Control-Allow-Origin : “*” header present in http-bind calls.

Would you recommend us to shift to using NginX from Jetty? Will that resolve the issue since as soon as we see this error, we see a connection failed event and the user call drops. Thus it is critical for us to have a resolution of this issue and we are anxious to hear from you.