Bosh Timeout Error - Jigasi dial-in calls drops off randomly within few minutes after joining the conference

mehtapaxshal · October 9, 2020, 10:35pm

Hi all,

I am getting below timeout error while sending http-bind request for dial-in user and so, call drops off within few minutes after joining the call.

nginx error logs:

Summary

2020/10/09 17:02:27 [error] 17182#17182: *460624 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 127.0.0.1, server: www.domain.com, request: “POST /http-bind?room=&token=HTTP/1.1”, upstream: “http://[::1]:5280/http-bind?room=&token=”, host: “www.domain.com:443”

Jigasi error logs:

Summary

2020-10-09 21:02:27.274 WARNING: [316] org.jivesoftware.smack.bosh.XMPPBOSHConnection.shutdown() shutdown

java.lang.NullPointerException

at org.igniterealtime.jbosh.BOSHClient.send(BOSHClient.java:494)

at org.igniterealtime.jbosh.BOSHClient.disconnect(BOSHClient.java:586)

at org.igniterealtime.jbosh.BOSHClient.disconnect(BOSHClient.java:567)

at org.jivesoftware.smack.bosh.XMPPBOSHConnection.shutdown(XMPPBOSHConnection.java:266)

at org.jivesoftware.smack.bosh.XMPPBOSHConnection.notifyConnectionError(XMPPBOSHConnection.java:417)

at org.jivesoftware.smack.bosh.XMPPBOSHConnection$BOSHConnectionListener.connectionEvent(XMPPBOSHConnection.java:464)

at org.igniterealtime.jbosh.BOSHClient.fireConnectionClosedOnError(BOSHClient.java:1684)

at org.igniterealtime.jbosh.BOSHClient.dispose(BOSHClient.java:713)

at org.igniterealtime.jbosh.BOSHClient.processExchange(BOSHClient.java:1138)

at org.igniterealtime.jbosh.BOSHClient.processMessages(BOSHClient.java:999)

at org.igniterealtime.jbosh.BOSHClient.access$300(BOSHClient.java:100)

at org.igniterealtime.jbosh.BOSHClient$RequestProcessor.run(BOSHClient.java:1728)

at java.base/java.lang.Thread.run(Thread.java:834)

2020-10-09 21:02:27.275 WARNING: [316] org.jivesoftware.smack.AbstractXMPPConnection.callConnectionClosedOnErrorListener() Connection XMPPBOSHConnection[jigasi_user@auth.domain.com/2736bc18] (2) closed with error

org.igniterealtime.jbosh.BOSHException: Could not parse body:

<html>

<head><title>504 Gateway Time-out</title></head>

<body bgcolor="white">

<center><h1>504 Gateway Time-out</h1></center>

<hr><center>nginx/1.14.2</center>

</body>

</html>

at org.igniterealtime.jbosh.BodyParserXmlPull.parse(BodyParserXmlPull.java:132)

at org.igniterealtime.jbosh.StaticBody.fromString(StaticBody.java:114)

at org.igniterealtime.jbosh.ApacheHTTPResponse.awaitResponse(ApacheHTTPResponse.java:246)

at org.igniterealtime.jbosh.ApacheHTTPResponse.getBody(ApacheHTTPResponse.java:192)

at org.igniterealtime.jbosh.BOSHClient.processExchange(BOSHClient.java:1123)

at org.igniterealtime.jbosh.BOSHClient.processMessages(BOSHClient.java:999)

at org.igniterealtime.jbosh.BOSHClient.access$300(BOSHClient.java:100)

at org.igniterealtime.jbosh.BOSHClient$RequestProcessor.run(BOSHClient.java:1728)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.lang.IllegalStateException: Root element was not 'body' in the 'http://jabber.org/protocol/httpbind' namespace. (Was 'html' in '')

at org.igniterealtime.jbosh.BodyParserXmlPull.parse(BodyParserXmlPull.java:98)

... 8 more

2020-10-09 21:02:27.276 SEVERE: [316] impl.protocol.jabber.ProtocolProviderServiceJabberImpl.connectionClosedOnError().2305 connectionClosedOnError Could not parse body:

<html>

<head><title>504 Gateway Time-out</title></head>

<body bgcolor="white">

<center><h1>504 Gateway Time-out</h1></center>

<hr><center>nginx/1.14.2</center>

</body>

</html>

org.igniterealtime.jbosh.BOSHException: Could not parse body:

<html>

<head><title>504 Gateway Time-out</title></head>

<body bgcolor="white">

<center><h1>504 Gateway Time-out</h1></center>

<hr><center>nginx/1.14.2</center>

</body>

</html>

at org.igniterealtime.jbosh.BodyParserXmlPull.parse(BodyParserXmlPull.java:132)

at org.igniterealtime.jbosh.StaticBody.fromString(StaticBody.java:114)

at org.igniterealtime.jbosh.ApacheHTTPResponse.awaitResponse(ApacheHTTPResponse.java:246)

at org.igniterealtime.jbosh.ApacheHTTPResponse.getBody(ApacheHTTPResponse.java:192)

at org.igniterealtime.jbosh.BOSHClient.processExchange(BOSHClient.java:1123)

at org.igniterealtime.jbosh.BOSHClient.processMessages(BOSHClient.java:999)

at org.igniterealtime.jbosh.BOSHClient.access$300(BOSHClient.java:100)

at org.igniterealtime.jbosh.BOSHClient$RequestProcessor.run(BOSHClient.java:1728)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.lang.IllegalStateException: Root element was not 'body' in the 'http://jabber.org/protocol/httpbind' namespace. (Was 'html' in '')

at org.igniterealtime.jbosh.BodyParserXmlPull.parse(BodyParserXmlPull.java:98)

... 8 more

Any idea what can be wrong here and how to solve it? Thanks

damencho · October 10, 2020, 2:57am

Can you share your jigasi config for bosh url?

mehtapaxshal · October 10, 2020, 4:31am

Sure. Here it is:

#Sample config with one XMPP and one SIP account configured
# Replace {sip-pass-hash} with SIP user password hash
# as well as other account properties

# Name of default JVB room that will be joined if no special header is included
# in SIP invite
org.jitsi.jigasi.DEFAULT_JVB_ROOM_NAME=siptest

net.java.sip.communicator.impl.protocol.SingleCallInProgressPolicy.enabled=false

# Should be enabled when using translator mode
#net.java.sip.communicator.impl.neomedia.audioSystem.audiosilence.captureDevice_list=["AudioSilenceCaptureDevice:noTransferData"]

# Adjust opus encoder complexity
net.java.sip.communicator.impl.neomedia.codec.audio.opus.encoder.COMPLEXITY=10

# Disables packet logging
net.java.sip.communicator.packetlogging.PACKET_LOGGING_ENABLED=true

net.java.sip.communicator.impl.protocol.sip.acc1403273890647=acc1403273890647
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.ACCOUNT_UID=SIP\:dial_in_user@domain.com
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PASSWORD=<password>
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROTOCOL_NAME=SIP
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.SERVER_ADDRESS=domain.com
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROXY_ADDRESS=proxy_domain.com
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROXY_AUTO_CONFIG=false
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROXY_PORT=5061
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PREFERRED_TRANSPORT=TLS
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.USER_ID=dial_in_user@domain.com
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.KEEP_ALIVE_INTERVAL=25
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.KEEP_ALIVE_METHOD=OPTIONS
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.VOICEMAIL_ENABLED=false
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.AMR-WB/16000=750
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.G722/8000=700
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.GSM/8000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.H263-1998/90000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.H264/90000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.PCMA/8000=600
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.PCMU/8000=650
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/12000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/16000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/24000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/8000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.VP8/90000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.iLBC/8000=10
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.opus/48000=1000
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.red/90000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.speex/16000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.speex/32000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.speex/8000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.telephone-event/8000=1
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.ulpfec/90000=0
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.OVERRIDE_ENCODINGS=true
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.DEFAULT_ENCRYPTION=false



# If an authenticated (hidden) domain is used to connect to a conference,
# PREVENT_AUTH_LOGIN will prevent the SIP participant from being seen as a
# hidden participant in the conference
#net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PREVENT_AUTH_LOGIN=FALSE

# Used when incoming calls are used in multidomain environment, used to detect subdomains
# used for constructing callResource and eventually contacting jicofo
#net.java.sip.communicator.impl.protocol.sip.acc1403273890647.DOMAIN_BASE=<jitsi_domain>

# the pattern to be used as bosh url when using bosh in multidomain environment
net.java.sip.communicator.impl.protocol.sip.acc1403273890647.BOSH_URL_PATTERN=https://{host}{subdomain}/http-bind?room={roomName}&token=<valid_jwt>

# can be enabled to disable audio mixing and use translator, jigasi will act as jvb, just forward every ssrc stream it receives.
#net.java.sip.communicator.impl.protocol.sip.acc1403273890647.USE_TRANSLATOR_IN_CONFERENCE=true

# We can use the prefix org.jitsi.jigasi.xmpp.acc to override any of the
# properties that will be used for creating xmpp account for communication.

# The following two props assume we are using jigasi on the same machine as
# the xmpp server.
org.jitsi.jigasi.xmpp.acc.IS_SERVER_OVERRIDDEN=true
#org.jitsi.jigasi.xmpp.acc.SERVER_ADDRESS=127.0.0.1
org.jitsi.jigasi.xmpp.acc.SERVER_ADDRESS=<jitsi_domain>
org.jitsi.jigasi.xmpp.acc.VIDEO_CALLING_DISABLED=true
org.jitsi.jigasi.xmpp.acc.JINGLE_NODES_ENABLED=false
org.jitsi.jigasi.xmpp.acc.AUTO_DISCOVER_STUN=false
org.jitsi.jigasi.xmpp.acc.IM_DISABLED=true
org.jitsi.jigasi.xmpp.acc.SERVER_STORED_INFO_DISABLED=true
org.jitsi.jigasi.xmpp.acc.IS_FILE_TRANSFER_DISABLED=true
# Or you can use bosh for the connection establishment by specifing the URL to use.
org.jitsi.jigasi.xmpp.acc.BOSH_URL_PATTERN=https://{host}{subdomain}/http-bind?room={roomName}&token=<valid_jwt>

# can be enabled to disable audio mixing and use translator, jigasi will act as jvb, just forward every ssrc stream it receives.
#org.jitsi.jigasi.xmpp.acc.USE_TRANSLATOR_IN_CONFERENCE=true

# If you want jigasi to perform authenticated login instead of anonymous login
# to the XMPP server, you can set the following properties.
org.jitsi.jigasi.xmpp.acc.USER_ID=jigasi_user@auth.<jitsi_domain>
org.jitsi.jigasi.xmpp.acc.PASS=<password>
org.jitsi.jigasi.xmpp.acc.ANONYMOUS_AUTH=false
org.jitsi.jigasi.xmpp.acc.ALLOW_NON_SECURE=true
# If you want to use the SIP user part of the incoming/outgoing call SIP URI
# you can set the following property to true.
# org.jitsi.jigasi.USE_SIP_USER_AS_XMPP_RESOURCE=true

# Activate this property if you are using self-signed certificates or other
# type of non-trusted certicates. In this mode your service trust in the
# remote certificates always.
net.java.sip.communicator.service.gui.ALWAYS_TRUST_MODE_ENABLED=true

# Enable this property to be able to shutdown gracefully jigasi using
# a rest command
# org.jitsi.jigasi.ENABLE_REST_SHUTDOWN=true

# Options regarding Transcription. Read the README for a detailed description
# about each property

#org.jitsi.jigasi.ENABLE_TRANSCRIPTION=false
#org.jitsi.jigasi.ENABLE_SIP=true

# whether to use the more expensive, but better performing
# "video" model when doing transcription
# org.jitsi.jigasi.transcription.USE_VIDEO_MODEL = false

# delivering final transcript
# org.jitsi.jigasi.transcription.DIRECTORY=/var/lib/jigasi/transcripts
# org.jitsi.jigasi.transcription.BASE_URL=http://localhost/
# org.jitsi.jigasi.transcription.jetty.port=-1
# org.jitsi.jigasi.transcription.ADVERTISE_URL=false

# save formats
# org.jitsi.jigasi.transcription.SAVE_JSON=false
# org.jitsi.jigasi.transcription.SAVE_TXT=true

# send formats
# org.jitsi.jigasi.transcription.SEND_JSON=true
# org.jitsi.jigasi.transcription.SEND_TXT=false

# translation
# org.jitsi.jigasi.transcription.ENABLE_TRANSLATION=false

# record audio. Currently only wav format is supported
# org.jitsi.jigasi.transcription.RECORD_AUDIO=false
# org.jitsi.jigasi.transcription.RECORD_AUDIO_FORMAT=wav

# execute one or more scripts when a transcript or recording is saved
# org.jitsi.jigasi.transcription.EXECUTE_SCRIPTS=true
# org.jitsi.jigasi.transcription.SCRIPTS_TO_EXECUTE_LIST_SEPARATOR=","
# org.jitsi.jigasi.transcription.SCRIPTS_TO_EXECUTE_LIST=script/example_handle_transcript_directory.sh

# filter out silent audio
#org.jitsi.jigasi.transcription.FILTER_SILENCE = false

# properties for optionally sending statistics to a DataDog server
#org.jitsi.ddclient.prefix=jitsi.jigasi
#org.jitsi.ddclient.host=localhost
#org.jitsi.ddclient.port=8125

# sip health checking
# Enables sip health checking by specifying a number/uri to call
# the target just needs to auto-connect the call play some audio,
# the call must be established for less than 10 seconds
# org.jitsi.jigasi.HEALTH_CHECK_SIP_URI=healthcheck
#
# The interval between healthcheck calls, by default is 5 minutes
# org.jitsi.jigasi.HEALTH_CHECK_INTERVAL=300000
#
# The timeout of healthcheck, if there was no successful health check for
# 10 minutes (default value) we consider jigasi unhealthy
# org.jitsi.jigasi.HEALTH_CHECK_TIMEOUT=600000

# Enabled or disable the notification when max occupants limit is reached
# org.jitsi.jigasi.NOTIFY_MAX_OCCUPANTS=false

damencho · October 10, 2020, 5:26am

Don’t see anything unusual … Wonder why would roomname and jwt be stripped from the url … Can you check your prosody logs do you see any errors.

mehtapaxshal · October 12, 2020, 4:51pm

Seems like its working now after setting below parameters in nginx:

proxy_connect_timeout       300;
proxy_send_timeout          300;
proxy_read_timeout          300;
send_timeout                300;

Interesting though, AFAIK, the default timeout is 60 seconds. Just wonder, if it can take more than 60 seconds and times out Damien?

damencho · October 12, 2020, 5:11pm

Ah maybe you don’t have keepalive on the client connection and after you join if for a minute or so there is nothing happening on the xmpp side, no one is joining/leaving/mutting the connection can timeout.
Try adding:

org.jitsi.jigasi.xmpp.acc.KEEP_ALIVE_METHOD=XEP-0199
org.jitsi.jigasi.xmpp.acc.KEEP_ALIVE_INTERVAL=30

Does it make a difference without touching the nginx timeouts?
If that fixes it for you, we need to add it to the config template.

damencho · October 12, 2020, 5:12pm

Good catch!

mehtapaxshal · October 12, 2020, 6:48pm

I tried adding these params and removed those timeouts I set on nginx. It errors out as original after few minutes.
I see an error on prosody though when dial-in user joins as below:

|Oct 12 14:43:38 conference.domain.com:muc_domain_mapper|warn|Session filters applied|
|Oct 12 14:43:38 mod_bosh|info|New BOSH session, assigned it sid '8892eb71-088f-4703-95e8-fe1c9acdbbda'|
|Oct 12 14:43:38 bosh8892eb71-088f-4703-95e8-fe1c9acdbbda|info|Authenticated as jigasi_user@auth.<jitsi_domain>
|Oct 12 14:43:40 s2sout55d394252200|info|Sending error replies for 1 queued stanzas because of failed outgoing connection to focus.domain.com

While call getting dropped off, I don’t see any errors.

damencho · October 12, 2020, 6:52pm

Hum, its actually you are getting timeouts from nginx … So it is the connection between nginx and prosody which timeouts … That is strange, I haven’t seen this in default installations … Without prosody throwing an error.

damencho · October 12, 2020, 6:53pm

Where is this coming from? Isn’t this the focus component address that is already configured in prosody?

mehtapaxshal · October 12, 2020, 7:11pm

So, To give more details,

I am having multiple domains setup for Jitsi on 2 different servers. let say, domain1-1.com and domain1-2.com for server 1 and domain2-1.com and domain2-2.com for server 2.
I am using Jigasi that is on different server, let say domain3.com and trying to put dial-in user using domain3 jigasi to the conference happening on domain2-2.com.

Now, the default focus on server 2 will be focus.domain2-2.com, however, the Jigasi user needs to do xmpp authentication and so, I added same prosody user on server2 as that on server1.

So, if the conference is on server 1, everything works good as the default focus user and the prosody auth is for same domain, however, I am facing above issue for conferences on server2.

I hope you understood the scenario above

mehtapaxshal · October 12, 2020, 7:14pm

I had below in prosody on server2:

VirtualHost "auth.domain2-2.com"
    ssl = {
        key = "/etc/prosody/certs/auth.domain2-2.com.key";
        certificate = "/etc/prosody/certs/auth.domain2-2.com.crt";
    }
    authentication = "internal_plain"

I added prosody user for jigasi and added below config in prosody:

VirtualHost "auth.domain1-2.com"
    authentication = "internal_plain"

damencho · October 12, 2020, 7:48pm

And this domain 1 or 2?

mehtapaxshal · October 12, 2020, 7:53pm

That is domain 2.

damencho · October 12, 2020, 7:55pm

I see, hum, I need to check the code how do we construct the focus address and will let you know

mehtapaxshal · October 12, 2020, 8:08pm

Sure. Thanks. Meanwhile, these are detailed prosody logs for error, in case you wonder:

|Oct 12 12:41:54 s2sout557b90646370|debug|s2s connection attempt failed: unable to resolve service|
|Oct 12 12:41:54 s2sout557b90646370|debug|s2s disconnected: auth.domain1.com->focus.domain2.com (unable to resolve service)|
|Oct 12 12:41:54 s2sout557b90646370|debug|Destroying outgoing session auth.domain1.com->focus.domain2.com: unable to resolve service|
|Oct 12 12:41:54 s2sout557b90646370|info|Sending error replies for 1 queued stanzas because of failed outgoing connection to focus.domain2.com|
|Oct 12 12:41:54 stanzarouter|debug|Received[s2sin]: <iq from='focus.domain2.com' type='error' id='iHWP5-619' to='jigasi_user@auth.domain1.com/2472f712'>|

damencho · October 12, 2020, 8:55pm

You need to pass the correct domain as a custom sip header.
The default header name for that is Jitsi-Conference-Domain-Base. So if you use jigasi to call domain1-2.com pass there domain1-2.com or if you call domain2-2.com than pass domain1-2.com.
If you want to change the name of the header you can use the sip account property: JITSI_MEET_DOMAIN_BASE_HEADER_NAME to set the custom name.
This will solve your problem.

mehtapaxshal · October 12, 2020, 10:11pm

I am already passing the header name depending on where the conference is going on. Here are detailed pcap logs inside jigasi, for reference:

Content-Type: application/sdp

Content-Disposition: session

Jitsi-Conference-Domain-Base: domain2-2.com

Jitsi-Conference-Room: 18db5e21-0cd7-11eb-aaf6-0adbb24bf6fc@conference.domain2-2.com

damencho · October 12, 2020, 10:34pm

github.com

jitsi/jigasi/blob/5e21c0ed19aa107407e1d98988b43633781cc25f/src/main/java/org/jitsi/jigasi/JvbConference.java#L1650


// we need to configure them or retrieve them in the future
focusInviteIQ.addProperty("channelLastN", "-1");
focusInviteIQ.addProperty("disableRtx", "false");
focusInviteIQ.addProperty("startBitrate", "800");
focusInviteIQ.addProperty("openSctp", "true");

try
{
    focusInviteIQ.setType(IQ.Type.set);
    focusInviteIQ.setTo(JidCreate.domainBareFrom(
        gatewaySession.getFocusResourceAddr()
            + "." + callContext.getDomain()));
}
catch (XmppStringprepException e)
{
    logger.error(this.callContext +
        " Could not create destination address for focus invite", e);
    return;
}

// this check just skips an exception when running tests

It creates it this way. And domain comes from:
The sip account property domain base, which you don’t use: https://github.com/jitsi/jigasi/blob/2a2e63c3a53c807688e7b16c8c62004eb52b8f40/src/main/java/org/jitsi/jigasi/SipGateway.java#L211
Or from the header which you pass: https://github.com/jitsi/jigasi/blob/777da0ea60b90547977063fae11c233370830402/src/main/java/org/jitsi/jigasi/SipGatewaySession.java#L668
Than I have no idea where it would get the wrong domain …

Next step is to debug it, maybe the easiest way is to add here: https://github.com/jitsi/jigasi/blob/5f9f6f78fd8661d1fa4ee747ce00a1690a1aaade/src/main/java/org/jitsi/jigasi/CallContext.java#L205
new Exception(domain).printStackTrace();

To see the setting of the wrong domain and the stack trace where it is set.

mehtapaxshal · October 14, 2020, 12:51am

Thanks damien.

Seems like its working now (even without increasing the nginx proxy timeout I mentioned above). I believe there was an incorrect internal host name on my server itself which seems to be causing this issue of back and forth communication.

Anyways, thanks for your time and help.