Jigasi with haproxy scaling architecture

Hi guys,

I have a scalable architecture setup using Haproxy and multiple Jitsi Shards in turn having multiple JVBs per shards. I have multiple Jigasi setup on both shards having a single dial-in number associated. I wonder how dial-in user can end up joining correct meeting room from both shards.

Let say, I have a meeting_A created in Jitsi-Meet-Shard-1. Now, dial-in users will dials in number and how to decide which jigasi should receive this call and put the user into correct meeting? Here, Jigasi on Jitsi-Meet-Shard-1 should take in this call and then, it can redirect to proper meeting.

Is there any configuration I need to do on Jigasi/Haproxy side to make this work?

Thanks.

So your IVR chooses a jigasi by some rule or randomly, dial in jigasi instances can be without connection to actual shard, so they connect using bosh url as normal web clients.

Uh oh! I did a quick install of Jigasi on my Jitsi-Meet-Shard-1. I checked my sip properties file inside jigasi and I don’t see any bosh url configured. I read about Call control MUCs (brewery) , is this what I am suppose to follow and add --nocomponent=true in startup options?

Also, if I do so, what should be the bosh url in that case? Will it be for Jitsi-Meet-Shard-1 or Jitsi-Meet-Shard-2 or Jitsi-Meet-Haproxy?

FYI: I also have multiple-domain setup on each shards. So, right now, Our conference mapper sends the room name as room-name@conference.subdomain.com which can decide to which room it should go.

More inputs will be appreciated. Thanks


Mucs are preffered method, we will soon move the default install to use it, but that is needed only in case if outgoing calls are used.
1 Like

I added the bosh url pattern as mentioned by you. However, running into this issue now after that:

2020-09-25 21:08:40.242 INFO: [55] org.jitsi.jigasi.SipGateway.incomingCallReceived().216 [ctx=1601068120235325136732] Incoming call received...

2020-09-25 21:08:40.249 INFO: [61] org.jitsi.jigasi.SipGatewaySession.run().1524 [ctx=1601068120235325136732] Wait thread cancelled

2020-09-25 21:08:40.254 INFO: [55] org.jitsi.jigasi.JvbConference.start().437 [ctx=1601068120235325136732] Starting JVB conference room: db78089d-db36-11ea-aaf6-0adbb24bf6fc

2020-09-25 21:08:40.297 INFO: [55] org.jitsi.jigasi.JvbConference.setXmppProvider().576 [ctx=1601068120235325136732] Using ProtocolProviderServiceJabberImpl(Jabber:6c1e45cd@<domain>/6c1e45cd)

2020-09-25 21:08:40.403 INFO: [63] org.igniterealtime.jbosh.BOSHClient.init() Starting with 1 request processors

2020-09-25 21:08:40.785 SEVERE: [63] impl.protocol.jabber.ProtocolProviderServiceJabberImpl.connectAndLogin().1003 Failed to connect to XMPP service

**org.jivesoftware.smack.SmackException$SecurityRequiredByClientException: SSL/TLS required by client but not supported by server**

at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:390)

at net.java.sip.communicator.impl.protocol.jabber.ProtocolProviderServiceJabberImpl.connectAndLogin(ProtocolProviderServiceJabberImpl.java:1309)

at net.java.sip.communicator.impl.protocol.jabber.ProtocolProviderServiceJabberImpl.connectAndLogin(ProtocolProviderServiceJabberImpl.java:970)

at net.java.sip.communicator.impl.protocol.jabber.ProtocolProviderServiceJabberImpl.initializeConnectAndLogin(ProtocolProviderServiceJabberImpl.java:795)

at net.java.sip.communicator.impl.protocol.jabber.ProtocolProviderServiceJabberImpl.register(ProtocolProviderServiceJabberImpl.java:500)

at org.jitsi.jigasi.util.RegisterThread.run(RegisterThread.java:59)

2020-09-25 21:08:40.789 SEVERE: [63] org.jitsi.jigasi.JvbConference.registrationStateChanged().631 [ctx=1601068120235325136732] XMPP Connection failed.

2020-09-25 21:08:40.791 WARNING: [63] org.jitsi.jigasi.JvbConference.leaveConferenceRoom().934 [ctx=1601068120235325136732] MUC room is null

Any idea what can be the issue?

Also, 1 big question I have is, does the conference should be passed as room-name@conference.subdomain.com or it can stay as room-name itself in the conference header for multi-domain multi-shard setup?

With @…

What setting did you add, was it with http or https? Try actually the example from line 85, the one with params, jigasi should replace them correctly. Set also the domain base which should be your deployment dns subdomain.com

I tried http, but errors out because there is a redirection to https.

I tried using https. Also, I used the example from line 85, where jigasi replaces the parameters, but still gives the same error.

Here is the Jigasi sip-properties file:

#Sample config with one XMPP and one SIP account configured

# Replace {sip-pass-hash} with SIP user password hash

# as well as other account properties

# Name of default JVB room that will be joined if no special header is included

# in SIP invite

org.jitsi.jigasi.DEFAULT_JVB_ROOM_NAME=siptest

net.java.sip.communicator.impl.protocol.SingleCallInProgressPolicy.enabled=false

# Should be enabled when using translator mode

#net.java.sip.communicator.impl.neomedia.audioSystem.audiosilence.captureDevice_list=["AudioSilenceCaptureDevice:noTransferData"]

# Adjust opus encoder complexity

net.java.sip.communicator.impl.neomedia.codec.audio.opus.encoder.COMPLEXITY=10

# Disables packet logging

net.java.sip.communicator.packetlogging.PACKET_LOGGING_ENABLED=false

net.java.sip.communicator.impl.protocol.sip.acc1403273890647=acc1403273890647

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.ACCOUNT_UID=SIP\:sip_user@sip_domain.com

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PASSWORD=<base64 pwd for sip user>

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROTOCOL_NAME=SIP

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.SERVER_ADDRESS=<sip_domain.com>

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROXY_ADDRESS=<sip_proxy_domain.com>

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROXY_AUTO_CONFIG=false

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PROXY_PORT=5061

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PREFERRED_TRANSPORT=TLS

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.USER_ID=<sip_user@sip_domain.com>

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.KEEP_ALIVE_INTERVAL=25

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.KEEP_ALIVE_METHOD=OPTIONS

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.VOICEMAIL_ENABLED=false

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.AMR-WB/16000=750

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.G722/8000=700

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.GSM/8000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.H263-1998/90000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.H264/90000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.PCMA/8000=600

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.PCMU/8000=650

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/12000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/16000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/24000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.SILK/8000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.VP8/90000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.iLBC/8000=10

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.opus/48000=1000

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.red/90000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.speex/16000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.speex/32000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.speex/8000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.telephone-event/8000=1

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.Encodings.ulpfec/90000=0

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.OVERRIDE_ENCODINGS=true

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.DEFAULT_ENCRYPTION=false


# If an authenticated (hidden) domain is used to connect to a conference,

# PREVENT_AUTH_LOGIN will prevent the SIP participant from being seen as a

# hidden participant in the conference

#net.java.sip.communicator.impl.protocol.sip.acc1403273890647.PREVENT_AUTH_LOGIN=FALSE

# Used when incoming calls are used in multidomain environment, used to detect subdomains

# used for constructing callResource and eventually contacting jicofo

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.DOMAIN_BASE=<shard1-sub_domain1.com>

# the pattern to be used as bosh url when using bosh in multidomain environment
#net.java.sip.communicator.impl.protocol.sip.acc1403273890647.BOSH_URL_PATTERN=https://{host}{subdomain}/http-bind?room={roomName}

# can be enabled to disable audio mixing and use translator, jigasi will act as jvb, just forward every ssrc stream it receives.
#net.java.sip.communicator.impl.protocol.sip.acc1403273890647.USE_TRANSLATOR_IN_CONFERENCE=true

# We can use the prefix org.jitsi.jigasi.xmpp.acc to override any of the

# properties that will be used for creating xmpp account for communication.

# The following two props assume we are using jigasi on the same machine as

# the xmpp server.

org.jitsi.jigasi.xmpp.acc.IS_SERVER_OVERRIDDEN=true

org.jitsi.jigasi.xmpp.acc.SERVER_ADDRESS=127.0.0.1

org.jitsi.jigasi.xmpp.acc.VIDEO_CALLING_DISABLED=true

org.jitsi.jigasi.xmpp.acc.JINGLE_NODES_ENABLED=false

org.jitsi.jigasi.xmpp.acc.AUTO_DISCOVER_STUN=false

org.jitsi.jigasi.xmpp.acc.IM_DISABLED=true

org.jitsi.jigasi.xmpp.acc.SERVER_STORED_INFO_DISABLED=true

org.jitsi.jigasi.xmpp.acc.IS_FILE_TRANSFER_DISABLED=true

# Or you can use bosh for the connection establishment by specifing the URL to use.

#org.jitsi.jigasi.xmp    p.acc.BOSH_URL_PATTERN=https://<shard1-sub_domain1.com>/http-bind?room={roomName}

#Used when outgoing calls are used in multidomain environment, used to detect subdomains
#org.jitsi.jigasi.xmpp.acc.DOMAIN_BASE=<<DOMAIN_BASE>>

    org.jitsi.jigasi.xmpp.acc.BOSH_URL_PATTERN=https://{host}{subdomain}/http-bind?room={roomName}

# can be enabled to disable audio mixing and use translator, jigasi will act as jvb, just forward every ssrc stream it receives.

#org.jitsi.jigasi.xmpp.acc.USE_TRANSLATOR_IN_CONFERENCE=true

# If you want jigasi to perform authenticated login instead of anonymous login

# to the XMPP server, you can set the following properties.

org.jitsi.jigasi.xmpp.acc.USER_ID=<jigasi_user_test@auth.current_server_domain.com>

org.jitsi.jigasi.xmpp.acc.PASS=test

org.jitsi.jigasi.xmpp.acc.ANONYMOUS_AUTH=false

# If you want to use the SIP user part of the incoming/outgoing call SIP URI

# you can set the following property to true.

# org.jitsi.jigasi.USE_SIP_USER_AS_XMPP_RESOURCE=true

# Activate this property if you are using self-signed certificates or other

# type of non-trusted certicates. In this mode your service trust in the

# remote certificates always.

net.java.sip.communicator.service.gui.ALWAYS_TRUST_MODE_ENABLED=true

# Enable this property to be able to shutdown gracefully jigasi using

# a rest command

# org.jitsi.jigasi.ENABLE_REST_SHUTDOWN=true

# Options regarding Transcription. Read the README for a detailed description

# about each property

#org.jitsi.jigasi.ENABLE_TRANSCRIPTION=false

#org.jitsi.jigasi.ENABLE_SIP=true

# whether to use the more expensive, but better performing

# "video" model when doing transcription

# org.jitsi.jigasi.transcription.USE_VIDEO_MODEL = false

# delivering final transcript

# org.jitsi.jigasi.transcription.DIRECTORY=/var/lib/jigasi/transcripts

# org.jitsi.jigasi.transcription.BASE_URL=http://localhost/

# org.jitsi.jigasi.transcription.jetty.port=-1

# org.jitsi.jigasi.transcription.ADVERTISE_URL=false

# save formats

# org.jitsi.jigasi.transcription.SAVE_JSON=false

# org.jitsi.jigasi.transcription.SAVE_TXT=true

# send formats

# org.jitsi.jigasi.transcription.SEND_JSON=true

# org.jitsi.jigasi.transcription.SEND_TXT=false

# translation

# org.jitsi.jigasi.transcription.ENABLE_TRANSLATION=false

# record audio. Currently only wav format is supported

# org.jitsi.jigasi.transcription.RECORD_AUDIO=false

# org.jitsi.jigasi.transcription.RECORD_AUDIO_FORMAT=wav

# execute one or more scripts when a transcript or recording is saved

# org.jitsi.jigasi.transcription.EXECUTE_SCRIPTS=true

# org.jitsi.jigasi.transcription.SCRIPTS_TO_EXECUTE_LIST_SEPARATOR=","

# org.jitsi.jigasi.transcription.SCRIPTS_TO_EXECUTE_LIST=script/example_handle_transcript_directory.sh

# filter out silent audio

#org.jitsi.jigasi.transcription.FILTER_SILENCE = false

# properties for optionally sending statistics to a DataDog server

#org.jitsi.ddclient.prefix=jitsi.jigasi

#org.jitsi.ddclient.host=localhost

#org.jitsi.ddclient.port=8125

# sip health checking

# Enables sip health checking by specifying a number/uri to call

# the target just needs to auto-connect the call play some audio,

# the call must be established for less than 10 seconds

# org.jitsi.jigasi.HEALTH_CHECK_SIP_URI=healthcheck

#

# The interval between healthcheck calls, by default is 5 minutes

# org.jitsi.jigasi.HEALTH_CHECK_INTERVAL=300000

#

# The timeout of healthcheck, if there was no successful health check for

# 10 minutes (default value) we consider jigasi unhealthy

# org.jitsi.jigasi.HEALTH_CHECK_TIMEOUT=600000

# Enabled or disable the notification when max occupants limit is reached

# org.jitsi.jigasi.NOTIFY_MAX_OCCUPANTS=false

Am I missing anything here?

Getting same error as above using above config.

I am able to make it work and connect but without any xmpp bosh url configuration (the one you mentioned above.) I just have these things enabled like:

# used for constructing callResource and eventually contacting jicofo

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.DOMAIN_BASE=<shard1-sub_domain1.com>

# the pattern to be used as bosh url when using bosh in multidomain environment

net.java.sip.communicator.impl.protocol.sip.acc1403273890647.BOSH_URL_PATTERN=https://{subdomain}/http-bind?room={roomName}

However, I wonder how haproxy plays role with jigasi. There is Jigasi-1 on shard-1 and Jigasi-2 on shard-2. And both shards in-turn have multiple domains configured. If we need to pass room name in header as room-name@conference.<sub_domain.com> then, we need to keep track of which shard and on which domain the meeting is running and our conference mapper should return that room-name like room-name@conference.shard1-sub_domain1.com

So, we can not directly use the table mapping created on haproxy for multiple shard architecture?

Sorry for trouble but, trying to achieve best possible solution and leverage what jitsi provides already as I don’t see much in the forums regarding scenario I try to achieve. Thanks

Hi @damencho
Some more clarifications will be great here. Can you help me out with the issue above?

Thanks.

Are you talking here just for dial-in?

Yes, just for dial-in.

And how do you access your deployment feom the browser? These shard1 shard2 and different domains confuse me

So, I am following the jitsi scalable architecture mentioned here where the starting point is haproxy.

I configured haproxy to have stick tables that stores the conference-shard mapping. The browser works fine because the requests are forwarded to corresponding jitsi meet shards based on the stick-table mapping.

However, I wonder how to make Jigasi work with haproxy thing.

The bosh url it uses bosh the same way as the web client, and will go through same haproxies

That’s why there are domain and room params in{} in the url as a template and jigasi replaces them with corresponding values

I see, got that point of yours. So, If I am correct, domain should be haproxy’s domain and room-name is automatically replaced from the header ?

Yep. Even domain can be replaced, based on domain_base sip header, but this is mostly used in case same jigasi is used for multiple deployments.

I see. One more doubt:

You mentioned above that conference should be provided as room-name@conference.domain.com. Is that how the server will be chosen? If so, basically, my application needs to have some kind of logic to query haproxy to figure out the room-server mapping and then append that server domain with room-name and pass it on in the header?

This is done by the conference mapper, this is how it converts from conference id (pin) in the ivr and the result is this. https://api.jitsi.net/conferenceMapper?id=3617664764

Sure damien, I am aware of that part already. We already have our own Conference Mapper that provides room-name from the provided conference id. I guess, I got the answer as well that our conference mapper needs to know about shards as well where the conference is going on and thats how the mapper can provide complete room-name.

Thanks for all help btw. Appreciated!