[jitsi-users] Dispatching to correct jvb pool member / shared state


#1

lets say I have 2 JVB instances registered with jicofo.

Firstly, I am assuming that what a call is provisioned jicofo earmarks that
call for a particular jvb pool member and only that pool member has been
provisoned with the necessary state to know about the call.

Please tell me if the above assumption is wrong. e.g. is it the case that
the jvb pool members somehow have a shared state where any pool member can
handle any call?

Assuming the assumption about there being a neeed to dispatch to one
particular pool member is true: what is the expected way to accomplish
this? I can think of a few options:

1] multiple public IPs for each JVB pool member.
2] single public IP with a non-overlapping port range for each JVB (e.g.
jvb1 is port range 10000 to 10099, jvb2 is 11000 to 11099 etc) so that a
NAT can forward traffic to the correct port member dispatching on the port
3] single public IP with a load balancer. but in that sense how would the
load balancer know which pool member was correct?
4] or something else?

Secondly, is there any concept of fail-over redundancy for jicofo? at the
moment this seems like a single point of failure? or can I run multiple
jicofos? or do I need a HA strategy for failing jicofo?

Thanks,
RD


#2

Hi Raoul,

lets say I have 2 JVB instances registered with jicofo.

Firstly, I am assuming that what a call is provisioned jicofo earmarks that call for a particular jvb pool member and only that pool member has been provisoned with the necessary state to know about the call.

Please tell me if the above assumption is wrong. e.g. is it the case that the jvb pool members somehow have a shared state where any pool member can handle any call?

Jicofo selects a single jitsi-videobridge instance for each conference. In the future it might select more than on jitsi-videobridge instance, but the selection logic is still in jicofo.

Assuming the assumption about there being a neeed to dispatch to one particular pool member is true: what is the expected way to accomplish this? I can think of a few options:

1] multiple public IPs for each JVB pool member.

I don't understand why you would need multiple IPs for a given pool member. We use it with a single public IP for each pool member.

2] single public IP with a non-overlapping port range for each JVB (e.g. jvb1 is port range 10000 to 10099, jvb2 is 11000 to 11099 etc) so that a NAT can forward traffic to the correct port member dispatching on the port

This works for UDP, but with TCP it defeats the purpose, as you want to specifically expose port 443 because of restrictive firewalls.

3] single public IP with a load balancer. but in that sense how would the load balancer know which pool member was correct?

This could work, but you would need to implement the load balancer.

4] or something else?

Secondly, is there any concept of fail-over redundancy for jicofo? at the moment this seems like a single point of failure? or can I run multiple jicofos? or do I need a HA strategy for failing jicofo?

Since clients only communicate with jicofo over XMPP/BOSH/HTTP, you can implement this on the HTTP level. You just need to make sure that everyone connecting to the same conference room is routed to the same jicofo instance.

Hope that helps,
Boris

···

On 13/12/2017 10:41, Raoul Duke wrote:


#3

Hi Boris,

lets say I have 2 JVB instances registered with jicofo.

Firstly, I am assuming that what a call is provisioned jicofo earmarks
that call for a particular jvb pool member and only that pool member has
been provisoned with the necessary state to know about the call.

Please tell me if the above assumption is wrong. e.g. is it the case
that the jvb pool members somehow have a shared state where any pool member
can handle any call?

Jicofo selects a single jitsi-videobridge instance for each conference. In
the future it might select more than on jitsi-videobridge instance, but the
selection logic is still in jicofo.

So just to be clear: that means if jicofo selects jvb3 then *only* jvb3 can
handle the call (and not, say, jvb1 or jvb2)?

Assuming the assumption about there being a neeed to dispatch to one
particular pool member is true: what is the expected way to accomplish
this? I can think of a few options:

1] multiple public IPs for each JVB pool member.

I don't understand why you would need multiple IPs for a given pool
member. We use it with a single public IP for each pool member.

I think my wording was unclear. I mean each jvb has a distinct public IP
so that the correct pool member can be unambiguously dispatched to by
public IP:

e.g.

jvb1 is <public address 1>
jvb2 is <public address 2>
jvb3 is <public address 3>

which is OK as it means the clients can connect to the correct jvb by IP.
but what if I only have 1 public IP?

2] single public IP with a non-overlapping port range for each JVB (e.g.

jvb1 is port range 10000 to 10099, jvb2 is 11000 to 11099 etc) so that a
NAT can forward traffic to the correct port member dispatching on the port

This works for UDP, but with TCP it defeats the purpose, as you want to
specifically expose port 443 because of restrictive firewalls.

I'm not sure what you mean wrt. TCP port 443 (I am not using meet, just
videobridge, jicofo, xmpp from a native android application).
To be clear, I am discussing the UDP SRTP media channels.

3] single public IP with a load balancer. but in that sense how would the

load balancer know which pool member was correct?

This could work, but you would need to implement the load balancer.

but what would the load balancing logic be to target the correct pool
member? if only jicofo knows which pool member is correct then what is the
logic the load balancer can implement to target the correct pool member?

NOTE: it may be I have some incorrect assumptions if my responses seem out
of whack with your mental model. If so please clarify.

Thanks,
RD

···

On Wed, Dec 13, 2017 at 6:02 PM, Boris Grozev <boris@jitsi.org> wrote:

On 13/12/2017 10:41, Raoul Duke wrote:


#4

Hi Raoul,

>
> Hi Boris,
>
> lets say I have 2 JVB instances registered with jicofo.
>
> Firstly, I am assuming that what a call is provisioned jicofo
> earmarks that call for a particular jvb pool member and only
> that pool member has been provisoned with the necessary state to
> know about the call.
>
> Please tell me if the above assumption is wrong. e.g. is it the
> case that the jvb pool members somehow have a shared state where
> any pool member can handle any call?
>
> Jicofo selects a single jitsi-videobridge instance for each
> conference. In the future it might select more than on
> jitsi-videobridge instance, but the selection logic is still in jicofo.
>
> So just to be clear: that means if jicofo selects jvb3 then *only* jvb3
> can handle the call (and not, say, jvb1 or jvb2)?
Correct. Note that jicofo actively communicates with the chosen jitsi-videobridge and configures it to accept a session from a given participant.
>
> Assuming the assumption about there being a neeed to dispatch to
> one particular pool member is true: what is the expected way to
> accomplish this? I can think of a few options:
>
> 1] multiple public IPs for each JVB pool member.
>
> I don't understand why you would need multiple IPs for a given pool
> member. We use it with a single public IP for each pool member.
>
> I think my wording was unclear. I mean each jvb has a distinct public
> IP so that the correct pool member can be unambiguously dispatched to by
> public IP:
>
> e.g.
>
> jvb1 is <public address 1>
> jvb2 is <public address 2>
> jvb3 is <public address 3>
>
> which is OK as it means the clients can connect to the correct jvb by
> IP.
Yes, that makes sense.

> but what if I only have 1 public IP?
Unfortunately this is more tricky. See below.>
>
> 2] single public IP with a non-overlapping port range for each
> JVB (e.g. jvb1 is port range 10000 to 10099, jvb2 is 11000 to
> 11099 etc) so that a NAT can forward traffic to the correct port
> member dispatching on the port
>
> This works for UDP, but with TCP it defeats the purpose, as you want
> to specifically expose port 443 because of restrictive firewalls.
>
> I'm not sure what you mean wrt. TCP port 443 (I am not using meet, just
> videobridge, jicofo, xmpp from a native android application).
> To be clear, I am discussing the UDP SRTP media channels.
Some clients are behind restrictive firewalls which do not allow UDP. Because of this in production we have jitsi-videobridge use TCP/443 for the media connections (that is, SRTP goes over TCP/443). These firewalls often also block TCP ports other than 443, so multiplexing ports on the same IP address doesn't work.

Probably the easiest option is to use the port multiplexing which you suggested for UDP, and add a TURN server running on TCP/443, which then connects to jitsi-videobridge over UDP, i.e.:

client <-- tcp/443 --> TURN server <-- udp/10123 --> jitsi-videobridge

> 3] single public IP with a load balancer. but in that sense how
> would the load balancer know which pool member was correct?
>
> This could work, but you would need to implement the load balancer.
>
> but what would the load balancing logic be to target the correct pool
> member? if only jicofo knows which pool member is correct then what is
> the logic the load balancer can implement to target the correct pool member?
> NOTE: it may be I have some incorrect assumptions if my responses seem
> out of whack with your mental model. If so please clarify.
We're out of sync, but I'm not sure where. This is the flow of a conference with jitsi-meet:
1. jitsi-meet connects to jicofo
2. jicofo selects a jitsi-videobridge and allocates channels on it
3. jicofo provides jitsi-meet with the address of the jitsi-videobridge for the conference
4. jitsi-meet connects to jitsi-videobridge (with ICE)

So if the client connects to the correct jicofo instance, jicofo will take care to connect it to the correct jitsi-videobridge.

Regards,
Boris

···

On 13/12/2017 13:21, Raoul Duke wrote:
> On Wed, Dec 13, 2017 at 6:02 PM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org>> wrote:
> On 13/12/2017 10:41, Raoul Duke wrote:


#5

Hi Boris,

>
>
>
> I'm not sure what you mean wrt. TCP port 443 (I am not using meet, just
> videobridge, jicofo, xmpp from a native android application).
> To be clear, I am discussing the UDP SRTP media channels.
Some clients are behind restrictive firewalls which do not allow UDP.
Because of this in production we have jitsi-videobridge use TCP/443 for the
media connections (that is, SRTP goes over TCP/443). These firewalls often
also block TCP ports other than 443, so multiplexing ports on the same IP
address doesn't work.

Probably the easiest option is to use the port multiplexing which you
suggested for UDP, and add a TURN server running on TCP/443, which then
connects to jitsi-videobridge over UDP, i.e.:

client <-- tcp/443 --> TURN server <-- udp/10123 --> jitsi-videobridge

Oh that is interesting information. Thankyou.

> 3] single public IP with a load balancer. but in that sense how
> would the load balancer know which pool member was correct?
>
>
> This could work, but you would need to implement the load balancer.
>
>
> but what would the load balancing logic be to target the correct pool
> member? if only jicofo knows which pool member is correct then what is
> the logic the load balancer can implement to target the correct pool
member?
> NOTE: it may be I have some incorrect assumptions if my responses seem
> out of whack with your mental model. If so please clarify.
We're out of sync, but I'm not sure where. This is the flow of a
conference with jitsi-meet:
1. jitsi-meet connects to jicofo
2. jicofo selects a jitsi-videobridge and allocates channels on it
3. jicofo provides jitsi-meet with the address of the jitsi-videobridge
for the conference
4. jitsi-meet connects to jitsi-videobridge (with ICE)

So if the client connects to the correct jicofo instance, jicofo will take
care to connect it to the correct jitsi-videobridge.

From my perspective my thought process is:

* lets say there are 3 jvb instances in a pool (jvb1, jvb2 and jvb3)
behind just one public IP (NAT)
* lets say that a given call is orchestrated by jicofo to use jvb2
* lets say that an ICE candidate is proposed to the user of UDP port
(10003) for jvb2
* how can I ensure that my single public IP can NAT to the correct jvb
(jvb2)

I understand that I can use a pre-defined port range for each jvb which is
configured in the NAT to forward to the correct pool member. but now my
NAT has to explicitly "know" about all my pool members. In HTTP, say, I
could hide my pool members behind a load balancer and it would health check
them etc.

So I was wondering: is there a way to use a load balancer instead of a NAT
port range for each pool member? Maybe it is just a red herring I have
created by mentioning load balancers. I'm really just curious if there is
any way to use a load balancer in this case. But what would the logic in
the load balancer be to know which jvb it should target? Like I say it may
be just a red herring.

Is there any plans to make it so jvbs can be more "dumb" so that any pool
member can handle a media stream. e.g. by having some shared state between
pool members (like coturn can do with redis). If that makes any sense
(maybe not).

Thanks so much for your responses and patience,

RD

···

On Wed, Dec 13, 2017 at 9:09 PM, Boris Grozev <boris@jitsi.org> wrote:

On 13/12/2017 13:21, Raoul Duke wrote:


#6

Hi Raoul,

Hi Boris,

[snip]

From my perspective my thought process is:

* lets say there are 3 jvb instances in a pool (jvb1, jvb2 and jvb3) behind just one public IP (NAT)
* lets say that a given call is orchestrated by jicofo to use jvb2
* lets say that an ICE candidate is proposed to the user of UDP port (10003) for jvb2
* how can I ensure that my single public IP can NAT to the correct jvb (jvb2)

OK, I think I understand now.

I understand that I can use a pre-defined port range for each jvb which is configured in the NAT to forward to the correct pool member. but now my NAT has to explicitly "know" about all my pool members. In HTTP, say, I could hide my pool members behind a load balancer and it would health check them etc.

Note that unless you want to use jigasi or other components which do not (yet) support bundle and rtcp-mux, the range can be collapsed to a single port.

So I was wondering: is there a way to use a load balancer instead of a NAT port range for each pool member? Maybe it is just a red herring I have created by mentioning load balancers. I'm really just curious if there is any way to use a load balancer in this case. But what would the logic in the load balancer be to know which jvb it should target? Like I say it may be just a red herring.

One option, which we used at some point in the past, is to use the ICE USERNAME field for this. Part of the USERNAME is generated by jvb, and you can configure it to encode some identifier recognized by your load balancer (see [0]).

But it is probably easier to deploy the jvb machines with configuration about the port that they use, and update the NAT rules.

Is there any plans to make it so jvbs can be more "dumb" so that any pool member can handle a media stream. e.g. by having some shared state between pool members (like coturn can do with redis). If that makes any sense (maybe not).

We don't have any plans for this.

Regards,
Boris

[0] https://github.com/jitsi/jitsi-videobridge/blob/master/src/main/java/org/jitsi/videobridge/IceUdpTransportManager.java#L141

···

On 13/12/2017 15:33, Raoul Duke wrote:

On Wed, Dec 13, 2017 at 9:09 PM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org>> wrote: