AWS Jicofo can't make jibri start recording - Problems with Jibri

Hello guys

I need to setup a Jitsi server with autoscalable Jibri and Jvb at AWS.

I’ve installed Jitsi in a ec2 instance with static ip and jibri in another ec2 instance with non static ip. I’ve followed the documentation and gave my jibri instace access to the ports it needs in the jitsi instance(i’ve used the instance public ip and AWS Security Groups) it all worked just fine.

I don’t have much knowledge about networks but i think i can’t scale jibri with dynamic IPs if i want to specifically allow only jibri instaces(and other jitsi components) to access prosody port, for security purposes.

So, i’ve changed my jibri’s config.js xmpp_server_hosts and xmpp_domain to my jibri instance private dns/ip. Jibri connected to the MucClient but it failed to record, it doesn’t even show the preparing to record pop-up.

My jicofo log (“my public dns” is the dns i submitted at the installation, the same i in browser to access
the application):

Jicofo 2020-06-29 15:50:57.000 INFO: [28] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Received Jibri jibribrewery@internal.auth.my public dns/jibri-nickname status
Jicofo 2020-06-29 15:50:57.001 INFO: [28] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.my public dns/jibri-nickname available: true
Jicofo 2020-06-29 15:51:05.742 INFO: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Starting session with Jibri jibribrewery@internal.auth.my public dns/jibri-nickname
Jicofo 2020-06-29 15:51:05.744 INFO: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Starting Jibri jibribrewery@internal.auth.my public dns/jibri-nickname for stream ID: null in room: inappropriateauctionscommencepatiently@conference.my public dns
Jicofo 2020-06-29 15:51:20.744 SEVERE: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Unexpected response to start request: null
Jicofo 2020-06-29 15:51:20.744 SEVERE: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Failed to send start Jibri IQ: org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Unexpected response
org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Unexpected response
at org.jitsi.jicofo.recording.jibri.JibriSession.sendJibriStartIq(JibriSession.java:582)
at org.jitsi.jicofo.recording.jibri.JibriSession.startInternal(JibriSession.java:370)
at org.jitsi.jicofo.recording.jibri.JibriSession.start(JibriSession.java:335)
at org.jitsi.jicofo.recording.jibri.JibriRecorder.handleStartRequest(JibriRecorder.java:178)
at org.jitsi.jicofo.recording.jibri.CommonJibriStuff.handleIQRequest(CommonJibriStuff.java:271)
at org.jitsi.jicofo.recording.jibri.OperationSetJibri.handleIQRequest(OperationSetJibri.java:98)
at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-06-29 15:51:20.745 INFO: [226] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Failed to start a Jibri session:Internal server error
org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Internal server error
at org.jitsi.jicofo.recording.jibri.JibriSession.startInternal(JibriSession.java:376)
at org.jitsi.jicofo.recording.jibri.JibriSession.start(JibriSession.java:335)
at org.jitsi.jicofo.recording.jibri.JibriRecorder.handleStartRequest(JibriRecorder.java:178)
at org.jitsi.jicofo.recording.jibri.CommonJibriStuff.handleIQRequest(CommonJibriStuff.java:271)
at org.jitsi.jicofo.recording.jibri.OperationSetJibri.handleIQRequest(OperationSetJibri.java:98)
at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-06-29 15:52:21.620 INFO: [28] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Received Jibri jibribrewery@internal.auth.my public dns/jibri-nickname status
Jicofo 2020-06-29 15:52:21.620 INFO: [28] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.my public dns/jibri-nickname available: true
Jicofo 2020-06-29 15:52:29.123 INFO: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Starting session with Jibri jibribrewery@internal.auth.my public dns/jibri-nickname
Jicofo 2020-06-29 15:52:29.124 INFO: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Starting Jibri jibribrewery@internal.auth.my public dns/jibri-nickname for stream ID: null in room: inappropriateauctionscommencepatiently@conference.my public dns
Jicofo 2020-06-29 15:52:44.124 SEVERE: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Unexpected response to start request: null
Jicofo 2020-06-29 15:52:44.124 SEVERE: [226] org.jitsi.jicofo.recording.jibri.JibriSession.log() Failed to send start Jibri IQ: org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Unexpected response
org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Unexpected response
at org.jitsi.jicofo.recording.jibri.JibriSession.sendJibriStartIq(JibriSession.java:582)
at org.jitsi.jicofo.recording.jibri.JibriSession.startInternal(JibriSession.java:370)
at org.jitsi.jicofo.recording.jibri.JibriSession.start(JibriSession.java:335)
at org.jitsi.jicofo.recording.jibri.JibriRecorder.handleStartRequest(JibriRecorder.java:178)
at org.jitsi.jicofo.recording.jibri.CommonJibriStuff.handleIQRequest(CommonJibriStuff.java:271)
at org.jitsi.jicofo.recording.jibri.OperationSetJibri.handleIQRequest(OperationSetJibri.java:98)
at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-06-29 15:52:44.124 INFO: [226] org.jitsi.jicofo.recording.jibri.JibriRecorder.log() Failed to start a Jibri session:Internal server error
org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Internal server error
at org.jitsi.jicofo.recording.jibri.JibriSession.startInternal(JibriSession.java:376)
at org.jitsi.jicofo.recording.jibri.JibriSession.start(JibriSession.java:335)
at org.jitsi.jicofo.recording.jibri.JibriRecorder.handleStartRequest(JibriRecorder.java:178)
at org.jitsi.jicofo.recording.jibri.CommonJibriStuff.handleIQRequest(CommonJibriStuff.java:271)
at org.jitsi.jicofo.recording.jibri.OperationSetJibri.handleIQRequest(OperationSetJibri.java:98)
at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Jicofo 2020-06-29 16:06:41.164 INFO: [28] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Received Jibri jibribrewery@internal.auth.my public dns/jibri-nickname status
Jicofo 2020-06-29 16:06:41.164 INFO: [28] org.jitsi.jicofo.recording.jibri.JibriDetector.log() Jibri: jibribrewery@internal.auth.my public dns/jibri-nickname available: true

Is it possible to do what i want here? What am i doing wrong? Am i missing some config? Is there another way to autoscale it?

I can’t review this text right now, english is not my mother language so, if you don’t understand something tell me, it will be a pleasure to clarify

what is the java version

use vpc in aws (“if i want to specifically allow only jibri instaces(and other jitsi components) to access prosody port, for security purposes”)

Hi, thanks for replying :smiley:

I’m using java 8 (OPENJDK) in all my instances.

So… I tried to allow my jibris subnet to access the needed ports from the jisti instance, by editing my subgroups. It didn’t work. But, if i allow the public ip of one of my jibri instances, that specific instance starts working fine. I believe it’s because jibri is built to connect via public net.

I’ll try to create a NAT gateway to make all my jibris have the same ip(out of my vpc) so i can allow that to access prosody and, if i do so, all my jibris in the gateway should work.

I don’t think that Jibri cares the network type. It only tries to connect using the host address. If you add a line to /etc/hosts using the private IP of the Jitsi host, Jibri will connect via the private IP.

1 Like

yeah, i kinda did that(used my jibri ec2 private dns), jibri did connect, but jicofo couldn’t “use that jibri”, even after giving permissions to my jitsi instance at jibri’s security group. Like the logs above, jicofo knew that there was a availabe jibri, it changed that jibri from IDLE to BUSY, but couldn’t make it record.
I’ll try appending to /etc/hosts later, thank you for replying, will let u know if it works

Its should work. We tried the same thing using digital ocean VPC and it works fine.

Guys, at my jibri Config.json, should i change all domain fields to my private dns or change just some specific domain fields, like xmpp_domain and/or xmpp_server_hosts ? @emrah @therecordingexperts

@henriquemgomes did you solve the issue ?

Not yet mate. I had to work on other things in the last days.
I will probably use the nat gateway to make them to have the same ip.

@henriquemgomes thank you .

I had the similar issue also using EC2 instance “t2.micro” (free tier restriction). Digging for 3 days in stackoverflow and community@jitsi, I couldn’t understand the problem due to lack of information in logs (jicofo & jibri & browser) However, after changing my instance type to t3.medium, the problem goes away.

Back then(when using t2.Micro) my Jibri recoding error repeated the following lines (no matter what kind of dns/hostname I assigned to my XMPP_DOMAIN or XMPP_HOST)

org.jitsi.jicofo.recording.jibri.JibriSession.log() Unexpected response to start request: null
org.jitsi.jicofo.recording.jibri.JibriSession.log() Failed to send start Jibri IQ: org.jitsi.jicofo.recording.jibri.JibriSession$StartException: Unexpected response
1 Like

Hi,
After seeing @CharlottQanon reply i changed my instance from a t2.medium to t3.medium.
It solved the jicofo “unexpected response” error (i don’t know why and i can’t guarantee it was the instance type)

So, i’ve changed my jibri config.json, setting xmpp_server_hosts to my jitsi meet server private ip and xmpp_domain to my public dns(same i put in browser to use the application)

Why i did that: while configuring prosody to work with jibri, we need to create some components in prosody.cfg.lua, these components are: internal muc component and a virtual host for the recorder(internal.auth.yourpubdns… and recorder.yourpubdns…) so i believe i can’t use a different domain than my public dns.

That made jibri successfully connect to the muc/brewery room via private ip

when I tried to test the recording I decided to give access from anywhere(jitsi instace inbound rules), it worked flawlessly.

What i know so far: I can make jibri join the muc using my private ip/dns but, when i press the record button, it times out. Maybe, after the start recording command, jibri make a new connection via public network and that is what we need to change and make it connect via private network. I believe someone like @damencho can confirm that. Right now i am using the netstat command to monitor the instancesg ports and trying to discover which one times out.

@Supun_Madushanka Solved It!
The problem was selenium trying to connect via public net

My scenario is 1 ec2 instance with Jitsi(i’ll call it JVM) and 1 ec2 instance with jibri(JBR)

I don’t want anyone but my ip whitelist to access my jitsi meet, so in my JVM security group i gave all traffic access to my ip list and my vpc. In theory, any JBR in my vpc would be able to connect to JVM. But JBR uses Selenium to control a google chrome that will connect to my jitsi using the dns, it is just like a person entering the domain in a browser and joining a room. The problem is, my jibri public ip isn’t in my ip whitelist(and will never be)

The solution is to force selenium resolve this dns to JVM by appending a line in your /etc/hosts
with the content: yourjitsiurl.com, after that you need to make sure that the ports JBR needs are open to your VPC at your JVM security group.

Ports jibri need to work:

80/tcp

433/tcp

5222/tcp

5223/tcp

4443/tcp

10000:20000/udp

5369/tcp