Easy way to create a Jitsi cluster

emrah · March 11, 2023, 8:38am

Glad to here it is solved for you but I didn’t understand why it doesn’t work at first. When you check the created instance, is there any running container on it?

On jibri instance

lxc-ls -f

PRT · March 11, 2023, 9:22am

Outputs:

JIBRI created by non modified (base file with no changes by me) aws-jibri-auto-scale script and not added by add-jibri-node IP (recording not working and not getting added to JMS)
-bash: lxc-ls: command not found
JIBRI create by the modified (by me) aws-jibri-auto-scale i.e the add-jibri-node IP was executed in JMS through extra code mentioned earlier by me (recording working and detected by JMS)

NAME                                        STATE   AUTOSTART GROUPS             IPV4          IPV6 UNPRIVILEGED
eb-bullseye                                 STOPPED 0         -                  -             -    false
eb-jibri-99961da4a93949a59928a9ee67157f79-1 RUNNING 1         eb-group, eb-jibri 172.22.22.239 -    false
eb-jibri-template                           STOPPED 1         eb-group, eb-jibri -             -    false

JMS

NAME                  STATE   AUTOSTART GROUPS           IPV4         IPV6 UNPRIVILEGED
eb-builder            STOPPED 0         eb-group         -            -    false
eb-bullseye           STOPPED 0         -                -            -    false
eb-component-selector RUNNING 1         eb-group, onboot 172.22.22.16 -    false
eb-jitsi              RUNNING 1         eb-group, onboot 172.22.22.14 -    false

Please note that i have created the JIBRI for the launch template by running only the lines mentioned
in the url
(emrah-bullseye-templates/jitsi-cluster.md at main · emrahcom/emrah-bullseye-templates · GitHub)
I only executed the lines of Additional Jibri node node section in the jibri ec2 from which i made the AMI for the launch template

emrah · March 11, 2023, 9:31am

Then there is something wrong with AMI you used for launch template.

To be clear, this AMI must be created using an already installed jibri machine. If you create your AMI using this way, you should find why the installed packages are not there. It may be related with the storage type which prevents permanent data.

PRT · March 11, 2023, 10:04am

Thank you very much Emrah,

When I created a new launch template by using the AMI created from a JIBRI node for which I have earlier run the command add-jibri-node IP in JMS everything starts working.

The autoscaling is working like a charm, thank you very much for providing such an easy setup, and apologies for any inconvenience caused.

The mistake I was making was using a machine without a JIBRI machine because of my wrongful assumption that after only running the command in the Additional Jibri node section of your document (emrah-bullseye-templates/jitsi-cluster.md at main · emrahcom/emrah-bullseye-templates · GitHub) everything will work.

PRT · March 13, 2023, 2:54pm

Hi Emrah,

Shall I download the same script for aws-jvb-auto-scale or the one already present will work.

I have noticed that the change mentioned below is present in aws-jibri-auto-scale but it is not present in aws-jvb-auto-scale

#container
        con=$(ssh -qF $SSH_CONFIG $ip -- \
            lxc-ls --running --filter 'eb-jibri-\[0-9a-z\]+-1$' || true)
        [[ -z "$con" ]] && continue

Shall I use the same snippet and replace jibri with jvb.

emrah · March 13, 2023, 3:09pm

JVB script is OK because additional Jibri and additional JVB have different structure.

In an additional JVB, there is always one JVB container with a static name which is eb-jvb.

In an additional Jibri, there may be more than one Jibri instance with a dynamically generated name.

PRT · March 14, 2023, 7:49am

Hi Emrah,

I was setting up the JVB for autoscaling, the particular command curl -s http://127.0.0.1:8080/colibri/conferences is not working as expected in the script the same thing happens when i run the command in the running container, while curl -s http://127.0.0.1:8080/colibri/stats works fine.

curl -s http://127.0.0.1:8080/colibri/conferences

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404 Not Found</h2>
<table>
<tr><th>URI:</th><td>/colibri/conference</td></tr>
<tr><th>STATUS:</th><td>404</td></tr>
<tr><th>MESSAGE:</th><td>Not Found</td></tr>
<tr><th>SERVLET:</th><td>org.glassfish.jersey.servlet.ServletContainer-23ee75c5</td></tr>
</table>
<hr/><a href="https://eclipse.org/jetty">Powered by Jetty:// 11.0.10</a><hr/>

</body>
</html>

 # curl -s http://127.0.0.1:8080/colibri/stats
{"endpoints_with_suspended_sources":0,"inactive_endpoints":0,"inactive_conferences":0,"total_ice_succeeded_relayed":0,"colibri2":true,"total_loss_degraded_participant_seconds":0,"bit_rate_download":0.0,"local_active_endpoints":0,"muc_clients_connected":1,"total_participants":0,"total_packets_received":0,"rtt_aggregate":0.0,"packet_rate_upload":0,"p2p_conferences":0,"total_aimd_bwe_expirations":0,"total_loss_limited_participant_seconds":0,"preemptive_kfr_suppressed":0,"local_endpoints":0,"octo_send_bitrate":0.0,"total_dominant_speaker_changes":0,"endpoints_with_spurious_remb":0,"receive_only_endpoints":0,"total_colibri_web_socket_messages_received":0,"octo_receive_bitrate":0.0,"total_visitors":0,"version":"2.2.79-gf6426ea0","total_ice_succeeded":0,"total_colibri_web_socket_messages_sent":0,"total_bytes_sent_octo":0,"total_data_channel_messages_received":0,"total_conference_seconds":0,"visitors":0,"num_eps_oversending":0,"bit_rate_upload":0.0,"total_conferences_completed":0,"octo_conferences":0,"num_eps_no_msg_transport_after_delay":0,"endpoints_sending_video":0,"packet_rate_download":0,"muc_clients_configured":1,"outgoing_loss":0.0,"overall_loss":0.0,"conference_sizes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"total_packets_sent_octo":0,"total_layering_changes_received":0,"total_relays":0,"conferences_by_video_senders":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"endpoints_with_high_outgoing_loss":0,"stress_level":0.0,"jitter_aggregate":0.0,"drain":false,"total_video_stream_milliseconds_received":0,"total_ice_succeeded_tcp":0,"octo_endpoints":0,"current_timestamp":"2023-03-14 07:46:04.345","num_relays_no_msg_transport_after_delay":0,"conferences":0,"participants":0,"total_keyframes_received":0,"average_participant_stress":0.01,"largest_conference":0,"total_packets_sent":0,"endpoints":0,"total_data_channel_messages_sent":0,"incoming_loss":0.0,"total_bytes_received_octo":0,"octo_send_packet_rate":0,"conferences_by_audio_senders":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"total_conferences_created":0,"total_ice_failed":0,"preemptive_kfr_sent":0,"threads":35,"videochannels":0,"total_packets_received_octo":0,"endpoints_reconnected":0,"graceful_shutdown":false,"octo_receive_packet_rate":0,"total_bytes_received":0,"total_loss_controlled_participant_seconds":0,"total_partially_failed_conferences":0,"endpoints_disconnected":0,"endpoints_sending_audio":0,"dtls_failed_endpoints":0,"healthy":true,"total_bytes_sent":0,"mucs_configured":1,"total_failed_conferences":0,"mucs_joined":1}

emrah · March 14, 2023, 8:38am

checking…

emrah · March 14, 2023, 9:13am

You are right, there is an issue because of some colibri changes. It is fixed now. You may run the following command on Jitsi host to get the fixed version of script:

wget -O /usr/local/sbin/aws-jvb-auto-scale https://raw.githubusercontent.com/emrahcom/emrah-bullseye-templates/main/machines/eb-jitsi-host/usr/local/sbin/aws-jvb-auto-scale

For more information about the issue, see

PRT · March 14, 2023, 9:29am

Thanks a lot, Emrah,

The JVB autoscaling is working fine now, thank you very much for helping

emrah · March 14, 2023, 9:49am

As you know these features are not documented and they were implemented for our special needs. Therefore I have to inform you about some behaviours:

JVB autoscaler creates a seperate JVB instance for each conference. It doesn’t check the load, it checks the number of conferences
Normally an additional Jibri can run multiple Jibri services by running multiple Jibri containers but Jibri autoscaler assumes that there is only one Jibri service on each instance. Therefore use an EC2 instance which has 4 cores

PRT · March 14, 2023, 9:57am

Thank you Emrah,

I have used 4 cores for the Jibri instance, for the JVB script I will try to modify it as per the load.

Thank you for the helpful information

PRT · March 14, 2023, 10:57am

Hi Emrah,

I just had a doubt,

Does the JMS automatically switch traffic to an external JVB added from the autoscale script when the load is high or do we have to configure it somewhere, for instance, I have 2 external JVBs now, when will the JMS use any one of them?

emrah · March 14, 2023, 11:13am

jicofo will decide which JVB should be used. You don’t need to configure anything.

PRT · March 14, 2023, 11:15am

Thanks a lot, Emrah.

PRT · March 15, 2023, 6:34am

Hi Emrah,

is there a way to set-up JWT token authentication in the JMS, earlier in the standalone Jitsi Meet I used the following commands:

apt-get install lua5.2 liblua5.2-dev luarocks
apt-get install jitsi-meet-tokens

I tried running this inside the eb-jitsi container (JMS) it prompts me to enter the application id and secret as shown below, and both the app id and secret in the prosody file
/etc/prosody/conf.avail/[your-hostname].cfg.lua
was set with empty strings

emrah · March 15, 2023, 7:41am

You don’t need the first part, these packages should be already there. You need only the second command:

apt-get install jitsi-meet-tokens

But there is a problem in your dialog screen which doesn’t look correct. This may be related with your terminal emulator. As a quick solution, you may set these values on the commandline:

APP_ID=yourappid
APP_SECRET=yourappsecret
debconf-set-selections <<< "jitsi-meet-tokens jitsi-meet-tokens/appid string $APP_ID"
debconf-set-selections <<< "jitsi-meet-tokens jitsi-meet-tokens/appsecret password $APP_SECRET"
apt-get -y --allow-change-held-packages install jitsi-meet-tokens
apt-mark hold jitsi-meet-tokens

PRT · March 15, 2023, 10:26am

Hi Emrah,

Thank you, the JWT auth is working now, but I am receiving the following error in jibri node, the recorder is being detected but the recording is not starting, can you please guide me

2023-03-15 09:57:08.843 INFO: [89] AbstractPageObject.visit#32: Visiting url https://jitsi.mydomain.com
2023-03-15 09:57:10.332 INFO: [89] AbstractPageObject.visit#38: Waited 1.486223563s for driver to load page
2023-03-15 09:57:10.384 INFO: [89] AbstractPageObject.visit#32: Visiting url https://jitsi.mydomain.com/12345#config.iAmRecorder=true&config.externalConnectUrl=null&config.startWithAudioMuted=true&config.startWithVideoMuted=true&interfaceConfig.APP_NAME="Jibri"&config.analytics.disabled=true&config.p2p.enabled=false&config.prejoinPageEnabled=false&config.prejoinConfig.enabled=false&config.requireDisplayName=false
2023-03-15 09:57:11.315 INFO: [89] AbstractPageObject.visit#38: Waited 930.055105ms for driver to load page
2023-03-15 09:57:11.349 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: Cannot read properties of undefined (reading 'isJoined')
2023-03-15 09:57:11.855 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: Cannot read properties of undefined (reading 'isJoined')
2023-03-15 09:57:12.361 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: Cannot read properties of undefined (reading 'isJoined')
2023-03-15 09:57:12.868 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: Cannot read properties of undefined (reading 'isJoined')
2023-03-15 09:57:13.374 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:13.895 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:14.401 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:14.907 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:15.413 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:15.919 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:16.425 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:16.930 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:17.435 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:17.940 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:18.446 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:18.951 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:19.457 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:19.962 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:20.467 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:20.972 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:21.478 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:21.983 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:22.489 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:22.994 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined
2023-03-15 09:57:23.499 FINE: [89] CallPage.visit$lambda-2$lambda-1#58: Not joined yet: APP is not defined

emrah · March 15, 2023, 11:31am

There is a special case for jibri when JWT is enabled. You should add the recorder account into admins list.

In your prosody config (/etc/prosody/conf.d/jitsi.yourdomain.com.cfg.lua)
Find the conference MUC
Component "conference.jitsi.yourdomain.com" "muc"

Add the recorder account into admins

admins = { "focus@auth.jitsi.yourdomain.com", "recorder@recorder.jitsi.yourdomain.com" }

PRT · March 15, 2023, 11:51am

Thank you Emrah recording is working fine now