Testing Octo Setup

omesta · May 14, 2020, 6:17pm

I have just finished upgrade and tested; it works now
i can now see octo conference in colibri stats, i can see packets going on each jvb and i can see the different region information on the screen…

Thank you all @msansnom @akalana @migo @blivingston @creativeguitar for your help; i think it is all about jicofo version

now i have
dpkg -s jicofo | grep Version
Version: 1.0-586-1

bbaldino · May 14, 2020, 6:44pm

A conference is considered ‘failed’ if it has an endpoint which failed to connect via ICE and no endpoints which succeeded to connect via ICE, so it’s not the same as the Octo signaling issue. You’d have to look into the ICE failures in the logs to determine if it was a problem. Do you have TURN set up?

migo · May 14, 2020, 6:53pm

Hi @bbaldino, I’ve checked your suggestions and nr. 3 is likely causing Octo droped packets. There are no:

root@virt1:~/scripts# cat /var/log/jitsi/jvb.log|grep “Unsupported media type”
root@virt1:~/scripts# cat /var/log/jitsi/jvb.log|grep “Invalid Octo packet”
root@virt1:~/scripts#

only:

root@virt1:~/scripts# cat /var/log/jitsi/jvb.log|grep “Octo packets for unknown conference”|tail -n 5
2020-05-14 17:46:29.907 WARNING: [64] [relayId=19.xx.1.1xx:4096] BridgeOctoTransport.dataReceived#139: Received 100000 Octo packets for unknown conference ff0939
2020-05-14 18:23:24.137 WARNING: [64] [relayId=19.xx.1.1xx:4096] BridgeOctoTransport.dataReceived#139: Received 10 Octo packets for unknown conference ff3d05
2020-05-14 18:23:25.947 WARNING: [64] [relayId=19.xx.1.1xx:4096] BridgeOctoTransport.dataReceived#139: Received 100 Octo packets for unknown conference ff3d05
2020-05-14 18:55:50.281 WARNING: [64] [relayId=19.xx.1.1xx:4096] BridgeOctoTransport.dataReceived#139: Received 10 Octo packets for unknown conference ff6e6a
2020-05-14 18:55:50.948 WARNING: [64] [relayId=19.xx.1.1xx:4096] BridgeOctoTransport.dataReceived#139: Received 100 Octo packets for unknown conference ff6e6a
root@virt1:~/scripts#

root@virt1:~/scripts# cat /var/log/jitsi/jvb.log|grep “Octo packets for unknown conference”|wc -l
110
root@virt1:~/scripts#

From time stamp it seems Octo droped packets number increases when last conference participant on JVB leaves room (conference destroyed on that JVB?) and conference is hosted only on the second JVB. Here is corresponding Jicofo log for that room:

meet:~/scripts# cat /var/log/jitsi/jicofo.log|grep ff6e6a
Jicofo 2020-05-14 18:11:11.219 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU, UKU]]
Jicofo 2020-05-14 18:11:11.220 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Creating an Octo participant for Bridge[jid=jvbbrewery@internal.auth.meet.UKU.domain.com/0b4864ed-b468-4760-b358-19d4d8b40fb4, relayId=19.xx.1.9:4096, region=UKU, stress=0.02] in JitsiMeetConferenceImpl[id=ff6e6a, name=balogh@conference.meet.UKU.domain.com]
Jicofo 2020-05-14 18:11:11.221 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Creating an Octo participant for Bridge[jid=jvbbrewery@internal.auth.meet.UKU.domain.com/6e01078b-a171-4213-b0ef-c82aa89e6d1b, relayId=19.xx.1.1xx:4096, region=UKU, stress=0.04] in JitsiMeetConferenceImpl[id=ff6e6a, name=balogh@conference.meet.UKU.domain.com]
Jicofo 2020-05-14 18:11:11.221 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU, UKU][UKU, UKU]]
Jicofo 2020-05-14 18:11:39.457 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU, UKU, UKU][UKU, UKU]]
Jicofo 2020-05-14 18:55:50.205 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU, UKU, UKU][UKU]]
Jicofo 2020-05-14 18:55:50.206 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU, UKU, UKU][UKU]]
Jicofo 2020-05-14 18:55:53.813 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU, UKU]]
Jicofo 2020-05-14 18:55:55.010 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU]]
Jicofo 2020-05-14 18:55:55.010 INFO: [30] org.jitsi.jicofo.JitsiMeetConferenceImpl.log() Region info, conference=ff6e6a octo_enabled= true: [[UKU]]
meet:~/scripts#

Do you need more info from logs or it is clear now?

Thank you very much!

Milan

bbaldino · May 14, 2020, 6:57pm

Yes that makes sense, in that case those drops shouldn’t be a problem.

migo · May 14, 2020, 7:10pm

Thank you @bbaldino, those ICE failures would be logged as WARNINGS?

root@virt1:~/scripts# cat /var/log/jitsi/jvb.log|grep “WARN”|grep “ICE”
root@virt1:~/scripts#

Yes we had turn setup, I think at least. Package is installed with no errors, are there any additional steps needed? How it can affect those jitsi_total_failed_conferences numbers? Strange is that before Octo setup there were zero jitsi_total_failed_conferences for over one month time period.

Thank you!

Milan

migo · May 14, 2020, 8:30pm

Hi @bbaldino, JVB startup seem ok?

2020-05-14 22:26:04.996 INFO: [21] OctoRelayService.start#55: Created Octo UDP transport
2020-05-14 22:26:05.000 WARNING: [62] MucClient.createXMPPTCPConnectionConfiguration#115: Disabling certificate verification!
2020-05-14 22:26:05.003 INFO: [21] [relayId=19x.xx.1x.15:4096] BridgeOctoTransport.#78: Created OctoTransport
2020-05-14 22:26:05.005 INFO: [21] TaskPools.#81: TaskPools detected 24 processors, creating the CPU pool with that many threads
2020-05-14 22:26:05.051 INFO: [61] org.ice4j.ice.harvest.StunMappingCandidateHarvester.discover: Discovered public address 19x.xx.1x.15:58934/udp from STUN server 3.125.222.191:443/udp using local address 19x.xx.1x.15:0/udp
2020-05-14 22:26:05.051 INFO: [59] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Using org.ice4j.ice.harvest.StunMappingCandidateHarvester, face=/19x.xx.1x.15, mask=/19x.xx.1x.15
2020-05-14 22:26:05.052 INFO: [59] org.ice4j.ice.harvest.MappingCandidateHarvesters.initialize: Initialized mapping harvesters (delay=97ms). stunDiscoveryFailed=false
2020-05-14 22:26:05.194 INFO: [62] [hostname=meet.domain.com id=shard] MucClient$1.connected#266: Connected.
2020-05-14 22:26:05.194 INFO: [62] [hostname=meet.domain.com id=shard] MucClient.lambda$getConnectAndLoginCallable$7#648: Logging in.
2020-05-14 22:26:05.271 INFO: [62] [hostname=meet.domain.com id=shard] MucClient$MucWrapper.join#751: Joined MUC: jvbbrewery@internal.auth.meet.domain.com
2020-05-14 22:26:14.972 INFO: [60] Videobridge.createConference#320: create_conf, id=300f4202cc20814c gid=null logging=false
2020-05-14 22:26:15.091 INFO: [60] org.ice4j.ice.harvest.AbstractUdpListener.: Initialized AbstractUdpListener with address 19x.xx.1x.15:10000/udp. Receive buffer size 10485760 (asked for 10485760)
2020-05-14 22:26:15.092 INFO: [60] org.ice4j.ice.harvest.SinglePortUdpHarvester.: Initialized SinglePortUdpHarvester with address 19x.xx.1x.15:10000/udp
SCTP JNI load: Linux OS detected
SCTP lib loaded

Or not?

Thank you!

Milan

bbaldino · May 14, 2020, 10:47pm

Looks fine to me.

migo · May 15, 2020, 7:01am

Hi @bbaldino, thank you. I think your suggestion on jitsi_total_failed_conferences count may be related to Turn setup is right.
I’ve disbaled udp traffic from my PC (iptables -I OUTPUT 1 -p udp -d jvb-server.mydomain.com -j REJECT) and video doesn’t work(reconnecting loop after while in UI), so I assume Turn is not working. On signaling server is coturn service running and listening:

meet:~/scripts# netstat -nap|grep tcp 0 0 19.xx.1.63:4445 tcp 0 0 19.xx.1.63:4445 tcp 0 0 19.xx.1.63:4445 tcp 0 0 19.xx.1.63:4445 tcp 0 0 127.0.0.1:4445 tcp 0 0 127.0.0.1:4445 tcp 0 0 127.0.0.1:4445 tcp 0 0 127.0.0.1:4445 tcp6 0 0 ::1:4445 tcp6 0 0 ::1:4445 tcp6 0 0 ::1:4445 tcp6 0 0 ::1:4445 sctp ::1:4445 sctp 19.xx.1.63:4445 sctp 127.0.0.1:4445 udp 0 0 19.xx.1.63:443 udp 0 0 19.xx.1.63:443 udp 0 0 19.xx.1.63:443 udp 0 0 19.xx.1.63:443 udp 0 0 127.0.0.1:443 udp 0 0 127.0.0.1:443 udp 0 0 127.0.0.1:443 udp 0 0 127.0.0.1:443 udp 0 0 19.xx.1.63:4445 udp 0 0 19.xx.1.63:4445 udp 0 0 19.xx.1.63:4445 udp 0 0 19.xx.1.63:4445 udp 0 0 127.0.0.1:4445 udp 0 0 127.0.0.1:4445 udp 0 0 127.0.0.1:4445 udp 0 0 127.0.0.1:4445 udp6 0 0 ::1:443 udp6 0 0 ::1:443 udp6 0 0 ::1:443 udp6 0 0 ::1:443 udp6 0 0 ::1:4445 udp6 0 0 ::1:4445 udp6 0 0 ::1:4445 udp6 0 0 ::1:4445 unix 2 DGRAM 14939 638/turnserver

When I look at: https://github.com/jitsi/jitsi-meet/blob/master/doc/manual-install.md there is port 4443/TCP on JVBs servers open to WAN, but there is no process listening on that port:

root@BackupStorage:~/scripts# netstat -anp|grep 4443
root@BackupStorage:~/scripts#

root@virt1:~/scripts# netstat -nap |grep 4443
root@virt1:~/scripts#

Where should I look for error? Is turn reconfiguration or some additional steps needed when running Octo?

Thank you for your help!

Milan

bbaldino · May 15, 2020, 4:03pm

We don’t listen on 443/4443 on the JVB directly anymore, we rely on a separate TURN for that so you have to set up a TURN server.

migo · May 15, 2020, 6:58pm

Hi @bbaldino, thank you. Yes I’ve setup turn server, but it can’t be reached on stun:meet.mydomain.com:443 when i check it with: https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/

but when I open 4445/UDP to WAN and check it with: stun:meet.mydomain.com:4445 it works:

Time Component Type Foundation Protocol Address Port Priority
0.003 1 host 0 udp c35b9071-a71b-4952-9497-e683f8ee60cf.local 54703 126 | 32512 | 255
0.004 1 host 2 tcp c35b9071-a71b-4952-9497-e683f8ee60cf.local 9 125 | 32704 | 255
0.004 2 host 0 udp c35b9071-a71b-4952-9497-e683f8ee60cf.local 35536 126 | 32512 | 254
0.004 2 host 2 tcp c35b9071-a71b-4952-9497-e683f8ee60cf.local 9 125 | 32704 | 254
0.013 1 srflx 1 udp 5.212.154.8 54703 100 | 32543 | 255
0.035 2 srflx 1 udp 5.212.154.8 35536 100 | 32543 | 254
0.036 Done

meet:/etc# cat /etc/turnserver.conf
#jitsi-meet coturn config. Do not modify this line
use-auth-secret
keep-address-family
static-auth-secret=ffUzeWdshL
realm=meet.mydomain.com
cert=/etc/jitsi/meet/meet.mydomain.com.crt
pkey=/etc/jitsi/meet/meet.mydomain.com.key

no-tcp
listening-port=4446
tls-listening-port=4445
external-ip=meet.mydomain.com

syslog

meet:/etc# cat /etc/nginx/modules-enabled/60-jitsi-meet.conf
#this is jitsi-meet nginx module configuration
#this forward all http traffic to the nginx virtual host port
#and the rest to the turn server

stream {
upstream web {
server 127.0.0.1:4444;
}
upstream turn {
server 127.0.0.1:4445;
}
# since 1.13.10
map $ssl_preread_alpn_protocols $upstream {
~\bh2\b web;
~\bhttp/1. web;
default turn;
}

server {
    listen 443;
    listen [::]:443;

    # since 1.11.5
    ssl_preread on;
    proxy_pass $upstream;

    # Increase buffer to serve video
    proxy_buffer_size 10m;
}

}
meet:/etc#

Do you see something wrong here? How can I force TCP connection from client to test if turn works, when I set: iptables -I OUTPUT 1 -p udp --dport 10000 -j DROP it doesn’t work.

Thank you,

Milan

bbaldino · May 15, 2020, 8:34pm

Sorry, I’m not familiar with the TURN setup. Maybe @damencho

migo · May 15, 2020, 8:37pm

@bbaldino Thank you anyway!

Milan

migo · May 18, 2020, 7:28pm

Hi @bbaldino, thank you pointing me on Turn server, it wasn’t reachable from WAN, now it is fully working and jitsi_total_failed_conferences count is zero for today. Today was calm day on meet, what alarms me are: Unable to find encoding matching packet: 921824 messages

Can you give me some hint for this too, please?

root@BackupStorage:~/scripts# ./stats.sh
JVB stats:
jitsi_bit_rate_upload: 0
jitsi_bit_rate_download: 0
jitsi_participants: 0
jitsi_total_participants: 275
jitsi_conferences: 0
jitsi_largest_conference: 0
jitsi_endpoints_sending_video: 0
jitsi_endpoints_sending_audio: 0
jitsi_receive_only_endpoints: 0
jitsi_threads: 68
jitsi_total_ice_failed: 0
jitsi_total_failed_conferences: 0
jitsi_total_partially_failed_conferences: 0
Octo stats:
jitsi_octo_send_bitrate: 0
jitsi_octo_receive_bitrate: 0
jitsi_octo_send_packet_rate: 0
jitsi_octo_receive_packet_rate: 0
jitsi_octo_conferences: 0
jitsi_octo_endpoints: 0
jitsi_total_packets_sent_octo: 16114872
jitsi_total_packets_received_octo: 15841613
jitsi_total_packets_dropped_octo: 629506

Dropnute pakety v JVB: 0 0
Unable to find encoding matching packet: 372589
Negative rtt: 24802
Resource temporarily unavailable: 0
Couldn’t find packet detail for the seq nums: 1204
Suspiciously high rtt value (Client CPU problems): 108
Zatazenie: 21:21:01 up 37 days, 9:30, 3 users, load average: 0.00, 0.00, 0.00
Netstat: 153656 receive buffer errors 19 send buffer errors

and second JVB:

root@virt1:~/scripts# ./stats.sh
JVB stats:
jitsi_bit_rate_upload: 0
jitsi_bit_rate_download: 0
jitsi_participants: 0
jitsi_total_participants: 198
jitsi_conferences: 0
jitsi_largest_conference: 0
jitsi_endpoints_sending_video: 0
jitsi_endpoints_sending_audio: 0
jitsi_receive_only_endpoints: 0
jitsi_threads: 86
jitsi_total_ice_failed: 0
jitsi_total_failed_conferences: 0
jitsi_total_partially_failed_conferences: 0
Octo stats:
jitsi_octo_send_bitrate: 0
jitsi_octo_receive_bitrate: 0
jitsi_octo_send_packet_rate: 0
jitsi_octo_receive_packet_rate: 0
jitsi_octo_conferences: 0
jitsi_octo_endpoints: 0
jitsi_total_packets_sent_octo: 15839836
jitsi_total_packets_received_octo: 16111564
jitsi_total_packets_dropped_octo: 63301

Dropnute pakety v JVB: 0 0
Unable to find encoding matching packet: 921824
Negative rtt: 8305
Resource temporarily unavailable: 0
Couldn’t find packet detail for the seq nums: 497
Suspiciously high rtt value (Client CPU problems): 13
Zatazenie: 21:20:48 up 45 days, 23:26, 3 users, load average: 0.14, 0.03, 0.01
Netstat: 155661 receive buffer errors 41 send buffer errors

Thank you,

Milan

Bheema_Vatti · May 24, 2020, 3:18pm

Hi omesta,

Please help how you configured I configuration looks like you, but not working, how do you upgraded jicofo, I am unable to upgrade the package, and what are package need to install, I have followed quick installation. It showing

apt-get upgrade jicofo
Reading package lists… Done
Building dependency tree
Reading state information… Done
jicofo is already the newest version (1.0-567-1).
Calculating upgrade… Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

bheema

omesta · May 24, 2020, 3:32pm

I have updated using unstable build of jitsi-meet

Bheema_Vatti · May 24, 2020, 3:38pm

Thanks for the information,
Oh, octo not in stable code. What about configurations as you mentioned above same and meet/config.js file, did you added region information like region1 or aws region. I have only single region.

omesta · May 24, 2020, 3:40pm

Just region1 should be enough to see it working

Bheema_Vatti · May 26, 2020, 4:28pm

Thanks for the update, what about configuration for jvb, jicofo and meet files as same what you updated in this thread in above.

santanu · August 7, 2020, 7:18am

When I am enabling octo in my config.js file, my browser is not working. Not able to access the https://myserver.

Please suggest.

moto_droid · October 20, 2020, 9:04am

@msansnom jicofo-octo.txt (10.6 KB) @damencho i am not sure my octo is working fine or not? How can i check?
See the logs: