Continuous googPlisSent being sent with 1080p camera and 1% packet loss


#1

Hi,

I have installed a local jitsi-meet using the debian packages. I then have used “tc” to emulate packet loss between the jitsi server (where jvb is installed) and a laptop. So, the setup is like:

  • Server with jitsi.
  • Same server is used as a client with a 1080p camera.
  • Laptop connecting as a client.
  • Simulcast is enabled.
  • 1080 is the ideal and max constraint.

Network connection between server and the laptop client has 1% packet loss using the following commands:

tc qdisc add dev eno1 handle 1: root htb default 99
tc class add dev eno1 parent 1: classid 1:1 htb rate 1000mbit
tc class add dev eno1 parent 1:1 classid 1:11 htb rate 15000kbit
tc qdisc add dev eno1 parent 1:11 handle 51: netem loss 1.0% delay 0ms 0ms
tc filter add dev eno1 protocol ip prio 1 u32 match ip src 10.10.131.127 match ip dst 10.10.130.244 flowid 1:11

You can see the webrtc-internals here:

Could there be anything misconfigured? What could be happening here?

Thank you in advance,

Aleix


#2

Hi, it looks like your client is sending a PLI per 30 lost packets. I would expect retransmissions to help a bit more but we don’t have specific numbers for such scenarios. Could you please post your sip-communicator.properties file and also which Chrome version are you using? Thanks.


#3

I would also be curious to know what happens if you lower the resolution to 720p, if it’s easy to re-run the experiment.


#4

Thanks for the quick reply!

The Chrome version is 67.0.3396.99, but I have also tried 68.0.3440.84 with same results.

With 720p it behaves much better. You can see the results in the same gist. There’s also the sip-communicator.properties file.

One thing worth mentioning is that today I updated to the latest jitsi stable packages (see below), and suddenly the bandwidth for a 720p stream decreased from ~2Mbps to ~1Mbps, but not for all clients. Why could that be?

$ dpkg -l | grep jitsi
ii jitsi-meet 1.0.3229-1 all WebRTC JavaScript video conferences
ii jitsi-meet-prosody 1.0.2942-1 all Prosody configuration for Jitsi Meet
ii jitsi-meet-web 1.0.2942-1 all WebRTC JavaScript video conferences
ii jitsi-meet-web-config 1.0.2942-1 all Configuration for web serving of Jitsi Meet
ii jitsi-videobridge 1077-1 amd64 WebRTC compatible Selective Forwarding Unit (SFU)

Thank you!

Aleix


#5

What could be happening here is that we’re not replying to NACKs (because the JVB packet cache is too shallow for a 1080p stream) which messes up the client jitter buffer causing it to ask for a keyframe.

Could you please try increasing the cache depth by adding these properties in your sip-communicator.properties file:

org.jitsi.impl.neomedia.transform.CachingTransformer.CACHE_SIZE_PACKETS=2000
org.jitsi.impl.neomedia.transform.CachingTransformer.CACHE_SIZE_MILLIS=5000

Please note that I’m not suggesting these as some sort of optimal values, the defaults should be the optimal for most cases.

If, however, we find that increasing the cache depth has a positive impact, then we may have to re-evaluate and adjust the defaults (currently the defaults are an order of magnitude smaller than what I’m suggesting here).


#6

I added those two lines in the JVB sip-communicator.properties file and restarted the video bridge service. The values you provided didn’t quite work so I increased the number of packets cache and that fixed the issue.

See gist for the new webrtc-internals image.


#7

So I have this:

org.jitsi.impl.neomedia.transform.CachingTransformer.CACHE_SIZE_PACKETS=10000
org.jitsi.impl.neomedia.transform.CachingTransformer.CACHE_SIZE_MILLIS=5000

I didn’t really try to adjust the values more. But the explanation you gave makes total sense.


#8

Thumbs up for trying even higher values than the ones that I suggested! It looks like the defaults are way too low and need some adjustment; We’ll look into it. Thanks very much for reporting.


#9

I have also tried with 2% and 3% packet loss, see gist again. It seems the 10000 packets cache can handle up to that.

Thank you for the suggestion!


#10

Hi,

Interesting findings. These cache sizes (5000, 10000 packets) seem excessively large to me, and this will affect performance. As George just told me off-line, a 1080p stream should peek at somewhere around 500pps, so a 1000 packet cache should be plenty.

I don’t know what is going on, but I would double check that there are misses with a small cache, and that they disappear when the size is increased. Jitsi-videobridge logs the number of cache hits and misses (resulting from incoming NACKs) with a message like this when the stream is closed:
org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=255787077 max_size_bytes=15762,max_size_packets=29,total_hits=0,total_misses=0,total_packets=41,oldest_hit_ms=0

Note the “stream=XXX” part: a positive number indicates this is the outgoing packet cache, while -1 indicates it’s the incoming packet cache (used internally for simulcast).

In a scenario where packet loss occurs only between the bridge and the receiver (no loss between the sender and the bridge), the misses for the outgoing cache should be virtually 0. I think the misses for the incoming packet cache should always be close to 0, but George might correct me.

Regards,
Boris


#11

Hi,

yes, a while back I wrote an RTSP server (based on GStreamer) that mostly handled 1080p streams and I remember something around those numbers. However, the encoding was H.264 (using x264) but I don’t think it should be a lot of difference. Certainly a cache of 10000 seems high.

I just double checked the number of packets that a 1080p would use in one second with wireshark. I checked the number of packets for the video ssrc using RTP sequence numbers and I got ~600 packets, so that seems accurate. There are ~100 that belong to the audio stream.

This is the RawPacketCache.log() with default values (I also added the audio stream just for reference):

JVB 2018-08-03 10:38:11.190 INFO: [304] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=303028213 max_size_bytes=239590,max_size_packets=201,total_hits=648,total_misses=2506,total_packets=78787,oldest_hit_ms=47
JVB 2018-08-03 10:38:11.190 INFO: [304] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=-1 max_size_bytes=345791,max_size_packets=303,total_hits=0,total_misses=0,total_packets=54537,oldest_hit_ms=0

This is the RawPacketCache.log() with 2000 packet cache size (and 5000ms time cache):

JVB 2018-08-03 10:48:21.296 INFO: [52] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=1825006019 max_size_bytes=2340613,max_size_packets=2001,total_hits=1466,total_misses=4321,total_packets=168690,oldest_hit_ms=48
JVB 2018-08-03 10:48:21.297 INFO: [52] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=-1 max_size_bytes=2721453,max_size_packets=2518,total_hits=0,total_misses=0,total_packets=117981,oldest_hit_ms=0

This is the RawPacketCache.log() with 3000 packet cache size (and 5000ms time cache):

JVB 2018-08-03 11:18:49.810 INFO: [376] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=373176766 max_size_bytes=3507342,max_size_packets=3001,total_hits=2288,total_misses=394,total_packets=202711,oldest_hit_ms=106
JVB 2018-08-03 11:18:49.810 INFO: [376] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=-1 max_size_bytes=3736512,max_size_packets=3403,total_hits=0,total_misses=0,total_packets=141578,oldest_hit_ms=0

This is the RawPacketCache.log() with 5000 packet cache size (and 5000ms time cache):

JVB 2018-08-03 11:05:31.055 INFO: [380] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=52720596 max_size_bytes=5876251,max_size_packets=5001,total_hits=2260,total_misses=66,total_packets=202989,oldest_hit_ms=119492
JVB 2018-08-03 11:05:31.056 INFO: [380] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=-1 max_size_bytes=3928420,max_size_packets=3565,total_hits=0,total_misses=0,total_packets=142162,oldest_hit_ms=0

This is the RawPacketCache.log() with 10000 packet cache size (and 5000ms time cache):

JVB 2018-08-03 11:26:10.779 INFO: [303] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=1150686393 max_size_bytes=6591981,max_size_packets=5647,total_hits=2201,total_misses=86,total_packets=194453,oldest_hit_ms=119897
JVB 2018-08-03 11:26:10.780 INFO: [303] org.jitsi.impl.neomedia.rtp.RawPacketCache.log() CAT=stat closed,stream=-1 max_size_bytes=3818570,max_size_packets=3481,total_hits=0,total_misses=0,total_packets=136201,oldest_hit_ms=0

I have also update the gist with new images and renamed the file names so it’s clear what is what.

Hope this helps.


#12

I ran an experiment in production with one bridge using 1000 packets / 1s (the current default), and another using 10000 packets / 10s. I see much higher hit rate with the bigger cache (10% vs 43%), but I don’t know how much of it is due to noise. We don’t have the time to get to the bottom of this right now (I still think that 1000 should be enough), so we’re also going to work around it by increasing the size.

Boris