[jitsi-dev] [jitsi-videobridge] Video Freezing in Chrome (#156)


#1

Using new versions of the bridge, have been experiencing random freezing of the video channel. Doesn't occur for every participant.
I've included a webrtc-internals graph showing it happening just before 7:55pm. Nothing abnormal in bridge logs or our clients log (not using jitsi meet). We ended up rolling back to version 564 where this does not happen.
![talky-jvb-video-freeze](https://cloud.githubusercontent.com/assets/1449748/13441660/683e8f46-dfc6-11e5-823e-4d6514f96681.png)

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156


#2

We aren't using simulcast, it's true.

I'm glad you actually bring up RTCP termination. The documentation is outdated, and we aren't sure how to make it work.

We'd like to enable what was `org.jitsi.impl.neomedia.rtcp.termination.strategies.HighestQualityRTCPTerminationStrategy` but setting it according to the doc fails.

Maybe setting it correctly might fix the issue?

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-190908389


#3

We tried with BasicRTCPTermination, but it was throwing an index out of bounds exception.
One change I did make which seems to help was removing `org.jitsi.impl.neomedia.transform.srtp.SRTPCryptoContext.checkReplay=false`
Testing in our staging environment the issue seems to have gone away, but stage isn't always reliable for reproducing the bug. I'll plan a deploy tomorrow to production turning off rtcp termination and removing the checkReplay setting and update this issue again.
Thanks for the help!

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-190926506


#4

Just before 7:55 nacks and plis increase and received frame rate drops to
0, but I notice some peculiar behavior starting at 7:54. You don't seem to
be using simulcast. Are you using RTCP termination? Could you please enable
fine logging at the bridge and logging at the client and share the log
files with us?

···

On Tue, Mar 1, 2016 at 2:59 PM, Marcus Stong <notifications@github.com> wrote:

Using new versions of the bridge, have been experiencing random freezing
of the video channel. Doesn't occur for every participant.
I've included a webrtc-internals graph showing it happening just before
7:55pm. Nothing abnormal in bridge logs or our clients log (not using jitsi
meet). We ended up rolling back to version 564 where this does not happen.
[image: talky-jvb-video-freeze]
<https://cloud.githubusercontent.com/assets/1449748/13441660/683e8f46-dfc6-11e5-823e-4d6514f96681.png>


Reply to this email directly or view it on GitHub
<https://github.com/jitsi/jitsi-videobridge/issues/156>.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#5

Video is still freezing. It's pretty easy to reproduce right now on https://talky.io as I haven't rolled back yet.
I also have a webrtc-internals dump if that would help

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191841620


#6

George, thanks for the help! I think the graph above should suffice then.
Here's the sip-communicator.properties:

org.jitsi.videobridge.TCP_HARVESTER_MAPPED_PORT=443
org.jitsi.videobridge.TCP_HARVESTER_PORT=4443
org.jitsi.videobridge.STATISTICS_TRANSPORT=pubsub
org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_INTERVAL=15000
org.jitsi.videobridge.PUBSUB_SERVICE=pubsub.foo.bar
org.jitsi.videobridge.PUBSUB_NODE=videobridge
org.jitsi.videobridge.SINGLE_PORT_HARVESTER_PORT=-1
org.ice4j.ice.harvest.ALLOWED_INTERFACES=bond0

Here's a sampling of the logs https://ghostbin.com/paste/adfg9

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191854525


#7

george: http://fippo.github.io/webrtc-dump-importer/ gives you nice graphs in a matter of seconds. Zoomable even.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191856920


#8

Hey @fippo is there a way to make those dumps from js code, I'm asking whether it is possible to do the dumps while selenium testing? Thanks.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191858993


#9

@damencho do you know your own code? :-p
traceablepeerconnection was built exactly for this. I suppose you can also open webrtc-internals in an extra tab in selenium but never tried it.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191859870


#10

Here's a sampling of the logs https://ghostbin.com/paste/adfg9

2016-03-03 11:46:34.065 WARNING: [90782] org.jitsi.videobridge.transform.RtxTransformer.warn() Cannot find SSRC for RTX, retransmitting plain.

This could well indicate a problem with packet retransmissions, which could explain the freeze. We don't run into it, because we haven't yet enabled RTX.

One reason for the bridge not finding the SSRC could be that it wasn't signaled to it. Can you include more of the logs? Specifically the RECV/SENT lines. Also make sure you are using a recent bridge version (which includes Lance's fix).

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191860068


#11

Just found a little bug, preparing a fix. You may want to delay your testing a bit.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191862341


#12

okay great. I'll test on our stage site as soon as you let me know.
we had been using one of the latest versions with Lance's fix too, just so you know.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191866586


#13

Videobridge 672 includes the fix.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-191890615


#14

Deployed 672 with and without suggested NACK settings, and unfortunately it doesn't work at all now

org.jitsi.impl.osgi.framework.launch.FrameworkImpl.startLevelChanged() Error changing start level
org.osgi.framework.BundleException: BundleActivator.start
	at org.jitsi.impl.osgi.framework.BundleImpl.start(BundleImpl.java:313)
	at org.jitsi.impl.osgi.framework.launch.FrameworkImpl.startLevelChanged(FrameworkImpl.java:460)
	at org.jitsi.impl.osgi.framework.startlevel.FrameworkStartLevelImpl$Command.run(FrameworkStartLevelImpl.java:126)
	at org.jitsi.impl.osgi.framework.AsyncExecutor.runInThread(AsyncExecutor.java:111)
	at org.jitsi.impl.osgi.framework.AsyncExecutor.access$000(AsyncExecutor.java:17)
	at org.jitsi.impl.osgi.framework.AsyncExecutor$1.run(AsyncExecutor.java:220)
Caused by: java.lang.NoClassDefFoundError: net/java/sip/communicator/impl/protocol/jabber/extensions/colibri/HealthCheckIQ
	at org.jitsi.videobridge.VideobridgeBundleActivator.start(VideobridgeBundleActivator.java:59)
	at org.jitsi.impl.osgi.framework.BundleImpl.start(BundleImpl.java:293)
	... 5 more
Caused by: java.lang.ClassNotFoundException: net.java.sip.communicator.impl.protocol.jabber.extensions.colibri.HealthCheckIQ
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more
2016-03-03 23:00:00.766 SEVERE: [17] org.jitsi.videobridge.stats.PubSubStatsTransport.publishStatistics().282 Failed to publish to PubSub node: videobridge - it does not exist yet

Going to rollback one version and test with NACK settings as well.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-192089503


#15

670 with suggested NACK settings passes staging tests.
Will deploy to production tomorrow morning and report back

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-192090919


#16

Not sure what the problem with 672 is, possibly just the package was not properly built. In any case, if you get a chance to test this on 672+ without disabling NACK termination, please let us know.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-192107860


#17

Still freezing in production on 670 with NACK changes. Will give a release > 672 a try

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-192401252


#18

Still freezing in 681

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-195402170


#19

Disabling RTX seems like a promising fix. Wasn't able to reproduce freezing on stage. Will confirm for sure with production deploy Monday.

···

---
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-195568290


#20

Been running 681 for most of the week with RTX disabled in production.
Our feedback form didn't have one report of freezing and our friday update conference also was good.
Seems to be fixed!

···

---
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/jitsi/jitsi-videobridge/issues/156#issuecomment-198446548