[jitsi-dev] jvb high cpu usage


#1

we updated our bridge on monday and have started seeing some crazy cpu
usage, even with no clients connected to a call. was able to do some
profiling, attached the screenshots.

what i did was spin up a bridge (tip of master) and have 2 people join call
1, 2 people join call 2. then had everyone leave (i don't think those are
necessarily the specific repro steps, but in this case were enough to cause
the problem). even after they left, cpu usage on the bridge was very high.
in the background of the screenshot is the /colibri/stats page to show we
don't have any streams active at the time.


#2

Hey Brian,

we updated our bridge on monday and have started seeing some crazy cpu
usage, even with no clients connected to a call. was able to do some
profiling, attached the screenshots.

what i did was spin up a bridge (tip of master) and have 2 people join
call 1, 2 people join call 2. then had everyone leave (i don't think
those are necessarily the specific repro steps, but in this case were
enough to cause the problem).

Scary stuff! I think I made this possible with this one[0], but can you look through your logs for an exception, so we can confirm?

This should provide a fix:
https://github.com/jitsi/libjitsi/pull/151

Thank you very much for the report!

Regards,
Boris

[0] https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

···

On 19/04/16 20:19, Brian Baldino wrote:


#3

Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting testing
last night, I *think* the issue may correspond to this commit:
https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
is that the one that updated libjitsi to the version which has the commit
you posted here?

···

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org> wrote:

Hey Brian,

On 19/04/16 20:19, Brian Baldino wrote:

we updated our bridge on monday and have started seeing some crazy cpu
usage, even with no clients connected to a call. was able to do some
profiling, attached the screenshots.

what i did was spin up a bridge (tip of master) and have 2 people join
call 1, 2 people join call 2. then had everyone leave (i don't think
those are necessarily the specific repro steps, but in this case were
enough to cause the problem).

Scary stuff! I think I made this possible with this one[0], but can you
look through your logs for an exception, so we can confirm?

This should provide a fix:
https://github.com/jitsi/libjitsi/pull/151

Thank you very much for the report!

Regards,
Boris

[0]
https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting
testing last night, I /think/ the issue may correspond to this commit:
https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
  is that the one that updated libjitsi to the version which has the
commit you posted here?

No, that's not the one.

Boris

···

On 20/04/16 10:51, Brian Baldino wrote:

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org > <mailto:boris@jitsi.org>> wrote:

    Hey Brian,

    On 19/04/16 20:19, Brian Baldino wrote:

        we updated our bridge on monday and have started seeing some
        crazy cpu
        usage, even with no clients connected to a call. was able to do
        some
        profiling, attached the screenshots.

        what i did was spin up a bridge (tip of master) and have 2
        people join
        call 1, 2 people join call 2. then had everyone leave (i don't
        think
        those are necessarily the specific repro steps, but in this case
        were
        enough to cause the problem).

    Scary stuff! I think I made this possible with this one[0], but can
    you look through your logs for an exception, so we can confirm?

    This should provide a fix:
    https://github.com/jitsi/libjitsi/pull/151

    Thank you very much for the report!

    Regards,
    Boris

    [0]
    https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

    _______________________________________________
    dev mailing list
    dev@jitsi.org <mailto:dev@jitsi.org>
    Unsubscribe instructions and other list options:
    http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#5

Was the commit you think it is rolled in before or after that?

···

On Wed, Apr 20, 2016 at 8:59 AM, Boris Grozev <boris@jitsi.org> wrote:

On 20/04/16 10:51, Brian Baldino wrote:

Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting
testing last night, I /think/ the issue may correspond to this commit:

https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
  is that the one that updated libjitsi to the version which has the
commit you posted here?

No, that's not the one.

Boris

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org >> <mailto:boris@jitsi.org>> wrote:

    Hey Brian,

    On 19/04/16 20:19, Brian Baldino wrote:

        we updated our bridge on monday and have started seeing some
        crazy cpu
        usage, even with no clients connected to a call. was able to do
        some
        profiling, attached the screenshots.

        what i did was spin up a bridge (tip of master) and have 2
        people join
        call 1, 2 people join call 2. then had everyone leave (i don't
        think
        those are necessarily the specific repro steps, but in this case
        were
        enough to cause the problem).

    Scary stuff! I think I made this possible with this one[0], but can
    you look through your logs for an exception, so we can confirm?

    This should provide a fix:
    https://github.com/jitsi/libjitsi/pull/151

    Thank you very much for the report!

    Regards,
    Boris

    [0]

https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

    _______________________________________________
    dev mailing list
    dev@jitsi.org <mailto:dev@jitsi.org>
    Unsubscribe instructions and other list options:
    http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#6

Also, is your fix here (https://github.com/jitsi/jitsi-videobridge/pull/226)
part of this as well?

···

On Wed, Apr 20, 2016 at 9:01 AM, Brian Baldino <brian@highfive.com> wrote:

Was the commit you think it is rolled in before or after that?

On Wed, Apr 20, 2016 at 8:59 AM, Boris Grozev <boris@jitsi.org> wrote:

On 20/04/16 10:51, Brian Baldino wrote:

Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting
testing last night, I /think/ the issue may correspond to this commit:

https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
  is that the one that updated libjitsi to the version which has the
commit you posted here?

No, that's not the one.

Boris

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org >>> <mailto:boris@jitsi.org>> wrote:

    Hey Brian,

    On 19/04/16 20:19, Brian Baldino wrote:

        we updated our bridge on monday and have started seeing some
        crazy cpu
        usage, even with no clients connected to a call. was able to do
        some
        profiling, attached the screenshots.

        what i did was spin up a bridge (tip of master) and have 2
        people join
        call 1, 2 people join call 2. then had everyone leave (i don't
        think
        those are necessarily the specific repro steps, but in this case
        were
        enough to cause the problem).

    Scary stuff! I think I made this possible with this one[0], but can
    you look through your logs for an exception, so we can confirm?

    This should provide a fix:
    https://github.com/jitsi/libjitsi/pull/151

    Thank you very much for the report!

    Regards,
    Boris

    [0]

https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

    _______________________________________________
    dev mailing list
    dev@jitsi.org <mailto:dev@jitsi.org>
    Unsubscribe instructions and other list options:
    http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#7

Before. I think it is introduced with libjitsi maven 145 or 144.

Boris

···

On Apr 20, 2016 11:01 AM, "Brian Baldino" <brian@highfive.com> wrote:

Was the commit you think it is rolled in before or after that?

On Wed, Apr 20, 2016 at 8:59 AM, Boris Grozev <boris@jitsi.org> wrote:

On 20/04/16 10:51, Brian Baldino wrote:

Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting
testing last night, I /think/ the issue may correspond to this commit:

https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
  is that the one that updated libjitsi to the version which has the
commit you posted here?

No, that's not the one.

Boris

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org >>> <mailto:boris@jitsi.org>> wrote:

    Hey Brian,

    On 19/04/16 20:19, Brian Baldino wrote:

        we updated our bridge on monday and have started seeing some
        crazy cpu
        usage, even with no clients connected to a call. was able to do
        some
        profiling, attached the screenshots.

        what i did was spin up a bridge (tip of master) and have 2
        people join
        call 1, 2 people join call 2. then had everyone leave (i don't
        think
        those are necessarily the specific repro steps, but in this case
        were
        enough to cause the problem).

    Scary stuff! I think I made this possible with this one[0], but can
    you look through your logs for an exception, so we can confirm?

    This should provide a fix:
    https://github.com/jitsi/libjitsi/pull/151

    Thank you very much for the report!

    Regards,
    Boris

    [0]

https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

    _______________________________________________
    dev mailing list
    dev@jitsi.org <mailto:dev@jitsi.org>
    Unsubscribe instructions and other list options:
    http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#8

No, this one is irrelevant. My fix is now merged in libjitsi, but
videobridge's pom.XML has to be updated before it makes it to a package.

Boris

···

On Apr 20, 2016 12:43, "Brian Baldino" <brian@highfive.com> wrote:

Also, is your fix here (
https://github.com/jitsi/jitsi-videobridge/pull/226) part of this as well?

On Wed, Apr 20, 2016 at 9:01 AM, Brian Baldino <brian@highfive.com> wrote:

Was the commit you think it is rolled in before or after that?

On Wed, Apr 20, 2016 at 8:59 AM, Boris Grozev <boris@jitsi.org> wrote:

On 20/04/16 10:51, Brian Baldino wrote:

Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting
testing last night, I /think/ the issue may correspond to this commit:

https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
  is that the one that updated libjitsi to the version which has the
commit you posted here?

No, that's not the one.

Boris

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org >>>> <mailto:boris@jitsi.org>> wrote:

    Hey Brian,

    On 19/04/16 20:19, Brian Baldino wrote:

        we updated our bridge on monday and have started seeing some
        crazy cpu
        usage, even with no clients connected to a call. was able to do
        some
        profiling, attached the screenshots.

        what i did was spin up a bridge (tip of master) and have 2
        people join
        call 1, 2 people join call 2. then had everyone leave (i don't
        think
        those are necessarily the specific repro steps, but in this case
        were
        enough to cause the problem).

    Scary stuff! I think I made this possible with this one[0], but can
    you look through your logs for an exception, so we can confirm?

    This should provide a fix:
    https://github.com/jitsi/libjitsi/pull/151

    Thank you very much for the report!

    Regards,
    Boris

    [0]

https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

    _______________________________________________
    dev mailing list
    dev@jitsi.org <mailto:dev@jitsi.org>
    Unsubscribe instructions and other list options:
    http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#9

ah great, ok that makes sense. turns out i copied the wrong revision and i
believe 1f7982aa22d66916856e6dfed1a68fb81490641b is the oldest i found with
the problem (which corresponds to a jitsi meet update to 145, so that lines
up with what you said there). thanks for the quick update...will probably
just verify once the videobridge pom gets pushed.

···

On Wed, Apr 20, 2016 at 2:17 PM, Boris Grozev <boris@sip-communicator.org> wrote:

No, this one is irrelevant. My fix is now merged in libjitsi, but
videobridge's pom.XML has to be updated before it makes it to a package.

Boris
On Apr 20, 2016 12:43, "Brian Baldino" <brian@highfive.com> wrote:

Also, is your fix here (
https://github.com/jitsi/jitsi-videobridge/pull/226) part of this as
well?

On Wed, Apr 20, 2016 at 9:01 AM, Brian Baldino <brian@highfive.com> >> wrote:

Was the commit you think it is rolled in before or after that?

On Wed, Apr 20, 2016 at 8:59 AM, Boris Grozev <boris@jitsi.org> wrote:

On 20/04/16 10:51, Brian Baldino wrote:

Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting
testing last night, I /think/ the issue may correspond to this commit:

https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
  is that the one that updated libjitsi to the version which has the
commit you posted here?

No, that's not the one.

Boris

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org >>>>> <mailto:boris@jitsi.org>> wrote:

    Hey Brian,

    On 19/04/16 20:19, Brian Baldino wrote:

        we updated our bridge on monday and have started seeing some
        crazy cpu
        usage, even with no clients connected to a call. was able to
do
        some
        profiling, attached the screenshots.

        what i did was spin up a bridge (tip of master) and have 2
        people join
        call 1, 2 people join call 2. then had everyone leave (i don't
        think
        those are necessarily the specific repro steps, but in this
case
        were
        enough to cause the problem).

    Scary stuff! I think I made this possible with this one[0], but can
    you look through your logs for an exception, so we can confirm?

    This should provide a fix:
    https://github.com/jitsi/libjitsi/pull/151

    Thank you very much for the report!

    Regards,
    Boris

    [0]

https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

    _______________________________________________
    dev mailing list
    dev@jitsi.org <mailto:dev@jitsi.org>
    Unsubscribe instructions and other list options:
    http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#10

jvb@725 will be the golden build. It’s baking.

···

On Apr 20, 2016, at 5:10 PM, Brian Baldino <brian@highfive.com> wrote:

ah great, ok that makes sense. turns out i copied the wrong revision and i believe 1f7982aa22d66916856e6dfed1a68fb81490641b is the oldest i found with the problem (which corresponds to a jitsi meet update to 145, so that lines up with what you said there). thanks for the quick update...will probably just verify once the videobridge pom gets pushed.

On Wed, Apr 20, 2016 at 2:17 PM, Boris Grozev <boris@sip-communicator.org <mailto:boris@sip-communicator.org>> wrote:
No, this one is irrelevant. My fix is now merged in libjitsi, but videobridge's pom.XML has to be updated before it makes it to a package.

Boris

On Apr 20, 2016 12:43, "Brian Baldino" <brian@highfive.com <mailto:brian@highfive.com>> wrote:
Also, is your fix here (https://github.com/jitsi/jitsi-videobridge/pull/226) part of this as well?

On Wed, Apr 20, 2016 at 9:01 AM, Brian Baldino <brian@highfive.com <mailto:brian@highfive.com>> wrote:
Was the commit you think it is rolled in before or after that?

On Wed, Apr 20, 2016 at 8:59 AM, Boris Grozev <boris@jitsi.org <mailto:boris@jitsi.org>> wrote:
On 20/04/16 10:51, Brian Baldino wrote:
Hey Boris,
Thanks, will definitely give that patch a try. From my bisecting
testing last night, I /think/ the issue may correspond to this commit:
https://github.com/jitsi/jitsi-videobridge/commit/2edd775fd02fef31eadb8460f64c49bd47da48e9
  is that the one that updated libjitsi to the version which has the
commit you posted here?

No, that's not the one.

Boris

On Wed, Apr 20, 2016 at 1:08 AM, Boris Grozev <boris@jitsi.org <mailto:boris@jitsi.org> > <mailto:boris@jitsi.org <mailto:boris@jitsi.org>>> wrote:

    Hey Brian,

    On 19/04/16 20:19, Brian Baldino wrote:

        we updated our bridge on monday and have started seeing some
        crazy cpu
        usage, even with no clients connected to a call. was able to do
        some
        profiling, attached the screenshots.

        what i did was spin up a bridge (tip of master) and have 2
        people join
        call 1, 2 people join call 2. then had everyone leave (i don't
        think
        those are necessarily the specific repro steps, but in this case
        were
        enough to cause the problem).

    Scary stuff! I think I made this possible with this one[0], but can
    you look through your logs for an exception, so we can confirm?

    This should provide a fix:
    https://github.com/jitsi/libjitsi/pull/151

    Thank you very much for the report!

    Regards,
    Boris

    [0]
    https://github.com/jitsi/libjitsi/commit/eb77d07b05502eebb1df75bd075ff4171296abe3

    _______________________________________________
    dev mailing list
    dev@jitsi.org <mailto:dev@jitsi.org> <mailto:dev@jitsi.org <mailto:dev@jitsi.org>>
    Unsubscribe instructions and other list options:
    http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org <mailto:dev@jitsi.org>
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev