[jitsi-dev] jvb.log: Too many open files


#1

Hi all!

In a Jitsi Meet installation with the stable version installed from the
download.jitsi.org repository, I am observing this message. It seems to
be a possible known bug that has been reported some time ago in Github.

I would like to know if this may be the reason for that when I try to
start a conference I get continually "Unfortunately, something went
wrong. We're trying to fix this".

If that is the cause, I would like to know if you have a fix for this
that we can to have in the stable repository.

Thanks in advance.

Kind regards,
Daniel


#2

Hi Daniel,

Hi all!

In a Jitsi Meet installation with the stable version installed from the
download.jitsi.org repository, I am observing this message. It seems to
be a possible known bug that has been reported some time ago in Github.

I would like to know if this may be the reason for that when I try to
start a conference I get continually "Unfortunately, something went
wrong. We're trying to fix this".

Yes this could be the reason. The problems mentioned on the mailing list are all fixed, as far as I know. However, recently we observed something similar internally, and it hasn't been fixed yet. It is low priority for the moment, because we believe it only happens when the system is not used for a prolonged period of time.

I suggest that you up the ulimit for open files as Hugh suggested, and make health-checks from jicofo less frequent by increasing the value for the following property in /etc/jitsi/jicofo/sip-communicator.properties:
org.jitsi.jicofo.HEALTH_CHECK_INTERVAL

Regards,
Boris

···

On 31/01/2017 07:45, Daniel Bareiro wrote:


#3

Hi Daniel

I'm not sure what OS you are using. I'm testing with Debian on Amazon EC2.

That said, I've just increased the limits for open files, there's a good
thread here:
http://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux
but the exact answer will depend on how that bit of /etc is organised

I have been doing a little testing that suggests that the open files count
creeps up, but have had to stop, because another project has taken priority
now. May get back to it 'soon'.

Best regards Hugh

···

On 31 January 2017 at 13:45, Daniel Bareiro <daniel-listas@gmx.net> wrote:

Hi all!

In a Jitsi Meet installation with the stable version installed from the
download.jitsi.org repository, I am observing this message. It seems to
be a possible known bug that has been reported some time ago in Github.

I would like to know if this may be the reason for that when I try to
start a conference I get continually "Unfortunately, something went
wrong. We're trying to fix this".

If that is the cause, I would like to know if you have a fix for this
that we can to have in the stable repository.

Thanks in advance.

Kind regards,
Daniel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
http://www.hughbarnard.org
http://www.twitter.com/hughbarnard
http://www.big-wave-heuristics.com/
<http://www.hackney-environment-network.org.uk/>


#4

Hi Daniel (Boris)

Thanks, the cleanest way will be an extra file in
/etc/security/limits.d/<something>.conf, for example

$ sudo cat /etc/security/limits.d/90-nofiles.conf
loginuser soft nofile 10240
loginuser hard nofile 10240
root soft nofile 10241
root hard nofile 10241
serviceuser soft nofile 10242
serviceuser hard nofile 10242

The nofile entries provide per user open file limits. I'm away from my own
stuff at the moment, but I think I moved it to ~10K files.

Best regards Hugh

···

On 31 January 2017 at 15:05, Boris Grozev <boris@jitsi.org> wrote:

Hi Daniel,

On 31/01/2017 07:45, Daniel Bareiro wrote:

Hi all!

In a Jitsi Meet installation with the stable version installed from the
download.jitsi.org repository, I am observing this message. It seems to
be a possible known bug that has been reported some time ago in Github.

I would like to know if this may be the reason for that when I try to
start a conference I get continually "Unfortunately, something went
wrong. We're trying to fix this".

Yes this could be the reason. The problems mentioned on the mailing list
are all fixed, as far as I know. However, recently we observed something
similar internally, and it hasn't been fixed yet. It is low priority for
the moment, because we believe it only happens when the system is not used
for a prolonged period of time.

I suggest that you up the ulimit for open files as Hugh suggested, and
make health-checks from jicofo less frequent by increasing the value for
the following property in /etc/jitsi/jicofo/sip-communicator.properties:
org.jitsi.jicofo.HEALTH_CHECK_INTERVAL

Regards,
Boris

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
http://www.hughbarnard.org
http://www.twitter.com/hughbarnard
http://www.big-wave-heuristics.com/
<http://www.hackney-environment-network.org.uk/>


#5

Hi Daniel

Hi, Hugh.

I'm not sure what OS you are using. I'm testing with Debian on Amazon EC2.

I am sorry. I forgot to mention it. I'm using Debian Jessie 8.6 on OVH,
so the kernel is provided by them:

# uname -r
3.14.32-xxxx-grs-ipv6-64

That said, I've just increased the limits for open files, there's a good
thread here:
http://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux
but the exact answer will depend on how that bit of /etc is organised

This is the current value:

# ulimit -n
65536

Jicofo PID:

# ps -u jicofo | tail -1 | awk '{ print $1 }'
787

JVB PID:

# ps -u jvb | tail -1 | awk '{ print $1 }'
797

Resource limits for Jicofo:

···

On 31/01/17 11:07, Hugh Barnard wrote:

--------------------------------------------------------------------------
# cat /proc/787/limits
Limit Soft Limit Hard Limit
Units
Max cpu time unlimited unlimited
seconds
Max file size unlimited unlimited
bytes
Max data size unlimited unlimited
bytes
Max stack size 8388608 unlimited
bytes
Max core file size 0 unlimited
bytes
Max resident set unlimited unlimited
bytes
Max processes 63760 63760
processes
Max open files 4096 4096
files
Max locked memory 65536 65536
bytes
Max address space unlimited unlimited
bytes
Max file locks unlimited unlimited
locks
Max pending signals 63760 63760
signals
Max msgqueue size 819200 819200
bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
--------------------------------------------------------------------------

Resource limits for JVB:

--------------------------------------------------------------------------
root@ns381001:/var/log/jitsi# cat /proc/797/limits
Limit Soft Limit Hard Limit
Units
Max cpu time unlimited unlimited
seconds
Max file size unlimited unlimited
bytes
Max data size unlimited unlimited
bytes
Max stack size 8388608 unlimited
bytes
Max core file size 0 unlimited
bytes
Max resident set unlimited unlimited
bytes
Max processes 63760 63760
processes
Max open files 4096 4096
files
Max locked memory 65536 65536
bytes
Max address space unlimited unlimited
bytes
Max file locks unlimited unlimited
locks
Max pending signals 63760 63760
signals
Max msgqueue size 819200 819200
bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
--------------------------------------------------------------------------

This seems to show that both processes support a maximum of 4096 open
files. Or I'm wrong?

I have been doing a little testing that suggests that the open files
count creeps up, but have had to stop, because another project has taken
priority now. May get back to it 'soon'.

Do you think the message of "Unfortunately, something went wrong. We're
trying to fix this" is due to this" Too many open files"?

Thanks for your reply.

Kind regards,
Daniel


#6

Hi Daniel (Boris)

Hi, Hugh/Boris.

Thanks, the cleanest way will be an extra file in
/etc/security/limits.d/<something>.conf, for example

$ sudo cat /etc/security/limits.d/90-nofiles.conf
loginuser soft nofile 10240
loginuser hard nofile 10240
root soft nofile 10241
root hard nofile 10241
serviceuser soft nofile 10242
serviceuser hard nofile 10242

The nofile entries provide per user open file limits. I'm away from my
own stuff at the moment, but I think I moved it to ~10K files.

My /etc/security/limits.d directory is currently empty. So you suggest
adding a file that has something like:

jvb soft nofile 10240
jvb hard nofile 10240

With "jvb" it would be enough or should I also add the user "I jicofo"?

I understand that after making this change, I should restart
jitsi-videobridge (and eventually jicofo) to check that the changes have
been applied with /proc/<pid>/limits.

Now back to what said by Boris ("It is a low priority for the moment,
only it Because We Believe Happens When the system is not used for a
period of time Prolonged"), I am seeing this from day to day on the OVH
server. However on my local server with a standard Debian kernel is not
very common.

Debian standard server:

···

On 31/01/17 13:17, Hugh Barnard wrote:

------------------------------------------------------------------------
root@conference:/var/log/jitsi# ll jvb*
-rw-r--r-- 1 jvb jitsi 148211 Feb 2 08:24 jvb.log
-rw-r--r-- 1 jvb jitsi 5487124 Feb 2 06:26 jvb.log.1
-rw-r--r-- 1 jvb jitsi 319212 Feb 1 06:26 jvb.log.2.gz
-rw-r--r-- 1 jvb jitsi 188733 Jan 31 06:25 jvb.log.3.gz
-rw-r--r-- 1 jvb jitsi 81011 Jan 30 06:25 jvb.log.4.gz
-rw-r--r-- 1 jvb jitsi 98931 Jan 29 06:25 jvb.log.5.gz
-rw-r--r-- 1 jvb jitsi 474901 Jan 28 06:25 jvb.log.6.gz
-rw-r--r-- 1 jvb jitsi 365577 Jan 27 06:25 jvb.log.7.gz
------------------------------------------------------------------------
root@conference:/var/log/jitsi# zgrep -i "open files" jvb.log
root@conference:/var/log/jitsi# zgrep -i "open files" jvb.log.1
root@conference:/var/log/jitsi# zgrep -i "open files" jvb.log.2.gz
root@conference:/var/log/jitsi# zgrep -i "open files" jvb.log.3.gz
root@conference:/var/log/jitsi# zgrep -i "open files" jvb.log.4.gz
root@conference:/var/log/jitsi# zgrep -i "open files" jvb.log.5.gz
java.net.SocketException: Too many open files (Error creating socket)
root@conference:/var/log/jitsi# zgrep -i "open files" jvb.log.6.gz
------------------------------------------------------------------------

Result: a single occurrence in seven days.

Debian from OVH:

------------------------------------------------------------------------
# ll jvb*
-rw-r--r-- 1 jvb jitsi 2415831 févr. 2 12:28 jvb.log
-rw-r--r-- 1 jvb jitsi 5037465 févr. 2 06:25 jvb.log.1
-rw-r--r-- 1 jvb jitsi 412436 févr. 1 06:25 jvb.log.2.gz
-rw-r--r-- 1 jvb jitsi 295522 janv. 31 06:25 jvb.log.3.gz
-rw-r--r-- 1 jvb jitsi 277652 janv. 30 06:25 jvb.log.4.gz
-rw-r--r-- 1 jvb jitsi 80288 janv. 29 06:25 jvb.log.5.gz
-rw-r--r-- 1 jvb jitsi 318759 janv. 28 06:25 jvb.log.6.gz
-rw-r--r-- 1 jvb jitsi 286924 janv. 27 06:25 jvb.log.7.gz
------------------------------------------------------------------------
# zgrep fichiers jvb.log
#

# zgrep fichiers jvb.log.1
JVB 2017-02-01 21:37:05.913 INFOS: [358]
org.jitsi.videobridge.xmpp.ComponentImpl.log() SENT: <iq
id="eP4D8-22365" to="focus@auth.meet.softysoft.net/focus32024923635"
from="jitsi-videobridge.meet.softysoft.net" type="error"><healthcheck
xmlns="http://jitsi.org/protocol/healthcheck"></healthcheck><error
code="500" type="WAIT"><internal-server-error
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"></internal-server-error><text
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"
xml:lang="en">java.net.SocketException: Trop de fichiers ouverts (Error
creating socket)</text></error></iq>
root@ns381001:/var/log/jitsi# zgrep fichiers jvb.log.2.gz

# zgrep fichiers jvb.log.2.gz
JVB 2017-02-01 03:09:16.337 INFOS: [26]
org.jitsi.videobridge.xmpp.ComponentImpl.log() SENT: <iq
id="YbKBF-20674" to="focus@auth.meet.softysoft.net/focus31644445441"
from="jitsi-videobridge.meet.softysoft.net" type="error"><healthcheck
xmlns="http://jitsi.org/protocol/healthcheck"></healthcheck><error
code="500" type="WAIT"><internal-server-error
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"></internal-server-error><text
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"
xml:lang="en">java.net.SocketException: Trop de fichiers ouverts (Error
creating socket)</text></error></iq>

# zgrep fichiers jvb.log.3.gz
JVB 2017-01-30 23:36:45.859 INFOS: [159]
org.jitsi.videobridge.xmpp.ComponentImpl.log() SENT: <iq
id="5TR41-25253" to="focus@auth.meet.softysoft.net/focus31727224863"
from="jitsi-videobridge.meet.softysoft.net" type="error"><healthcheck
xmlns="http://jitsi.org/protocol/healthcheck"></healthcheck><error
code="500" type="WAIT"><internal-server-error
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"></internal-server-error><text
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"
xml:lang="en">java.net.SocketException: Trop de fichiers ouverts (Error
creating socket)</text></error></iq>

# zgrep fichiers jvb.log.4.gz
java.net.SocketException: Trop de fichiers ouverts (Error creating socket)

# zgrep fichiers jvb.log.5.gz
#

# zgrep fichiers jvb.log.6.gz
java.net.SocketException: Trop de fichiers ouverts (Error creating socket)
JVB 2017-01-27 21:48:39.866 INFOS: [33]
org.jitsi.videobridge.xmpp.ComponentImpl.log() SENT: <iq
id="183Nj-22187" to="focus@auth.meet.softysoft.net/focus32361954286"
from="jitsi-videobridge.meet.softysoft.net" type="error"><healthcheck
xmlns="http://jitsi.org/protocol/healthcheck"></healthcheck><error
code="500" type="WAIT"><internal-server-error
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"></internal-server-error><text
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" xml:lang="en">Trop de
fichiers ouverts (Socket creation failed)</text></error></iq>

# zgrep fichiers jvb.log.7.gz
#
------------------------------------------------------------------------

Results: five occurrences in seven days.

Thanks for your reply and your time.

Kind regards,
Daniel


#7

Hi again, Hugh/Boris.

···

On 02/02/17 08:36, Daniel Bareiro wrote:

My /etc/security/limits.d directory is currently empty. So you suggest
adding a file that has something like:

jvb soft nofile 10240
jvb hard nofile 10240

With "jvb" it would be enough or should I also add the user "I jicofo"?

I tried doing this configuration in /etc/security/limits.d but it did
not apply after a reboot. I think this is because these settings are for
users who have an active shell on the system. Can it be?

Kind regards,
Daniel


#8

Here is verified:

···

On 07/02/17 17:43, Daniel Bareiro wrote:

My /etc/security/limits.d directory is currently empty. So you suggest
adding a file that has something like:

jvb soft nofile 10240
jvb hard nofile 10240

With "jvb" it would be enough or should I also add the user "I jicofo"?

I tried doing this configuration in /etc/security/limits.d but it did
not apply after a reboot. I think this is because these settings are for
users who have an active shell on the system. Can it be?

----------------------------------------------------------------------------
root@conference:~# cat /etc/security/limits.d/jitsi-meet.conf
jvb soft nofile 10240
jvb hard nofile 10240
----------------------------------------------------------------------------
root@conference:~# ps -u jvb | tail -1 | awk '{ print $1 }'
434
----------------------------------------------------------------------------
root@conference:~# cat /proc/434/limits | grep open
Max open files 4096 4096 files
----------------------------------------------------------------------------
root@conference:~# su - jvb
jvb@conference:~$ ulimit -n
10240
----------------------------------------------------------------------------

Kind regards,
Daniel


#9

We just pushed some fixes which should alleviate this problem (available in jitsi-videobridge 928).

Regards,
Boris

···

On 07/02/2017 14:58, Daniel Bareiro wrote:

On 07/02/17 17:43, Daniel Bareiro wrote:

My /etc/security/limits.d directory is currently empty. So you suggest
adding a file that has something like:

jvb soft nofile 10240
jvb hard nofile 10240

With "jvb" it would be enough or should I also add the user "I jicofo"?

I tried doing this configuration in /etc/security/limits.d but it did
not apply after a reboot. I think this is because these settings are for
users who have an active shell on the system. Can it be?

Here is verified:

----------------------------------------------------------------------------
root@conference:~# cat /etc/security/limits.d/jitsi-meet.conf
jvb soft nofile 10240
jvb hard nofile 10240
----------------------------------------------------------------------------
root@conference:~# ps -u jvb | tail -1 | awk '{ print $1 }'
434
----------------------------------------------------------------------------
root@conference:~# cat /proc/434/limits | grep open
Max open files 4096 4096 files
----------------------------------------------------------------------------
root@conference:~# su - jvb
jvb@conference:~$ ulimit -n
10240
----------------------------------------------------------------------------

Kind regards,
Daniel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#10

Hi, Boris.

We just pushed some fixes which should alleviate this problem (available
in jitsi-videobridge 928).

Thanks for your reply.

Yes, a while ago Damian Minkov told me that you had introduced a fix on
the 882 version of the Bridge [1]. It seems that there were several more
changes if the current version of the bridge is 928.

Unfortunately I am using stable and here is not present this fix. Do you
think this fix may be available in stable in a short term?

Kind regards,
Daniel

[1] http://lists.jitsi.org/pipermail/dev/2017-March/032088.html

···

On 11/04/17 00:58, Boris Grozev wrote:


#11

Hi Boris, I've had to focus elsewhere recently, but thank you for this.
Best regards Hugh

···

On 11 April 2017 at 04:58, Boris Grozev <boris@jitsi.org> wrote:

We just pushed some fixes which should alleviate this problem (available
in jitsi-videobridge 928).

Regards,
Boris

On 07/02/2017 14:58, Daniel Bareiro wrote:

On 07/02/17 17:43, Daniel Bareiro wrote:

My /etc/security/limits.d directory is currently empty. So you suggest

adding a file that has something like:

jvb soft nofile 10240
jvb hard nofile 10240

With "jvb" it would be enough or should I also add the user "I jicofo"?

I tried doing this configuration in /etc/security/limits.d but it did

not apply after a reboot. I think this is because these settings are for
users who have an active shell on the system. Can it be?

Here is verified:

------------------------------------------------------------
----------------
root@conference:~# cat /etc/security/limits.d/jitsi-meet.conf
jvb soft nofile 10240
jvb hard nofile 10240
------------------------------------------------------------
----------------
root@conference:~# ps -u jvb | tail -1 | awk '{ print $1 }'
434
------------------------------------------------------------
----------------
root@conference:~# cat /proc/434/limits | grep open
Max open files 4096 4096 files
------------------------------------------------------------
----------------
root@conference:~# su - jvb
jvb@conference:~$ ulimit -n
10240
------------------------------------------------------------
----------------

Kind regards,
Daniel

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

--
http://www.hughbarnard.org
http://www.twitter.com/hughbarnard
http://www.big-wave-heuristics.com/
<http://www.hackney-environment-network.org.uk/>


#12

Hi Daniel,

Hi, Boris.

We just pushed some fixes which should alleviate this problem (available
in jitsi-videobridge 928).

Thanks for your reply.

Yes, a while ago Damian Minkov told me that you had introduced a fix on
the 882 version of the Bridge [1]. It seems that there were several more
changes if the current version of the bridge is 928.

Yes, there was one fix a few months ago, and two more fixes yesterday.

Unfortunately I am using stable and here is not present this fix. Do you
think this fix may be available in stable in a short term?

I don't know. If manually backporting is an option for you, it should be easy to do, you need these two commits:
https://github.com/jitsi/ice4j/commit/a524e1b2c009b5d35f807cf035e58cedde9ebfc8
https://github.com/jitsi/ice4j/commit/5085b042bbf29057e517489f286dc25320b4de36

And this one for the previous fix:
https://github.com/jitsi/ice4j/commit/7c63d5202c3ab453028eefa2a6f81cd2f1f70ca3

Regards,
Boris

···

On 11/04/2017 06:22, Daniel Bareiro wrote:

On 11/04/17 00:58, Boris Grozev wrote:


#13

Hi Daniel,

Hi, Boris.

Unfortunately I am using stable and here is not present this fix. Do you
think this fix may be available in stable in a short term?

I don't know. If manually backporting is an option for you, it should be
easy to do, you need these two commits:
https://github.com/jitsi/ice4j/commit/a524e1b2c009b5d35f807cf035e58cedde9ebfc8

https://github.com/jitsi/ice4j/commit/5085b042bbf29057e517489f286dc25320b4de36

And this one for the previous fix:
https://github.com/jitsi/ice4j/commit/7c63d5202c3ab453028eefa2a6f81cd2f1f70ca3

I see the files involved in these patches are:

* src/main/java/org/ice4j/ice/ComponentSocket.java
* src/main/java/org/ice4j/ice/harvest/SinglePortUdpHarvester.java
* src/main/java/org/ice4j/ice/Component.java
* src/main/java/org/ice4j/socket/MergingDatagramSocket.java

I was reviewing the files in stable and found this:

root@conference:/usr/share/jitsi-videobridge/lib# ll
ice4j-2.0-20161221.230043-4.jar
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 17:27 ice4j-2.0-20161221.230043-4.jar

Which contains these files (among others):

    18100 2017-01-04 20:27 org/ice4j/ice/Component.class
     4878 2017-01-04 20:27 org/ice4j/ice/ComponentSocket.class
     5764 2017-01-04 20:27
org/ice4j/ice/harvest/SinglePortUdpHarvester.class
    10422 2017-01-04 20:27 org/ice4j/socket/MergingDatagramSocket.class

So it looks like I would have to generate a new version for
ice4j-2.0-20161221.230043-4.jar, though I'm not sure how I would have to
proceed. Could you tell me what procedure I should use? Maybe I can try
it in a test environment.

Thanks for your reply.

Kind regards,
Daniel

···

On 11/04/17 11:16, Boris Grozev wrote:

On 11/04/2017 06:22, Daniel Bareiro wrote:


#14

Hi Daniel,

Hi Daniel,

Hi, Boris.

Unfortunately I am using stable and here is not present this fix. Do you
think this fix may be available in stable in a short term?

I don't know. If manually backporting is an option for you, it should be
easy to do, you need these two commits:
https://github.com/jitsi/ice4j/commit/a524e1b2c009b5d35f807cf035e58cedde9ebfc8

https://github.com/jitsi/ice4j/commit/5085b042bbf29057e517489f286dc25320b4de36

And this one for the previous fix:
https://github.com/jitsi/ice4j/commit/7c63d5202c3ab453028eefa2a6f81cd2f1f70ca3

I see the files involved in these patches are:

* src/main/java/org/ice4j/ice/ComponentSocket.java
* src/main/java/org/ice4j/ice/harvest/SinglePortUdpHarvester.java
* src/main/java/org/ice4j/ice/Component.java
* src/main/java/org/ice4j/socket/MergingDatagramSocket.java

I was reviewing the files in stable and found this:

root@conference:/usr/share/jitsi-videobridge/lib# ll
ice4j-2.0-20161221.230043-4.jar
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 17:27 ice4j-2.0-20161221.230043-4.jar

Which contains these files (among others):

    18100 2017-01-04 20:27 org/ice4j/ice/Component.class
     4878 2017-01-04 20:27 org/ice4j/ice/ComponentSocket.class
     5764 2017-01-04 20:27
org/ice4j/ice/harvest/SinglePortUdpHarvester.class
    10422 2017-01-04 20:27 org/ice4j/socket/MergingDatagramSocket.class

So it looks like I would have to generate a new version for
ice4j-2.0-20161221.230043-4.jar, though I'm not sure how I would have to
proceed. Could you tell me what procedure I should use? Maybe I can try
it in a test environment.

You can either grab the latest ice4j (ice4j-2.0-20170410.185542-12.jar) from the maven repository[0], or you can cherry-pick the commits on top of the version from which ice4j-2.0-20161221.230043-4.jar was built (which is 43e6e5359d7c304c87d1b385db8d38996a450183).

Regards,
Boris

[0] https://github.com/jitsi/jitsi-maven-repository/

···

On 12/04/2017 15:48, Daniel Bareiro wrote:

On 11/04/17 11:16, Boris Grozev wrote:

On 11/04/2017 06:22, Daniel Bareiro wrote:


#15

Hi Daniel,

Hi, Boris.

Unfortunately I am using stable and here is not present this fix. Do
you think this fix may be available in stable in a short term?

I don't know. If manually backporting is an option for you, it should be
easy to do, you need these two commits:
https://github.com/jitsi/ice4j/commit/a524e1b2c009b5d35f807cf035e58cedde9ebfc8

https://github.com/jitsi/ice4j/commit/5085b042bbf29057e517489f286dc25320b4de36

And this one for the previous fix:
https://github.com/jitsi/ice4j/commit/7c63d5202c3ab453028eefa2a6f81cd2f1f70ca3

I see the files involved in these patches are:

* src/main/java/org/ice4j/ice/ComponentSocket.java
* src/main/java/org/ice4j/ice/harvest/SinglePortUdpHarvester.java
* src/main/java/org/ice4j/ice/Component.java
* src/main/java/org/ice4j/socket/MergingDatagramSocket.java

I was reviewing the files in stable and found this:

root@conference:/usr/share/jitsi-videobridge/lib# ll
ice4j-2.0-20161221.230043-4.jar
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 17:27
ice4j-2.0-20161221.230043-4.jar

Which contains these files (among others):

    18100 2017-01-04 20:27 org/ice4j/ice/Component.class
     4878 2017-01-04 20:27 org/ice4j/ice/ComponentSocket.class
     5764 2017-01-04 20:27
org/ice4j/ice/harvest/SinglePortUdpHarvester.class
    10422 2017-01-04 20:27
org/ice4j/socket/MergingDatagramSocket.class

So it looks like I would have to generate a new version for
ice4j-2.0-20161221.230043-4.jar, though I'm not sure how I would have to
proceed. Could you tell me what procedure I should use? Maybe I can try
it in a test environment.

You can either grab the latest ice4j (ice4j-2.0-20170410.185542-12.jar)
from the maven repository[0], or you can cherry-pick the commits on top
of the version from which ice4j-2.0-20161221.230043-4.jar was built
(which is 43e6e5359d7c304c87d1b385db8d38996a450183).

[0] https://github.com/jitsi/jitsi-maven-repository/

I have tried with two workarounds: bringing the maximum of open files
from 4096 to 10240, and reducing the frequency of health checks to one
every 10 min using in /etc/jitsi/jicofo/sip-communicator.properties:

org.jitsi.jicofo.HEALTH_CHECK_INTERVAL=600000

But unfortunately none of that worked and today I had this issue again
(the last time I had it was last Saturday):

···

On 12/04/17 18:26, Boris Grozev wrote:

-----------------------------------------------------------------------
Jicofo 2017-05-03 17:59:36.968 WARNING: [95]
org.jitsi.jicofo.JvbDoctor.log() Health check failed on:
jitsi-videobridge.meet.sipalto.com error: <error code="500"
type="WAIT"><internal-server-error
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/><text xml:lang="en"
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas">java.net.SocketException:
Too many open files (Error creating socket)</text></error>
Jicofo 2017-05-03 17:59:36.968 INFO: [39]
org.jitsi.jicofo.BridgeSelector.removeJvbAddress().192 Removing JVB:
jitsi-videobridge.meet.sipalto.com
Jicofo 2017-05-03 17:59:36.968 INFO: [39]
org.jitsi.jicofo.JvbDoctor.log() Stopping health-check task for:
jitsi-videobridge.meet.sipalto.com
-----------------------------------------------------------------------

I noticed that now in Github you mention a "testing" repository. I think
that's new because I do not remember seeing it before.

This means that it is a repository with Jitsi Meet compiled with
dependencies of Debian testing or that is a "testing" version built
with dependencies of stable Debian?

In the second case, I would like to know if this version of Jitsi Meet
has these patches applied and it is stable enough to be used in
production, since this issue with the bridge is a problem that
interrupts the service at random.

The idea is to avoid using something unstable as nightly. If there is no
alternative, I will have to follow the path you suggested in the other
email, although that may make maintenance difficult.

Kind regards,
Daniel

[1] https://github.com/jitsi/jitsi-meet


#16

Hi Daniel,

Hi, Boris.

Unfortunately I am using stable and here is not present this fix. Do
you think this fix may be available in stable in a short term?

I don't know. If manually backporting is an option for you, it should be
easy to do, you need these two commits:
https://github.com/jitsi/ice4j/commit/a524e1b2c009b5d35f807cf035e58cedde9ebfc8

https://github.com/jitsi/ice4j/commit/5085b042bbf29057e517489f286dc25320b4de36

And this one for the previous fix:
https://github.com/jitsi/ice4j/commit/7c63d5202c3ab453028eefa2a6f81cd2f1f70ca3

I see the files involved in these patches are:

* src/main/java/org/ice4j/ice/ComponentSocket.java
* src/main/java/org/ice4j/ice/harvest/SinglePortUdpHarvester.java
* src/main/java/org/ice4j/ice/Component.java
* src/main/java/org/ice4j/socket/MergingDatagramSocket.java

I was reviewing the files in stable and found this:

root@conference:/usr/share/jitsi-videobridge/lib# ll
ice4j-2.0-20161221.230043-4.jar
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 17:27
ice4j-2.0-20161221.230043-4.jar

Which contains these files (among others):

    18100 2017-01-04 20:27 org/ice4j/ice/Component.class
     4878 2017-01-04 20:27 org/ice4j/ice/ComponentSocket.class
     5764 2017-01-04 20:27
org/ice4j/ice/harvest/SinglePortUdpHarvester.class
    10422 2017-01-04 20:27
org/ice4j/socket/MergingDatagramSocket.class

So it looks like I would have to generate a new version for
ice4j-2.0-20161221.230043-4.jar, though I'm not sure how I would have to
proceed. Could you tell me what procedure I should use? Maybe I can try
it in a test environment.

You can either grab the latest ice4j (ice4j-2.0-20170410.185542-12.jar)
from the maven repository[0], or you can cherry-pick the commits on top
of the version from which ice4j-2.0-20161221.230043-4.jar was built
(which is 43e6e5359d7c304c87d1b385db8d38996a450183).

[0] https://github.com/jitsi/jitsi-maven-repository/

I have tried with two workarounds: bringing the maximum of open files
from 4096 to 10240, and reducing the frequency of health checks to one
every 10 min using in /etc/jitsi/jicofo/sip-communicator.properties:

org.jitsi.jicofo.HEALTH_CHECK_INTERVAL=600000

But unfortunately none of that worked and today I had this issue again
(the last time I had it was last Saturday):

-----------------------------------------------------------------------
Jicofo 2017-05-03 17:59:36.968 WARNING: [95]
org.jitsi.jicofo.JvbDoctor.log() Health check failed on:
jitsi-videobridge.meet.sipalto.com error: <error code="500"
type="WAIT"><internal-server-error
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/><text xml:lang="en"
xmlns="urn:ietf:params:xml:ns:xmpp-stanzas">java.net.SocketException:
Too many open files (Error creating socket)</text></error>
Jicofo 2017-05-03 17:59:36.968 INFO: [39]
org.jitsi.jicofo.BridgeSelector.removeJvbAddress().192 Removing JVB:
jitsi-videobridge.meet.sipalto.com
Jicofo 2017-05-03 17:59:36.968 INFO: [39]
org.jitsi.jicofo.JvbDoctor.log() Stopping health-check task for:
jitsi-videobridge.meet.sipalto.com
-----------------------------------------------------------------------

Can you please confirm the versions of jitsi-videobridge and ice4j that you had running?

I noticed that now in Github you mention a "testing" repository. I think
that's new because I do not remember seeing it before.

This means that it is a repository with Jitsi Meet compiled with
dependencies of Debian testing or that is a "testing" version built
with dependencies of stable Debian?

I'm not aware of any special "testing" repository. Can you point me to where I refer to this? Perhaps I made a typo, or did not express myself clearly.

Regards,
Boris

···

On 03/05/2017 16:02, Daniel Bareiro wrote:

On 12/04/17 18:26, Boris Grozev wrote:


#17

Hi, Boris.

Can you please confirm the versions of jitsi-videobridge and ice4j that
you had running?

Yes. I currently have the packages versions of the stable repository:

# aptitude show jitsi-videobridge | grep Version
Version: 879-1

I do not see a package called ice4j. How do I know the version? Maybe
it's the one mentioned in this file?

root@Jitsi:/usr/share/jitsi-videobridge/lib# ll | grep ice4j
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 20:27 ice4j-2.0-20161221.230043-4.jar

In any case it must be the version currently available in the stable
repository.

I noticed that now in Github you mention a "testing" repository. I think
that's new because I do not remember seeing it before.

This means that it is a repository with Jitsi Meet compiled with
dependencies of Debian testing or that is a "testing" version built
with dependencies of stable Debian?

I'm not aware of any special "testing" repository. Can you point me to
where I refer to this? Perhaps I made a typo, or did not express myself
clearly.

Yes, here [1], in the "Download" section. Three repositories are mentioned:

* Stable:

https://jitsi.org/Main/InstallJitsiMeetDebianStableRepository

* Testing:

https://jitsi.org/Main/InstallJitsiMeetDebianTestingRepository

* Nightly:

https://jitsi.org/Main/InstallJitsiMeetDebianNightlyRepository

It was not something you mentioned in this thread but something I
noticed these days. I don't remember seeing a "testing" repository before.

Thanks for your reply.

Kind regards,
Daniel

[1] https://github.com/jitsi/jitsi-meet

···

On 03/05/17 18:26, Boris Grozev wrote:


#18

Hi, Boris.

Can you please confirm the versions of jitsi-videobridge and ice4j that
you had running?

Yes. I currently have the packages versions of the stable repository:

# aptitude show jitsi-videobridge | grep Version
Version: 879-1

I do not see a package called ice4j. How do I know the version? Maybe
it's the one mentioned in this file?

root@Jitsi:/usr/share/jitsi-videobridge/lib# ll | grep ice4j
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 20:27 ice4j-2.0-20161221.230043-4.jar

Yes, this is the file.

In any case it must be the version currently available in the stable
repository.

These are old versions with known (and fixed) leaks. This github issue contains a relevant discussion:
https://github.com/jitsi/jitsi-meet/issues/1390

I noticed that now in Github you mention a "testing" repository. I think
that's new because I do not remember seeing it before.

This means that it is a repository with Jitsi Meet compiled with
dependencies of Debian testing or that is a "testing" version built
with dependencies of stable Debian?

I'm not aware of any special "testing" repository. Can you point me to
where I refer to this? Perhaps I made a typo, or did not express myself
clearly.

Yes, here [1], in the "Download" section. Three repositories are mentioned:

* Stable:

https://jitsi.org/Main/InstallJitsiMeetDebianStableRepository

* Testing:

https://jitsi.org/Main/InstallJitsiMeetDebianTestingRepository

* Nightly:

https://jitsi.org/Main/InstallJitsiMeetDebianNightlyRepository

It was not something you mentioned in this thread but something I
noticed these days. I don't remember seeing a "testing" repository before.

Ah, I see. Damian explained it to me today: packages which pass a set of tests automatically go to "testing", while "stable" is only updated manually. Apparently "stable" still contains versions with the leak.

Regards,
Boris

···

On 03/05/2017 17:16, Daniel Bareiro wrote:

On 03/05/17 18:26, Boris Grozev wrote:


#19

Hi, Boris.

Can you please confirm the versions of jitsi-videobridge and ice4j that
you had running?

Yes. I currently have the packages versions of the stable repository:

# aptitude show jitsi-videobridge | grep Version
Version: 879-1

I do not see a package called ice4j. How do I know the version? Maybe
it's the one mentioned in this file?

root@Jitsi:/usr/share/jitsi-videobridge/lib# ll | grep ice4j
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 20:27
ice4j-2.0-20161221.230043-4.jar

Yes, this is the file.

Thanks for confirm.

In any case it must be the version currently available in the stable
repository.

These are old versions with known (and fixed) leaks. This github issue
contains a relevant discussion:
https://github.com/jitsi/jitsi-meet/issues/1390

Thanks for the reference. I was reading all the messages of this issue
and I think I read a reply where Damian said that the patches are
included in the testing repository. I also saw there some recent posts
from some people saying they have this issue although I'm not sure if
they tried the latest version.

I noticed that now in Github you mention a "testing" repository. I
think that's new because I do not remember seeing it before.

This means that it is a repository with Jitsi Meet compiled with
dependencies of Debian testing or that is a "testing" version built
with dependencies of stable Debian?

I'm not aware of any special "testing" repository. Can you point me to
where I refer to this? Perhaps I made a typo, or did not express myself
clearly.

Yes, here [1], in the "Download" section. Three repositories are
mentioned:

* Stable:

https://jitsi.org/Main/InstallJitsiMeetDebianStableRepository

* Testing:

https://jitsi.org/Main/InstallJitsiMeetDebianTestingRepository

* Nightly:

https://jitsi.org/Main/InstallJitsiMeetDebianNightlyRepository

It was not something you mentioned in this thread but something I
noticed these days. I don't remember seeing a "testing" repository
before.

Ah, I see. Damian explained it to me today: packages which pass a set of
tests automatically go to "testing", while "stable" is only updated
manually. Apparently "stable" still contains versions with the leak.

Thanks for checking this with Damian. Some questions about this:

Versions in this repository include this leaks fixes?

Dependencies to use this repository can be covered with Debian stable
repositories?

Would you recommend installing Jitsi Meet with this repository from the
point of view of stability? If so, I guess this "testing" version will
eventually become "stable". So I could go from "stable" to "testing" and
then back to "stable" when "testing" and "stable" are equivalent. The
idea would be to use something relatively stable and easy to maintain
until you release the new stable version with these fixes of the leaks.

Thanks for your reply.

Kind regards,
Daniel

···

On 03/05/17 20:43, Boris Grozev wrote:


#20

Hi, Boris.

Can you please confirm the versions of jitsi-videobridge and ice4j that
you had running?

Yes. I currently have the packages versions of the stable repository:

# aptitude show jitsi-videobridge | grep Version
Version: 879-1

I do not see a package called ice4j. How do I know the version? Maybe
it's the one mentioned in this file?

root@Jitsi:/usr/share/jitsi-videobridge/lib# ll | grep ice4j
-rw-r--r-- 1 jvb jitsi 499876 Jan 4 20:27
ice4j-2.0-20161221.230043-4.jar

Yes, this is the file.

Thanks for confirm.

In any case it must be the version currently available in the stable
repository.

These are old versions with known (and fixed) leaks. This github issue
contains a relevant discussion:
https://github.com/jitsi/jitsi-meet/issues/1390

Thanks for the reference. I was reading all the messages of this issue
and I think I read a reply where Damian said that the patches are
included in the testing repository. I also saw there some recent posts
from some people saying they have this issue although I'm not sure if
they tried the latest version.

I noticed that now in Github you mention a "testing" repository. I
think that's new because I do not remember seeing it before.

This means that it is a repository with Jitsi Meet compiled with
dependencies of Debian testing or that is a "testing" version built
with dependencies of stable Debian?

I'm not aware of any special "testing" repository. Can you point me to
where I refer to this? Perhaps I made a typo, or did not express myself
clearly.

Yes, here [1], in the "Download" section. Three repositories are
mentioned:

* Stable:

https://jitsi.org/Main/InstallJitsiMeetDebianStableRepository

* Testing:

https://jitsi.org/Main/InstallJitsiMeetDebianTestingRepository

* Nightly:

https://jitsi.org/Main/InstallJitsiMeetDebianNightlyRepository

It was not something you mentioned in this thread but something I
noticed these days. I don't remember seeing a "testing" repository
before.

Ah, I see. Damian explained it to me today: packages which pass a set of
tests automatically go to "testing", while "stable" is only updated
manually. Apparently "stable" still contains versions with the leak.

Thanks for checking this with Damian. Some questions about this:

Versions in this repository include this leaks fixes?

Dependencies to use this repository can be covered with Debian stable
repositories?

Yes to both questions.

Would you recommend installing Jitsi Meet with this repository from the
point of view of stability? If so, I guess this "testing" version will
eventually become "stable". So I could go from "stable" to "testing" and
then back to "stable" when "testing" and "stable" are equivalent. The
idea would be to use something relatively stable and easy to maintain
until you release the new stable version with these fixes of the leaks.

Well, testing does get broken occasionally. It would be nice if we could maintain "stable" more, but there are limited resources.

FWIW, meet.jit.si is currently running the following versions:
jitsi-videobridge 937,
jicofo 348,
jitsi-meet 1924

These are relatively stable, and they include the fixes for the leaks that you are running into.

Regards,
Boris

···

On 03/05/2017 19:57, Daniel Bareiro wrote:

On 03/05/17 20:43, Boris Grozev wrote: