[jitsi-dev] UDP Packet loss


#1

Dear all,

we recently noticed a huge packet loss on our jitsi server (about 1 million packets a day) is that a normal behavior ? do you have any recommendations on how to reduce this packet loss ? we're using Debian jessie 8.9

#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)

Here's the result of netstat -su this morning:
Udp:
     7262954631 packets received
     8382 packets to unknown port received.
     31852234 packet receive errors
     8297041937 packets sent
     RcvbufErrors: 31852234

and here's the result for this afternoon (4 hours later):
Udp:
     7283647268 packets received
     8382 packets to unknown port received.
     32176192 packet receive errors
     8323123628 packets sent
     RcvbufErrors: 32176192

Any help would be extremely appreciated.

Kind regards
Hamza


#2

Hi Hamza,

Are you using docker? Is the sysctl command available on the system that runs the JVB?

Best,
George

···

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Dear all,

we recently noticed a huge packet loss on our jitsi server (about 1 million packets a day) is that a normal behavior ? do you have any recommendations on how to reduce this packet loss ? we're using Debian jessie 8.9

#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)

Here's the result of netstat -su this morning:
Udp:
   7262954631 packets received
   8382 packets to unknown port received.
   31852234 packet receive errors
   8297041937 packets sent
   RcvbufErrors: 31852234

and here's the result for this afternoon (4 hours later):
Udp:
   7283647268 packets received
   8382 packets to unknown port received.
   32176192 packet receive errors
   8323123628 packets sent
   RcvbufErrors: 32176192

Any help would be extremely appreciated.

Kind regards
Hamza

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#3

Hello George,

No we're actually using VMWare for our VMs. and Yes, the sysctl command is available on the system.

Regards
Hamza

···

Le 19/09/2017 15:37, > George Politis (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

Hi Hamza,

Are you using docker? Is the sysctl command available on the system
that runs the JVB?

Best,
George

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE >> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Dear all,

we recently noticed a huge packet loss on our jitsi server (about 1 million packets a day) is that a normal behavior ? do you have any recommendations on how to reduce this packet loss ? we're using Debian jessie 8.9

#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)

Here's the result of netstat -su this morning:
Udp:
   7262954631 packets received
   8382 packets to unknown port received.
   31852234 packet receive errors
   8297041937 packets sent
   RcvbufErrors: 31852234

and here's the result for this afternoon (4 hours later):
Udp:
   7283647268 packets received
   8382 packets to unknown port received.
   32176192 packet receive errors
   8323123628 packets sent
   RcvbufErrors: 32176192

Any help would be extremely appreciated.

Kind regards
Hamza

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#4

Are you using the installer or compiling from source? There was an issue with the UDP socket buffer being too small that we’ve fixed in the installer. If you’re baking your own JVB you may want to tweak this.

···

On Sep 19, 2017, at 3:46 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello George,

No we're actually using VMWare for our VMs. and Yes, the sysctl command is available on the system.

Regards
Hamza

Le 19/09/2017 15:37, > George Politis (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

Hi Hamza,
Are you using docker? Is the sysctl command available on the system
that runs the JVB?
Best,
George

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:
Dear all,
we recently noticed a huge packet loss on our jitsi server (about 1 million packets a day) is that a normal behavior ? do you have any recommendations on how to reduce this packet loss ? we're using Debian jessie 8.9
#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
Here's the result of netstat -su this morning:
Udp:
  7262954631 packets received
  8382 packets to unknown port received.
  31852234 packet receive errors
  8297041937 packets sent
  RcvbufErrors: 31852234
and here's the result for this afternoon (4 hours later):
Udp:
  7283647268 packets received
  8382 packets to unknown port received.
  32176192 packet receive errors
  8323123628 packets sent
  RcvbufErrors: 32176192
Any help would be extremely appreciated.
Kind regards
Hamza
_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#5

I installed Jitsi Meet from the installer a couple months ago. I think this issue was fixed after I made the installation.
Could you please provide me with the recommended values for the UDP socket buffer ?

Many thanks for your help.

regards
Hamza

···

Le 19/09/2017 15:49, > George Politis (par Internet, dépôt gp@sip-communicator.org) a écrit :

Are you using the installer or compiling from source? There was an
issue with the UDP socket buffer being too small that we’ve fixed in
the installer. If you’re baking your own JVB you may want to tweak
this.

On Sep 19, 2017, at 3:46 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE >> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello George,

No we're actually using VMWare for our VMs. and Yes, the sysctl command is available on the system.

Regards
Hamza

Le 19/09/2017 15:37, > George Politis (par Internet, dépôt >> dev-bounces@jitsi.org) a écrit :

Hi Hamza,
Are you using docker? Is the sysctl command available on the system
that runs the JVB?
Best,
George

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE >>>> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:
Dear all,
we recently noticed a huge packet loss on our jitsi server (about 1 million packets a day) is that a normal behavior ? do you have any recommendations on how to reduce this packet loss ? we're using Debian jessie 8.9
#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
Here's the result of netstat -su this morning:
Udp:
  7262954631 packets received
  8382 packets to unknown port received.
  31852234 packet receive errors
  8297041937 packets sent
  RcvbufErrors: 31852234
and here's the result for this afternoon (4 hours later):
Udp:
  7283647268 packets received
  8382 packets to unknown port received.
  32176192 packet receive errors
  8323123628 packets sent
  RcvbufErrors: 32176192
Any help would be extremely appreciated.
Kind regards
Hamza
_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#6

Dear all,

I think the UDP packet loss issue is solved by changed the UDP socket buffer:
sudo sysctl -w net.core.rmem_max=10485760
sudo sysctl -w net.core.netdev_max_backlog=100000

Many thanks to @George

Regards
Hamza

···

Le 19/09/2017 16:05, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

I installed Jitsi Meet from the installer a couple months ago. I think
this issue was fixed after I made the installation.
Could you please provide me with the recommended values for the UDP
socket buffer ?

Many thanks for your help.

regards
Hamza

Le 19/09/2017 15:49, > George Politis (par Internet, dépôt > gp@sip-communicator.org) a écrit :

Are you using the installer or compiling from source? There was an
issue with the UDP socket buffer being too small that we’ve fixed in
the installer. If you’re baking your own JVB you may want to tweak
this.

On Sep 19, 2017, at 3:46 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE >>> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello George,

No we're actually using VMWare for our VMs. and Yes, the sysctl command is available on the system.

Regards
Hamza

Le 19/09/2017 15:37, > George Politis (par Internet, dépôt >>> dev-bounces@jitsi.org) a écrit :

Hi Hamza,
Are you using docker? Is the sysctl command available on the system
that runs the JVB?
Best,
George

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - >>>>> SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE >>>>> <hamza.khait@i-carre.net> wrote:
Dear all,
we recently noticed a huge packet loss on our jitsi server (about 1 million packets a day) is that a normal behavior ? do you have any recommendations on how to reduce this packet loss ? we're using Debian jessie 8.9
#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
Here's the result of netstat -su this morning:
Udp:
  7262954631 packets received
  8382 packets to unknown port received.
  31852234 packet receive errors
  8297041937 packets sent
  RcvbufErrors: 31852234
and here's the result for this afternoon (4 hours later):
Udp:
  7283647268 packets received
  8382 packets to unknown port received.
  32176192 packet receive errors
  8323123628 packets sent
  RcvbufErrors: 32176192
Any help would be extremely appreciated.
Kind regards
Hamza
_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#7

Hello,

Few days after changing rmem_max and netdev_max_backlog values, we noticed that we still have an important UDP packets loss on our Jitsi-Meet server. Here's the result of "netstat -su" since last reboot :
~$ sudo netstat -su
Udp:
     838424192 packets received
     55 packets to unknown port received.
     812094 packet receive errors
     976445626 packets sent
     RcvbufErrors: 812094

That means 812094 UDP packets are corrupt/lost (0.9% of the received packets) maybe because of incorrect headers or checksum or full RCV buffer size.
We developed a script that captures the corrupt/lost packets per second (CSV file attached), and every mintute once it exceeds 3000 pkt/sec we lose up to 12% of the received UDP packets per second, which is not normal.

In order to solve this issue, do you think the following solutions might help us to reduce the UDP packet drops? :
- Setting net.core.rmem_max to 16MB
- Checking and optimizing Java Garbage Collector, because regular UDP losses might be caused by JAVA GC
- Installing JVB on a physical machine rather than VM

What do you think about that ? is it worth it to increase the value of net.core.rmem_max?
Any suggestions would be extremely appreciated.

Best regards
Hamza

···

Le 20/09/2017 14:56, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

Dear all,

I think the UDP packet loss issue is solved by changed the UDP socket buffer:
sudo sysctl -w net.core.rmem_max=10485760
sudo sysctl -w net.core.netdev_max_backlog=100000

Many thanks to @George

Regards
Hamza

Le 19/09/2017 16:05, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE
ANNUAIRE ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a
écrit :

I installed Jitsi Meet from the installer a couple months ago. I think
this issue was fixed after I made the installation.
Could you please provide me with the recommended values for the UDP
socket buffer ?

Many thanks for your help.

regards
Hamza

Le 19/09/2017 15:49, > George Politis (par Internet, dépôt >> gp@sip-communicator.org) a écrit :

Are you using the installer or compiling from source? There was an
issue with the UDP socket buffer being too small that we’ve fixed in
the installer. If you’re baking your own JVB you may want to tweak
this.

On Sep 19, 2017, at 3:46 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE >>>> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello George,

No we're actually using VMWare for our VMs. and Yes, the sysctl command is available on the system.

Regards
Hamza

Le 19/09/2017 15:37, > George Politis (par Internet, dépôt >>>> dev-bounces@jitsi.org) a écrit :

Hi Hamza,
Are you using docker? Is the sysctl command available on the system
that runs the JVB?
Best,
George

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - >>>>>> SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE >>>>>> <hamza.khait@i-carre.net> wrote:
Dear all,
we recently noticed a huge packet loss on our jitsi server (about 1 million packets a day) is that a normal behavior ? do you have any recommendations on how to reduce this packet loss ? we're using Debian jessie 8.9
#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
Here's the result of netstat -su this morning:
Udp:
  7262954631 packets received
  8382 packets to unknown port received.
  31852234 packet receive errors
  8297041937 packets sent
  RcvbufErrors: 31852234
and here's the result for this afternoon (4 hours later):
Udp:
  7283647268 packets received
  8382 packets to unknown port received.
  32176192 packet receive errors
  8323123628 packets sent
  RcvbufErrors: 32176192
Any help would be extremely appreciated.
Kind regards
Hamza
_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#8

Hi,

The first thing to do would be to check the actual size of the buffer in
the logs of the bridge.

Regards,
Boris

···

On Thu, Jan 18, 2018 at 3:52 AM KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello,

Few days after changing rmem_max and netdev_max_backlog values, we
noticed that we still have an important UDP packets loss on our
Jitsi-Meet server. Here's the result of "netstat -su" since last reboot
:
~$ sudo netstat -su
Udp:
     838424192 packets received
     55 packets to unknown port received.
     812094 packet receive errors
     976445626 packets sent
     RcvbufErrors: 812094

That means 812094 UDP packets are corrupt/lost (0.9% of the received
packets) maybe because of incorrect headers or checksum or full RCV
buffer size.
We developed a script that captures the corrupt/lost packets per second
(CSV file attached), and every mintute once it exceeds 3000 pkt/sec we
lose up to 12% of the received UDP packets per second, which is not
normal.

In order to solve this issue, do you think the following solutions might
help us to reduce the UDP packet drops? :
- Setting net.core.rmem_max to 16MB
- Checking and optimizing Java Garbage Collector, because regular UDP
losses might be caused by JAVA GC
- Installing JVB on a physical machine rather than VM

What do you think about that ? is it worth it to increase the value of
net.core.rmem_max?
Any suggestions would be extremely appreciated.

Best regards
Hamza

Le 20/09/2017 14:56, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE > ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a écrit :
> Dear all,
>
> I think the UDP packet loss issue is solved by changed the UDP socket
> buffer:
> sudo sysctl -w net.core.rmem_max=10485760
> sudo sysctl -w net.core.netdev_max_backlog=100000
>
> Many thanks to @George
>
> Regards
> Hamza
>
> Le 19/09/2017 16:05, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE
> ANNUAIRE ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a
> écrit :
>> I installed Jitsi Meet from the installer a couple months ago. I think
>> this issue was fixed after I made the installation.
>> Could you please provide me with the recommended values for the UDP
>> socket buffer ?
>>
>> Many thanks for your help.
>>
>> regards
>> Hamza
>>
>> Le 19/09/2017 15:49, > George Politis (par Internet, dépôt > >> gp@sip-communicator.org) a écrit :
>>> Are you using the installer or compiling from source? There was an
>>> issue with the UDP socket buffer being too small that we’ve fixed in
>>> the installer. If you’re baking your own JVB you may want to tweak
>>> this.
>>>
>>>> On Sep 19, 2017, at 3:46 PM, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE > >>>> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:
>>>>
>>>> Hello George,
>>>>
>>>> No we're actually using VMWare for our VMs. and Yes, the sysctl
>>>> command is available on the system.
>>>>
>>>> Regards
>>>> Hamza
>>>>
>>>> Le 19/09/2017 15:37, > George Politis (par Internet, dépôt > >>>> dev-bounces@jitsi.org) a écrit :
>>>>> Hi Hamza,
>>>>> Are you using docker? Is the sysctl command available on the system
>>>>> that runs the JVB?
>>>>> Best,
>>>>> George
>>>>>> On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - > >>>>>> SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE > >>>>>> <hamza.khait@i-carre.net> wrote:
>>>>>> Dear all,
>>>>>> we recently noticed a huge packet loss on our jitsi server (about
>>>>>> 1 million packets a day) is that a normal behavior ? do you have
>>>>>> any recommendations on how to reduce this packet loss ? we're
>>>>>> using Debian jessie 8.9
>>>>>> #cat /proc/version
>>>>>> Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc
>>>>>> version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2
>>>>>> (2016-10-19)
>>>>>> Here's the result of netstat -su this morning:
>>>>>> Udp:
>>>>>> 7262954631 packets received
>>>>>> 8382 packets to unknown port received.
>>>>>> 31852234 packet receive errors
>>>>>> 8297041937 packets sent
>>>>>> RcvbufErrors: 31852234
>>>>>> and here's the result for this afternoon (4 hours later):
>>>>>> Udp:
>>>>>> 7283647268 packets received
>>>>>> 8382 packets to unknown port received.
>>>>>> 32176192 packet receive errors
>>>>>> 8323123628 packets sent
>>>>>> RcvbufErrors: 32176192
>>>>>> Any help would be extremely appreciated.
>>>>>> Kind regards
>>>>>> Hamza
>>>>>> _______________________________________________
>>>>>> dev mailing list
>>>>>> dev@jitsi.org
>>>>>> Unsubscribe instructions and other list options:
>>>>>> http://lists.jitsi.org/mailman/listinfo/dev
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> dev@jitsi.org
>>>>> Unsubscribe instructions and other list options:
>>>>> http://lists.jitsi.org/mailman/listinfo/dev
>>
>> _______________________________________________
>> dev mailing list
>> dev@jitsi.org
>> Unsubscribe instructions and other list options:
>> http://lists.jitsi.org/mailman/listinfo/dev
>
> _______________________________________________
> dev mailing list
> dev@jitsi.org
> Unsubscribe instructions and other list options:
> http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#9

Hello,

Here's the buffer size in JVB logs :

JVB 2018-01-19 11:05:31.585 INFOS: [11] org.jitsi.impl.configuration.ConfigurationServiceImpl.log() org.ice4j.ice.harvest.AbstractUdpListener.SO_RCVBUF=10485760
JVB 2018-01-19 11:06:06.071 INFOS: [50] org.ice4j.ice.harvest.AbstractUdpListener.<init>() Initialized AbstractUdpListener with address 10.167.124.125:10000/udp. Receive buffer size 10485760 (asked for 10485760)

The value is correct. Please find also the CSV file that I forgot to attach on my previous email.

Kind regards
Hamza

UDP_Packet_Loss.csv (399 KB)

···

Le 18/01/2018 16:30, > Boris Grozev (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

Hi,

The first thing to do would be to check the actual size of the buffer in the logs of the bridge.

Regards,
Boris

On Thu, Jan 18, 2018 at 3:52 AM KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE > ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello,

Few days after changing rmem_max and netdev_max_backlog values, we
noticed that we still have an important UDP packets loss on our
Jitsi-Meet server. Here's the result of "netstat -su" since last reboot
:
~$ sudo netstat -su
Udp:
838424192 packets received
55 packets to unknown port received.
812094 packet receive errors
976445626 packets sent
RcvbufErrors: 812094

That means 812094 UDP packets are corrupt/lost (0.9% of the received
packets) maybe because of incorrect headers or checksum or full RCV
buffer size.
We developed a script that captures the corrupt/lost packets per second
(CSV file attached), and every mintute once it exceeds 3000 pkt/sec we
lose up to 12% of the received UDP packets per second, which is not
normal.

In order to solve this issue, do you think the following solutions might
help us to reduce the UDP packet drops? :
- Setting net.core.rmem_max to 16MB
- Checking and optimizing Java Garbage Collector, because regular UDP
losses might be caused by JAVA GC
- Installing JVB on a physical machine rather than VM

What do you think about that ? is it worth it to increase the value of
net.core.rmem_max?
Any suggestions would be extremely appreciated.

Best regards
Hamza

Le 20/09/2017 14:56, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE >> ANNUAIRE >> ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

Dear all,

I think the UDP packet loss issue is solved by changed the UDP socket
buffer:
sudo sysctl -w net.core.rmem_max=10485760
sudo sysctl -w net.core.netdev_max_backlog=100000

Many thanks to @George

Regards
Hamza

Le 19/09/2017 16:05, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE
ANNUAIRE ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a
écrit :

I installed Jitsi Meet from the installer a couple months ago. I think
this issue was fixed after I made the installation.
Could you please provide me with the recommended values for the UDP
socket buffer ?

Many thanks for your help.

regards
Hamza

Le 19/09/2017 15:49, > George Politis (par Internet, dépôt >>>> gp@sip-communicator.org) a écrit :

Are you using the installer or compiling from source? There was an
issue with the UDP socket buffer being too small that we've fixed in
the installer. If you're baking your own JVB you may want to tweak
this.

On Sep 19, 2017, at 3:46 PM, KHAIT Hamza - >>>>>> SG/SPSSI/CPII/DOSE/ET/PNE >>>>>> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello George,

No we're actually using VMWare for our VMs. and Yes, the sysctl
command is available on the system.

Regards
Hamza

Le 19/09/2017 15:37, > George Politis (par Internet, dépôt >>>>>> dev-bounces@jitsi.org) a écrit :

Hi Hamza,
Are you using docker? Is the sysctl command available on the system
that runs the JVB?
Best,
George

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - >>>>>>>> SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE >>>>>>>> <hamza.khait@i-carre.net> wrote:
Dear all,
we recently noticed a huge packet loss on our jitsi server (about
1 million packets a day) is that a normal behavior ? do you have
any recommendations on how to reduce this packet loss ? we're
using Debian jessie 8.9
#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc
version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2
(2016-10-19)
Here's the result of netstat -su this morning:
Udp:
7262954631 packets received
8382 packets to unknown port received.
31852234 packet receive errors
8297041937 packets sent
RcvbufErrors: 31852234
and here's the result for this afternoon (4 hours later):
Udp:
7283647268 packets received
8382 packets to unknown port received.
32176192 packet receive errors
8323123628 packets sent
RcvbufErrors: 32176192
Any help would be extremely appreciated.
Kind regards
Hamza
_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#10

CORRECTION... Sorry I gave you the log of my testing server.

Here's the log of JVB on my production server :

JVB 2018-01-19 13:33:46.814 INFOS: [36] org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>() Initialized SinglePortUdpHarvester with address 10.167.124.124:10000/udp. Receive buffer size 106496

The buffer size is not correct actually. JVB receives 106496 while rmem_max is set to 10485760

That is a bit strange

Hamza

···

Le 19/01/2018 13:24, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

Hello,

Here's the buffer size in JVB logs :

JVB 2018-01-19 11:05:31.585 INFOS: [11]
org.jitsi.impl.configuration.ConfigurationServiceImpl.log()
org.ice4j.ice.harvest.AbstractUdpListener.SO_RCVBUF=10485760
JVB 2018-01-19 11:06:06.071 INFOS: [50]
org.ice4j.ice.harvest.AbstractUdpListener.<init>() Initialized
AbstractUdpListener with address 10.167.124.125:10000/udp. Receive
buffer size 10485760 (asked for 10485760)

The value is correct. Please find also the CSV file that I forgot to
attach on my previous email.

Kind regards
Hamza

Le 18/01/2018 16:30, > Boris Grozev (par Internet, dépôt > dev-bounces@jitsi.org) a écrit :

Hi,

The first thing to do would be to check the actual size of the buffer in the logs of the bridge.

Regards,
Boris

On Thu, Jan 18, 2018 at 3:52 AM KHAIT Hamza - >> SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE >> <hamza.khait@i-carre.net> wrote:

Hello,

Few days after changing rmem_max and netdev_max_backlog values, we
noticed that we still have an important UDP packets loss on our
Jitsi-Meet server. Here's the result of "netstat -su" since last reboot
:
~$ sudo netstat -su
Udp:
838424192 packets received
55 packets to unknown port received.
812094 packet receive errors
976445626 packets sent
RcvbufErrors: 812094

That means 812094 UDP packets are corrupt/lost (0.9% of the received
packets) maybe because of incorrect headers or checksum or full RCV
buffer size.
We developed a script that captures the corrupt/lost packets per second
(CSV file attached), and every mintute once it exceeds 3000 pkt/sec we
lose up to 12% of the received UDP packets per second, which is not
normal.

In order to solve this issue, do you think the following solutions might
help us to reduce the UDP packet drops? :
- Setting net.core.rmem_max to 16MB
- Checking and optimizing Java Garbage Collector, because regular UDP
losses might be caused by JAVA GC
- Installing JVB on a physical machine rather than VM

What do you think about that ? is it worth it to increase the value of
net.core.rmem_max?
Any suggestions would be extremely appreciated.

Best regards
Hamza

Le 20/09/2017 14:56, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE >>> ANNUAIRE >>> ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a écrit :

Dear all,

I think the UDP packet loss issue is solved by changed the UDP socket
buffer:
sudo sysctl -w net.core.rmem_max=10485760
sudo sysctl -w net.core.netdev_max_backlog=100000

Many thanks to @George

Regards
Hamza

Le 19/09/2017 16:05, > KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE
ANNUAIRE ET MESSAGERIE (par Internet, dépôt dev-bounces@jitsi.org) a
écrit :

I installed Jitsi Meet from the installer a couple months ago. I think
this issue was fixed after I made the installation.
Could you please provide me with the recommended values for the UDP
socket buffer ?

Many thanks for your help.

regards
Hamza

Le 19/09/2017 15:49, > George Politis (par Internet, dépôt >>>>> gp@sip-communicator.org) a écrit :

Are you using the installer or compiling from source? There was an
issue with the UDP socket buffer being too small that we've fixed in
the installer. If you're baking your own JVB you may want to tweak
this.

On Sep 19, 2017, at 3:46 PM, KHAIT Hamza - >>>>>>> SG/SPSSI/CPII/DOSE/ET/PNE >>>>>>> ANNUAIRE ET MESSAGERIE <hamza.khait@i-carre.net> wrote:

Hello George,

No we're actually using VMWare for our VMs. and Yes, the sysctl
command is available on the system.

Regards
Hamza

Le 19/09/2017 15:37, > George Politis (par Internet, dépôt >>>>>>> dev-bounces@jitsi.org) a écrit :

Hi Hamza,
Are you using docker? Is the sysctl command available on the system
that runs the JVB?
Best,
George

On Sep 19, 2017, at 3:31 PM, KHAIT Hamza - >>>>>>>>> SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE >>>>>>>>> <hamza.khait@i-carre.net> wrote:
Dear all,
we recently noticed a huge packet loss on our jitsi server (about
1 million packets a day) is that a normal behavior ? do you have
any recommendations on how to reduce this packet loss ? we're
using Debian jessie 8.9
#cat /proc/version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc
version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.36-1+deb8u2
(2016-10-19)
Here's the result of netstat -su this morning:
Udp:
7262954631 packets received
8382 packets to unknown port received.
31852234 packet receive errors
8297041937 packets sent
RcvbufErrors: 31852234
and here's the result for this afternoon (4 hours later):
Udp:
7283647268 packets received
8382 packets to unknown port received.
32176192 packet receive errors
8323123628 packets sent
RcvbufErrors: 32176192
Any help would be extremely appreciated.
Kind regards
Hamza
_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev


#11

I don't know why this happened, but it would explain your observations of dropped packets.

Regards,
Boris

···

On 19/01/2018 06:39, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE ET MESSAGERIE wrote:

CORRECTION... Sorry I gave you the log of my testing server.

Here's the log of JVB on my production server :

JVB 2018-01-19 13:33:46.814 INFOS: [36]
org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>() Initialized
SinglePortUdpHarvester with address 10.167.124.124:10000/udp. Receive
buffer size 106496

The buffer size is not correct actually. JVB receives 106496 while
rmem_max is set to 10485760

That is a bit strange


#12

Hello,

Just FYI, I'm not working with the latest JVB version on production, my current version is actually 0.1.820
Maybe an update would solve the issue... but I needed first to understand what happened.

regards
Hamza

···

Le 19/01/2018 18:06, > Boris Grozev (par Internet, dépôt boris@sip-communicator.org) a écrit :

On 19/01/2018 06:39, KHAIT Hamza - SG/SPSSI/CPII/DOSE/ET/PNE ANNUAIRE > ET MESSAGERIE wrote:

CORRECTION... Sorry I gave you the log of my testing server.

Here's the log of JVB on my production server :

JVB 2018-01-19 13:33:46.814 INFOS: [36]
org.ice4j.ice.harvest.SinglePortUdpHarvester.<init>() Initialized
SinglePortUdpHarvester with address 10.167.124.124:10000/udp. Receive
buffer size 106496

The buffer size is not correct actually. JVB receives 106496 while
rmem_max is set to 10485760

That is a bit strange

I don't know why this happened, but it would explain your observations
of dropped packets.

Regards,
Boris