[jitsi-dev] prosody update "possibly" broke my jitmeet

Peter_Villeneuve · 2014-05-08 19:48:02 UTC

Hi,

I foolishly updated a bunch of packages on my debian 7 server yesterday,
and now it seems my jitmeet installation is horribly broken. One of the
many packages that was updated to latest version was prosody, which is
where I suspect things went wrong.

Has anyone else experienced this? I tried downgrading to previous prosody
but the issue remains.

Basically I can no longer get ice candidates negotiated, so no audio or
video.

I've tried reinstalling jitmeet, restund, prosody, etc, all to no avail.
Jitsi videobridge seems to freeze when offering the first sdp offer, until
it finally times out.

The only clue in the all the logs I've seen is this apparent bosh error
with nginx:

[error] 7972#0: *69 upstream timed out (110: Connection timed out) while
reading response header from upstream, client: xx.xx.xx.xx, server:
meet.domain.com, request: "POST /http-bind HTTP/1.1", upstream:
"http://[::1]:5280/http-bind",
host: "meet.domain.com", referrer: "https://meet.domain.com/s6xtj6rzq90y66r

I've spent the last 24 hours trying to get things back to normal, all to no
avail.

Any clues as to what's wrong? Has this happened to anyone else?

Cheers,

Peter

lmarinov · 2014-05-09 06:10:27 UTC

It may be of use to provide the version/build number of the related software packages such as Jitsi Videobridge.

Peter_Villeneuve · 2014-05-09 14:20:27 UTC

Hi Lyubomir,

Thanks for helping out.

I'm using jitsi-videobridge-linux-x64-99 and latest jitmeet from git on my
debian 7 x64 server.
I've tried downgrading Prosody to 0.9.4 but the problem remains. I also
tried Prosody trunk nightly build 482 (2014-04-26, 44de98f516f5) and again
videobridge hangs (when negotiating ICE)? Interestingly, when I try to grab
the jitmeet logs Chrome and Chromium both crash (out of memory). All I
could do was grab the last stanzas from the videobridge console. See
pastebin of the one it hangs on forever here
<http://pastebin.com/iHuvmDNh>(presumably this leads to the bosh
timeout we see in the nginx logs)

I'm having a tough time isolating the problem. Is it a Prosody issue?
videobridge? turnserver? It's very frustrating to debug, specially since I
had it working perfectly before. There's a lesson in all this, of course.
Don't upgrade any packages without being extra careful *slaps head*

I see these errors in the Niginx logs:

[error] 7972#0: *897 upstream timed out *(110: Connection timed out) while
reading response header from upstream*, client: 85.xx.xx.xx, server:
meet.domain.com, request: "POST /http-bind HTTP/1.1", upstream: "
http://127.0.0.1:5280/http-bind", host: "meet.domain.com", referrer: "
https://meet.domain.com/9ij7kzh3ss4qjjor"

Now I'm not sure why it's timing out.
I can see the connection starting in the prosody logs but indeed there
seems to be some problems, even though the connection comes in successfully
and even receives a reply from videobridge.

May 09 13:58:13 mod_bosh debug BOSH body open (sid:
a68bc480-5f14-4145-a3bf-c0d21c68e566)
May 09 13:58:13 mod_bosh debug BOSH stanza received: <iq id='1:sendIQ'
type='get' to='jitsi-videobridge.meet.85.xx.xx.xx'>

May 09 13:58:13 bosha68bc480-5f14-4145-a3bf-c0d21c68e566 debug Received[c2s]:
<iq id='1:sendIQ' type='get' to='jitsi-videobridge.meet.85.xx.xx.xx'>
May 09 13:58:13 mod_bosh debug Session a68bc480-5f14-4145-a3bf-c0d21c68e566
has 1 out of 1 requests open
May 09 13:58:13 mod_bosh debug and there are 0 things in the send_buffer:
May 09 13:58:13 mod_bosh debug Have nothing to say, so leaving request
unanswered for now

···

May 09 13:58:14 jcp2651590 debug Received[component]: <iq id='1:sendIQ' type='result' to='4885a5c5-d51f-41e3-be7c-fe2ca061e33e@meet.85.xx.xx.xx/02d683aa-a37e-483c-8d6c-d6685e94efb4' from='jitsi-videobridge.meet.85.xx.xx.xx'>
May 09 13:58:14 mod_bosh debug We have an open request, so sending on that
May 09 13:58:14 mod_bosh debug Request destroyed: table: 0x278fd00
May 09 13:58:14 bosha68bc480-5f14-4145-a3bf-c0d21c68e566 debug BOSH session
marked as inactive (for 60s)
May 09 13:58:14 socket debug try to close client connection with id:
278b520
May 09 13:58:14 socket debug closing delayed until writebuffer is empty

May 09 13:59:10 socket debug accepted incoming client connection from: ::1
60861 to 5280
May 09 13:59:10 mod_bosh debug Handling new request table: 0x27e88d0: <body
rid='328938923' xmlns='http://jabber.org/protocol/httpbind'
sid='7d3b9bad-cd36-45d0-bde8-02caa9a13276'/>
----------
May 09 13:59:10 mod_bosh debug BOSH body open (sid:
7d3b9bad-cd36-45d0-bde8-02caa9a13276)
May 09 13:59:10 mod_bosh debug Session 7d3b9bad-cd36-45d0-bde8-02caa9a13276
has 1 out of 1 requests open
May 09 13:59:10 mod_bosh debug and there are 0 things in the send_buffer:
May 09 13:59:10 mod_bosh debug Have nothing to say, so leaving request
unanswered for now
May 09 13:59:11 mod_bosh debug table: 0x27d3d00 was soon to timeout (at
1399643954, now 1399643954), sending empty response
May 09 13:59:11 mod_bosh debug We have an open request, so sending on that
May 09 13:59:11 mod_bosh debug Request destroyed: table: 0x27d3d00
May 09 13:59:11 bosha68bc480-5f14-4145-a3bf-c0d21c68e566 debug BOSH session
marked as inactive (for 60s)
May 09 13:59:11 socket debug try to close client connection with id:
27c08b0
May 09 13:59:11 socket debug closing delayed until writebuffer is empty
May 09 13:59:11 socket debug closing client after writing
May 09 13:59:11 socket debug closing client with id: 27c08b0 client to
close
*May 09 13:59:14 socket debug connection failed in read event: closed *
May 09 13:59:14 socket debug closing client with id: 2793f10 closed

Any help or clues greatly appreciated.

Thanks,

Peter

On Fri, May 9, 2014 at 7:10 AM, Lyubomir Marinov <lyubomir.marinov@jitsi.org > wrote:

It may be of use to provide the version/build number of the related
software packages such as Jitsi Videobridge.

john1 · 2014-05-09 20:34:06 UTC

Hi Peter , Is it possible your upgrading process overwrite your existing prosody configuration file ? I had a similar problem solved commenting these 2 lines

c2s_require_encryption = true, and s2s_secure_auth = false

Hope it helps you. Regards - John

···

On 09/05/2014 16:20, Peter Villeneuve wrote:

Hi Lyubomir,

Thanks for helping out.

I'm using jitsi-videobridge-linux-x64-99 and latest jitmeet from git on my debian 7 x64 server.
I've tried downgrading Prosody to 0.9.4 but the problem remains. I also tried Prosody trunk nightly build 482 (2014-04-26, 44de98f516f5) and again videobridge hangs (when negotiating ICE)? Interestingly, when I try to grab the jitmeet logs Chrome and Chromium both crash (out of memory). All I could do was grab the last stanzas from the videobridge console. See pastebin of the one it hangs on forever here <http://pastebin.com/iHuvmDNh> (presumably this leads to the bosh timeout we see in the nginx logs)

I'm having a tough time isolating the problem. Is it a Prosody issue? videobridge? turnserver? It's very frustrating to debug, specially since I had it working perfectly before. There's a lesson in all this, of course. Don't upgrade any packages without being extra careful *slaps head*

I see these errors in the Niginx logs:

[error] 7972#0: *897 upstream timed out *(110: Connection timed out) while reading response header from upstream*, client: 85.xx.xx.xx, server: meet.domain.com <http://meet.domain.com>, request: "POST /http-bind HTTP/1.1", upstream: "http://127.0.0.1:5280/http-bind", host: "meet.domain.com <http://meet.domain.com>", referrer: "https://meet.domain.com/9ij7kzh3ss4qjjor"

Now I'm not sure why it's timing out.
I can see the connection starting in the prosody logs but indeed there seems to be some problems, even though the connection comes in successfully and even receives a reply from videobridge.

May 09 13:58:13 mod_boshdebugBOSH body open (sid: a68bc480-5f14-4145-a3bf-c0d21c68e566)
May 09 13:58:13 mod_boshdebugBOSH stanza received: <iq id='1:sendIQ' type='get' to='jitsi-videobridge.meet.85.xx.xx.xx'>

May 09 13:58:13 bosha68bc480-5f14-4145-a3bf-c0d21c68e566debugReceived[c2s]: <iq id='1:sendIQ' type='get' to='jitsi-videobridge.meet.85.xx.xx.xx'>
May 09 13:58:13 mod_boshdebugSession a68bc480-5f14-4145-a3bf-c0d21c68e566 has 1 out of 1 requests open
May 09 13:58:13 mod_boshdebugand there are 0 things in the send_buffer:
May 09 13:58:13 mod_boshdebugHave nothing to say, so leaving request unanswered for now
May 09 13:58:14 jcp2651590debugReceived[component]: <iq id='1:sendIQ' type='result' to='4885a5c5-d51f-41e3-be7c-fe2ca061e33e@meet.85.xx.xx.xx/02d683aa-a37e-483c-8d6c-d6685e94efb4' from='jitsi-videobridge.meet.85.xx.xx.xx'>
May 09 13:58:14 mod_boshdebugWe have an open request, so sending on that
May 09 13:58:14 mod_boshdebugRequest destroyed: table: 0x278fd00
May 09 13:58:14 bosha68bc480-5f14-4145-a3bf-c0d21c68e566debugBOSH session marked as inactive (for 60s)
May 09 13:58:14 socketdebugtry to close client connection with id: 278b520
May 09 13:58:14 socketdebugclosing delayed until writebuffer is empty

May 09 13:59:10 socketdebugaccepted incoming client connection from: ::1 60861 to 5280
May 09 13:59:10 mod_boshdebugHandling new request table: 0x27e88d0: <body rid='328938923' xmlns='http://jabber.org/protocol/httpbind' sid='7d3b9bad-cd36-45d0-bde8-02caa9a13276'/>
----------
May 09 13:59:10 mod_boshdebugBOSH body open (sid: 7d3b9bad-cd36-45d0-bde8-02caa9a13276)
May 09 13:59:10 mod_boshdebugSession 7d3b9bad-cd36-45d0-bde8-02caa9a13276 has 1 out of 1 requests open
May 09 13:59:10 mod_boshdebugand there are 0 things in the send_buffer:
May 09 13:59:10 mod_boshdebugHave nothing to say, so leaving request unanswered for now
May 09 13:59:11 mod_boshdebugtable: 0x27d3d00 was soon to timeout (at 1399643954, now 1399643954), sending empty response
May 09 13:59:11 mod_boshdebugWe have an open request, so sending on that
May 09 13:59:11 mod_boshdebugRequest destroyed: table: 0x27d3d00
May 09 13:59:11 bosha68bc480-5f14-4145-a3bf-c0d21c68e566debugBOSH session marked as inactive (for 60s)
May 09 13:59:11 socketdebugtry to close client connection with id: 27c08b0
May 09 13:59:11 socketdebugclosing delayed until writebuffer is empty
May 09 13:59:11 socketdebugclosing client after writing
May 09 13:59:11 socketdebugclosing client with id: 27c08b0 client to close
*May 09 13:59:14 socketdebugconnection failed in read event: closed *
May 09 13:59:14 socketdebugclosing client with id: 2793f10 closed

Any help or clues greatly appreciated.

Thanks,

Peter

On Fri, May 9, 2014 at 7:10 AM, Lyubomir Marinov > <lyubomir.marinov@jitsi.org <mailto:lyubomir.marinov@jitsi.org>> wrote:

It may be of use to provide the version/build number of the
related software packages such as Jitsi Videobridge.

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev

Matthew_Wild · 2014-05-09 22:46:37 UTC

Do you have a timeout set in your nginx config? Make sure it is set to
something greater than 60s (say... 65s or something).

Regards,
Matthew

···

On 9 May 2014 15:20, Peter Villeneuve <petervnv1@gmail.com> wrote:

Hi Lyubomir,

Thanks for helping out.

I'm using jitsi-videobridge-linux-x64-99 and latest jitmeet from git on my
debian 7 x64 server.
I've tried downgrading Prosody to 0.9.4 but the problem remains. I also
tried Prosody trunk nightly build 482 (2014-04-26, 44de98f516f5) and again
videobridge hangs (when negotiating ICE)? Interestingly, when I try to grab
the jitmeet logs Chrome and Chromium both crash (out of memory). All I could
do was grab the last stanzas from the videobridge console. See pastebin of
the one it hangs on forever here (presumably this leads to the bosh timeout
we see in the nginx logs)

I'm having a tough time isolating the problem. Is it a Prosody issue?
videobridge? turnserver? It's very frustrating to debug, specially since I
had it working perfectly before. There's a lesson in all this, of course.
Don't upgrade any packages without being extra careful *slaps head*

I see these errors in the Niginx logs:

[error] 7972#0: *897 upstream timed out (110: Connection timed out) while
reading response header from upstream, client: 85.xx.xx.xx, server:
meet.domain.com, request: "POST /http-bind HTTP/1.1", upstream:
"http://127.0.0.1:5280/http-bind", host: "meet.domain.com", referrer:
"https://meet.domain.com/9ij7kzh3ss4qjjor"

Now I'm not sure why it's timing out.

Peter_Villeneuve · 2014-05-10 00:21:00 UTC

Thanks guys. I've tried all that.
I think I narrowed it down to some issue with jitsi videobridge and ice
negotiation.
Now I hope some dev will confirm or dispell this. I posted some logs in
another post.

···

On Fri, May 9, 2014 at 11:46 PM, Matthew Wild <mwild1@gmail.com> wrote:

On 9 May 2014 15:20, Peter Villeneuve <petervnv1@gmail.com> wrote:
> Hi Lyubomir,
>
> Thanks for helping out.
>
> I'm using jitsi-videobridge-linux-x64-99 and latest jitmeet from git on
my
> debian 7 x64 server.
> I've tried downgrading Prosody to 0.9.4 but the problem remains. I also
> tried Prosody trunk nightly build 482 (2014-04-26, 44de98f516f5) and
again
> videobridge hangs (when negotiating ICE)? Interestingly, when I try to
grab
> the jitmeet logs Chrome and Chromium both crash (out of memory). All I
could
> do was grab the last stanzas from the videobridge console. See pastebin
of
> the one it hangs on forever here (presumably this leads to the bosh
timeout
> we see in the nginx logs)
>
> I'm having a tough time isolating the problem. Is it a Prosody issue?
> videobridge? turnserver? It's very frustrating to debug, specially since
I
> had it working perfectly before. There's a lesson in all this, of course.
> Don't upgrade any packages without being extra careful *slaps head*
>
> I see these errors in the Niginx logs:
>
> [error] 7972#0: *897 upstream timed out (110: Connection timed out) while
> reading response header from upstream, client: 85.xx.xx.xx, server:
> meet.domain.com, request: "POST /http-bind HTTP/1.1", upstream:
> "http://127.0.0.1:5280/http-bind", host: "meet.domain.com", referrer:
> "https://meet.domain.com/9ij7kzh3ss4qjjor"
>
> Now I'm not sure why it's timing out.

Do you have a timeout set in your nginx config? Make sure it is set to
something greater than 60s (say... 65s or something).

Regards,
Matthew

_______________________________________________
dev mailing list
dev@jitsi.org
Unsubscribe instructions and other list options:
http://lists.jitsi.org/mailman/listinfo/dev