Firefox broken - Websocket AND webrtc : I don't know why

Hi,

I recent deployment from meet-docker based on latest stable release, I figure out that Jitsi was not working on Firefox, but working fine on Chromium like browser.
After research I find that about:webrtc in firefox show me connections. it try to connect on udp/10000 while an ICE server (coturn) is setup (it knows it)
In chrome I also see some informations in chrome://webrtc-internal, but nothing regarding ICE.

Audio Constraints
{echoCancellation: {ideal: true}, deviceId: {ideal:    ["896a2dec11004b53d1eeaaa517cc5b40eba82ee365ecaeb927b55a34bdef7064"]}, autoGainControl:     {ideal: true}, noiseSuppression: {ideal: true}}
Video Constraints
{width: {ideal: 1280}, height: {ideal: 720}, facingMode: {ideal: ["user"]}}

On prosody/web/jicofo and jvb I didn’t found any configuration for webrtc. Websocket is activated everywhere and working fine except on Firefox.

If needed I can share some configurations file, just let me know which one.

Check your jicofo logs.

I found some strange lines but no errors. It looks like the session is expiring, but I don’t figure out why this appen and where to look

  2021-06-02T08:12:20.170Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.170Z INFO: Selected bridge Bridge[jid=jvbbrewery@internal-muc.jitsi-dev.region.meet/ip-1.dev, relayId=1.1.1.1:4096, region=region1, stress=0.03] with stress=0.0 for participantRegion=null
2021-06-02T08:12:20.171Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.171Z INFO: Added participant id= edf8ed13, bridge=jvbbrewery@internal-muc.jitsi-dev.region.meet/ip-1.dev
2021-06-02T08:12:20.172Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.172Z INFO: Doing feature discovery for RoomTest@muc.jitsi-dev.region.meet/edf8ed13
2021-06-02T08:12:20.172Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.172Z INFO: Successfully discovered features for RoomTest@muc.jitsi-dev.region.meet/edf8ed13 in 0
2021-06-02T08:12:20.172Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.172Z INFO: Using jvbbrewery@internal-muc.jitsi-dev.region.meet/ip-1.dev to allocate channels for: Participant[RoomTest@muc.jitsi-dev.region.meet/edf8ed13]@1203741502
2021-06-02T08:12:20.259Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.259Z INFO: Sending session-initiate to: RoomTest@muc.jitsi-dev.region.meet/edf8ed13
2021-06-02T08:12:20.283Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.283Z INFO: Received session-terminate from Participant[RoomTest@muc.jitsi-dev.region.meet/bf38d750]@1523647217, bridge session: BridgeSession[id=130563_cb1295, bridge=Bridge[jid=jvbbrewery@internal-muc.jitsi-dev.region.meet/ip-1.dev, relayId=1.1.1.1:4096, region=region1, stress=0.04]]@1234570490, restart: true
2021-06-02T08:12:20.283Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.283Z INFO: Terminating bf38d750, reason: null, send session-terminate: false
2021-06-02T08:12:20.283Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.283Z INFO: **Terminate session: RoomTest@muc.jitsi-dev.region.meet/bf38d750, reason: null, send terminate: false**
2021-06-02T08:12:20.284Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.284Z INFO: Removing sources from RoomTest@muc.jitsi-dev.region.meet/bf38d750: Sources{ video: [ssrc=2766091445 ] audio: [ssrc=924312656 ] }@2020974204
2021-06-02T08:12:20.284Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.284Z WARNING: Remove source: no jingle session for RoomTest@muc.jitsi-dev.region.meet/bf38d750
2021-06-02T08:12:20.284Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.284Z INFO: **Removed participant** bf38d750 removed=true
2021-06-02T08:12:20.284Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log
2021-06-02T08:12:20.284Z INFO: **Expiring channels for:** RoomTest@muc.jitsi-dev.region.meet/bf38d750 on: Bridge[jid=jvbbrewery@internal-muc.jitsi-dev.region.meet/ip-1.dev, relayId=1.1.1.1:4096, region=region1, stress=0.04]
2021-06-02T08:12:20.285Z Jun 02, 2021 10:12:20 AM org.jitsi.utils.logging2.LoggerImpl log

Jicofo conf :

/config/jicofo.conf
jicofo {
    
    authentication {
      enabled = true
      type = JWT
      login-url = "jitsi-dev.region.meet"  
    }
    
    // Configuration related to jitsi-videobridge
    bridge {
      # test value
      max-bridge-participants = "90"
      # dev only, be sur all jvb are used.
      selection-strategy = "RegionBasedBridgeSelectionStrategy" 
      brewery-jid = "jvbbrewery@internal-muc.jitsi-dev.region.meet"
    }
    // Configure the codecs and RTP extensions to be used in the offer sent to clients.
    codec {
      video { 
      }
    }
    conference {
    }    
    octo {
      enabled = true          
      id = "1"
    }
    rest {
      
      port = 8080
      
      
    }
    sctp {
      enabled = false
    }
    xmpp {
      client {
        enabled = true
        hostname = "xmpp.jitsi-dev.region.meet"
        domain = "auth.jitsi-dev.region.meet"
        username = "focus"
        password = "Hidden"
        conference-muc-jid = "muc.jitsi-dev.region.meet"
        client-proxy = "focus.jitsi-dev.region.meet"
        disable-certificate-verification = true
      }
    }

More info :
My prosody config file says :

turncredentials_secret = "HiddenPassword";
turncredentials_port = 443;
turncredentials_ttl = 86400;
turncredentials = {
  { type = "turns",
    host = "hiddenhost",
    port = 443,
    transport = "tcp"
  }

And on Firefox I find :
"2021-06-03T16:17:18.797Z create":"([{"iceServers":[{"urls":"turns:hiddenhost:443?transport=tcp","username":"1622823335","credential":"Ns/SDLFLQRANDOM="}],"bundlePolicy":"max-bundle"},{"optional":[{"rtcStatsSFUP2P":false}]}])"

So It look like username/password are autogenerated, I don’t understand where too.

What is exactly the problem you experience? That Firefox is not using the turnserver?

Jitsi is configured for WebSocket, with Turn server.
On chrome WebSocket works, but I also see WebRTC connections …
On Firefox no WebSocket connection, only WebRTC connections.
So I don’t understand why WebRTC connection is happening, and why/how turn is ‘magic’ setup with random username/password.

I don’t get websocket with turn? What does that mean?

For which websocket are you talking about? Is it the xmpp connection or the connection to the bridge?

How do you understand that is not working? Maybe share the errors you see?

These are short-lived credentials by spec and are generated by prosody using a shared secret and validated in the turn server using the same shared secret.

https://tools.ietf.org/html/draft-uberti-behave-turn-rest-00

By any chance are you using nginx to front your turn server and you are using ALPN to recognize the traffic (this is an old setup we used to have which breaks because of browsers not marking websocket traffic as web and it goes to the turn server, but we moved away from it a year and a half ago or more)?

Let me explain a little bit more my architecture.

Coturn dedidacted servers behind a loadbalancer. DNS name of the loadbalancer set un prosody configuration (like previously shared, ‘hiddenhost’), listening on tcp 443

Then Jitsi stack with dedicated JVB servers.
Jicofo/Prosody/Meet (web) on same node.

Meet is configured to handle websocket on nginx and js script.

So maybe I miss something but when websocket is enabled, webrtc connection should not happened ?
Regarding websocket in chrome I see :

wss://myjitsi.url/colibri-ws/JVB_IP/a95dafdd4/c59e738d?pwd=23p8fdf6s213tud8 => status 101 switching protocol
wss://jitsi_url/xmpp-websocket?room=my_room&token=eyJraWQiOiJvN... status 101 switching protocol

In webrtc chrome://webrtc-internals
https://jitsi_url/my_room, { iceServers: [turns:turn_url:443?transport=tcp], iceTransportPolicy: all, bundlePolicy: max-bundle, rtcpMuxPolicy: require, iceCandidatePoolSize: 0, sdpSemantics: "plan-b", extmapAllowMixed: true }, {advanced: [{googScreencastMinBitrate: {exact: 100}}, {googCpuOveruseDetection: {exact: true}}]}

So I see Websocket AND Webrtc connection in Chrome. I don’t know how both are actived, maybe it’s normal behaivour ? But I’m not sur.
In Chrome Audio/Video, but I assume that is over Websocket and not WebRTC (not enough information to be sur).

In Firefox same connection, websocket and webrtc, but not same status

wss://myjitsi.url/colibri-ws/JVB_IP/a95dafdd4/c59e738d?pwd=23p8fdf6s213tud8 => status 200 connected
wss://jitsi_url/xmpp-websocket?room=my_room&token=eyJraWQiOiJvN... status 200 connected

And in about:webrtc I doesn’t success to connect to any IP:10000 which is normal, port 10000 is not accessible, so that we use Coturn (which is not in the list of servers).
No audio/video on firefox.
In console.log

2021-06-07T09:56:51.047Z [JitsiMeetJS.js] <getGlobalOnErrorHandler>:  UnhandledError: RTCPeerConnection is gone (did you enter Offline mode?) Script: null Line: null Column: null StackTrace:  H/e.RTCPeerConnection.prototype.getStats@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:25:51401
A.prototype.getStats@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:17:95683
t@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:10:119999
setInterval handler*h.prototype.start@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:10:120315
b.prototype.startRemoteStats@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:67552
ae.prototype._acceptJvbIncomingCall@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:10:55456
ae.prototype.onIncomingCall@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:10:54239
a.prototype.emit@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:118807
onJingle@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:10:196801
run@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:26830
_dataRecv/<@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:35268
forEachChild@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:18492
_dataRecv@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:35117
_onMessage@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:64355
EventHandlerNonNull*_connect_cb_wrapper@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:61712
EventHandlerNonNull*_connect@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:60185
connect@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:29567
_interceptConnectArgs@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:25:7817
connect@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:161088
_connect@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:131185
connect@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:1:132266
c.prototype.connect@https://jitsi-url/libs/lib-jitsi-meet.min.js?v=4985:10:29871
e@https://jitsi-url/libs/app.bundle.min.js?v=4985:150:225737
m/<@https://jitsi-url/libs/app.bundle.min.js?v=4985:150:225949
m@https://jitsi-url/libs/app.bundle.min.js?v=4985:150:224957
f@https://jitsi-url/libs/app.bundle.min.js?v=4985:150:226143
$@https://jitsi-url/libs/app.bundle.min.js?v=4985:200:1295
createInitialLocalTracksAndConnect@https://jitsi-url/libs/app.bundle.min.js?v=4985:200:5959
init@https://jitsi-url/libs/app.bundle.min.js?v=4985:200:7758
async*h/</<@https://jitsi-url/libs/app.bundle.min.js?v=4985:150:292356
promise callback*h/<@https://jitsi-url/libs/app.bundle.min.js?v=4985:150:292332
a/</</<@https://jitsi-url/libs/app.bundle.min.js?v=4985:209:89687
@https://jitsi-url/libs/app.bundle.min.js?v=4985:658:29892
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:921766
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:919664
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:918506
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:918202
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:916904
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:916014
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:915149
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:913459
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:900220
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:896430
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:896191
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:894819
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:892330
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:890691
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:890509
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:889399
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:888863
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:886113
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:884005
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:883674
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:883349
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:883310
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:877724
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:876851
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:876556
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:873272
@https://jitsi-url/libs/app.bundle.min.js?v=4985:421:873163
Logger.js:154:22

2021-06-07T09:56:51.048Z [modules/statistics/RTPStatsCollector.js] <h.prototype.errorCallback>:  Get stats error DOMException: RTCPeerConnection is gone (did you enter Offline mode?) Logger.js:154:22
WebRTC: ICE failed, add a STUN server and see about:webrtc for more details
2021-06-07T09:56:58.348Z [modules/connectivity/IceFailedHandling.js] <_actOnIceFailed/<>:  session-terminate for ice restart - error: undefined Logger.js:154:22
2021-06-07T09:56:58.353Z [modules/RTC/BridgeChannel.js] <_handleChannel/e.onclose>:  Channel closed: 1006 Logger.js:154:22
Firefox ne peut établir de connexion avec le serveur à l’adresse wss://jitsi-url/colibri-ws/JVB_IP/63e0a0e45/c4c05558?pwd=7qb7omgk

I’m investigating last line.

When connections can’t reach JVB directly, I has to go trough the Coturn over tpc/443 or over nginx.
But I don’t full understand Websocket, and maybe Coturn is no more needed :
Client <=> wss <=> nginx proxy <=> JVB

The colibri-ws is a direct connection between client and the bridge that is used to receive stats from clients and messages like dominant speaker changed.

The xmpp-websocket is the signalling between client and prosody and that can also be bosh.

When clients cannot reach the jvb udp 10000, there is fallback to using the turnserver connection. We do not advice you in making all calls using TCP, that will definitely degrade quality of participants and should be used only as a fallback if udp is not available. For the turnserver the browser establishes a direct TCP connection to the turnserver, it does not go over websocket. You cannot proxy pass that connection as you do for bosh or websockets in nginx.
That is a stream from nginx point of view.
Here Setting up TURN · Jitsi Meet Handbook you can see an example how to pass that stream through nginx to turnserver based on domain name.

Hi, thanks for explanation, more clear.
I will debug my flaw to fully understand where it is blocked.