Enabled the jvb to use websocket. log report nonexistent endpoint

this environment is set up on docker and was working well on the udp/tcp mode.

now I am setting up a new environment on a HTTP only network. udp/tcp is not allowed. I am using this doc to enable websocket for jitsi

at first the meeting is good. after few seconds. the web client receive below error:

[modules/RTC/BridgeChannel.js] <WebSocket.e.onopen>: websocket channel opened
[modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage>: Channel new last-n event: {colibriClass: “LastNEndpointsChangeEvent”, lastNEndpoints: Array(0), endpointsEnteringLastN: Array(0), conferenceEndpoints: Array(0)}
[modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage>: SelectedUpdateEvent isSelected? true
[modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage>: Channel new last-n event: [“cfb5b4fc”] {colibriClass: “LastNEndpointsChangeEvent”, lastNEndpoints: Array(1), endpointsEnteringLastN: Array(1), conferenceEndpoints: Array(1)}
[modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage>: Endpoint connection status changed: ed68aadb active ? false
[modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage>: Endpoint connection status changed: cfb5b4fc active ? false
[modules/RTC/BridgeChannel.js] <WebSocket.e.onmessage>: Channel new last-n event: {colibriClass: “LastNEndpointsChangeEvent”, lastNEndpoints: Array(0), endpointsEnteringLastN: Array(0), conferenceEndpoints: Array(0)}
[modules/RTC/BridgeChannel.js] <WebSocket.e.onclose>: Channel closed by server

Channel closed: 1006
Bridge Channel send: no opened channel.
WebSocket connection to ‘wss://sandbox.example.com/colibri-ws/jvb1/5b4deb591b493cd4/ed68aadb?pwd=poqqopjvba1lgqbmt71n305di’ failed: Error during WebSocket handshake: Unexpected response code: 403

on jicofo, the video stream connections seems stopped.

Jicofo 2019-11-01 09:45:42.136 INFO: [38] org.jitsi.jicofo.Bridge.log() Video stream count for: jvbsub1.meet.jitsi: 4
Jicofo 2019-11-01 09:46:52.204 INFO: [38] org.jitsi.jicofo.Bridge.log() Video stream count for: jvbsub1.meet.jitsi: 0

jvb log:

for user ed68aadb, the jvb can receive message from it at the beginning:
JVB 2019-11-01 09:45:48.073 FINE: [29] org.jitsi.videobridge.rest.ColibriWebSocket.log() Received text from ed68aadb: {“colibriClass”:“EndpointMessage”,“msgPayload”:{“type”:“stats”,“values”:{“bitrate”:{“upload”:2067,“download”:2055,“audio”:{“upload”:34,“download”:33},“video”:{“upload”:2033,“download”:2022}},“packetLoss”:{“total”:0,“download”:0,“upload”:0},“connectionQuality”:100,“avgAudioLevels”:0.003131199072237312}},“to”:""}

few seconds after:

jvb throw error: Received request for a nonexistent endpoint:
I assume this means the JVB has the endpoints of a conference;

JVB 2019-11-01 09:45:48.123 FINE: [34] org.jitsi.videobridge.EndpointConnectionStatus.monitorEndpointActivity().286 ed68aadb not ready for activity checks yet
JVB 2019-11-01 09:45:48.623 FINE: [34] org.jitsi.videobridge.EndpointConnectionStatus.monitorEndpointActivity().274 ed68aadb is having trouble establishing the connection and will be marked as inactive

is it related to this error ?
10.244.0.163 is the web server
10.244.0.167 is the jvb

JVB 2019-11-01 09:46:47.139 FINE: [25] org.eclipse.jetty.io.AbstractConnection.onClose() onClose WebSocketServerConnection@7107dbf7::SocketChannelEndPoint@75c1d279{/10.244.0.163:56536<->/10.244.0.167:30090,CLOSED,fill=-,flush=-,to=44/60000}{io=0/0,kio=-1,kro=-1}->WebSocketServerConnection@7107dbf7[s=ConnectionState@b32657a[DISCONNECTED],f=org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$Flusher@339a0740[FAILED][queueSize=0,aggregateSize=0,terminated=java.io.EOFException: Disconnected],g=Generator[SERVER,validating,+rsv1],p=Parser@1a61ba68[ExtensionStack,s=START,c=0,len=38,f=null]]
JVB 2019-11-01 09:46:47.139 FINE: [25] org.eclipse.jetty.websocket.common.WebSocketSession.onClosed() [SERVER] WebSocketSession.onSessionClosed()
JVB 2019-11-01 09:46:47.143 FINE: [25] org.eclipse.jetty.io.ManagedSelector.safeInterestOps()
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at org.eclipse.jetty.io.ManagedSelector.safeInterestOps(ManagedSelector.java:308)
at org.eclipse.jetty.io.ChannelEndPoint.toEndPointString(ChannelEndPoint.java:432)
at org.eclipse.jetty.io.AbstractEndPoint.toString(AbstractEndPoint.java:447)
at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
at java.util.Formatter.format(Formatter.java:2520)
at java.util.Formatter.format(Formatter.java:2455)
at java.lang.String.format(String.java:2940)
at org.eclipse.jetty.io.AbstractConnection.toString(AbstractConnection.java:290)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.eclipse.jetty.websocket.common.WebSocketSession.toString(WebSocketSession.java:580)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:680)
at org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:224)
at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.setStopping(AbstractLifeCycle.java:194)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:88)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149)
at org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:590)
at org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:568)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.onSessionClosed(WebSocketServerFactory.java:531)
at org.eclipse.jetty.websocket.common.WebSocketSession.lambda$onClosed$0(WebSocketSession.java:396)
at org.eclipse.jetty.websocket.common.WebSocketSession.notifySessionListeners(WebSocketSession.java:564)
at org.eclipse.jetty.websocket.common.WebSocketSession.onClosed(WebSocketSession.java:396)
at org.eclipse.jetty.io.AbstractConnection.onClosed(AbstractConnection.java:231)
at org.eclipse.jetty.io.AbstractConnection.onClose(AbstractConnection.java:224)
at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)
at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:958)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
JVB 2019-11-01 09:46:47.143 FINE: [25] org.eclipse.jetty.io.ManagedSelector.safeReadyOps()
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
at org.eclipse.jetty.io.ManagedSelector.safeReadyOps(ManagedSelector.java:294)
at org.eclipse.jetty.io.ChannelEndPoint.toEndPointString(ChannelEndPoint.java:433)
at org.eclipse.jetty.io.AbstractEndPoint.toString(AbstractEndPoint.java:447)
at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
at java.util.Formatter.format(Formatter.java:2520)
at java.util.Formatter.format(Formatter.java:2455)
at java.lang.String.format(String.java:2940)
at org.eclipse.jetty.io.AbstractConnection.toString(AbstractConnection.java:290)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.eclipse.jetty.websocket.common.WebSocketSession.toString(WebSocketSession.java:580)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:680)
at org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:224)
at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.setStopping(AbstractLifeCycle.java:194)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:88)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149)
at org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:590)
at org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:568)
at org.eclipse.jetty.websocket.server.WebSocketServerFactory.onSessionClosed(WebSocketServerFactory.java:531)
at org.eclipse.jetty.websocket.common.WebSocketSession.lambda$onClosed$0(WebSocketSession.java:396)
at org.eclipse.jetty.websocket.common.WebSocketSession.notifySessionListeners(WebSocketSession.java:564)
at org.eclipse.jetty.websocket.common.WebSocketSession.onClosed(WebSocketSession.java:396)
at org.eclipse.jetty.io.AbstractConnection.onClosed(AbstractConnection.java:231)
at org.eclipse.jetty.io.AbstractConnection.onClose(AbstractConnection.java:224)
at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)
at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:958)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)

I believe below code is used to monitor endpoint health:

Endpoint endpoint = (Endpoint) abstractEndpoint;
String endpointId = endpoint.getID();

    // Go over all RTP channels to get the latest timestamp
    List<RtpChannel> rtpChannels = endpoint.getChannels();
    long lastActivity
        = rtpChannels.stream()
            .mapToLong(RtpChannel::getLastTransportActivityTime)
            .max().orElse(0);
    long mostRecentChannelCreated
        = rtpChannels.stream()
            .mapToLong(RtpChannel::getCreationTimestamp)
            .max().orElse(0);

    // Also check SctpConnection
    SctpConnection sctpConnection = endpoint.getSctpConnection();
    if (sctpConnection != null)
    {
        long lastSctpActivity
            = sctpConnection.getLastTransportActivityTime();
        if (lastSctpActivity > lastActivity)
        {
            lastActivity = lastSctpActivity;
        }
        long creationTimestamp = sctpConnection.getCreationTimestamp();
        if (creationTimestamp > mostRecentChannelCreated)
        {
            mostRecentChannelCreated = creationTimestamp;
        }
    }

I don’t know why it doesn’t work in websocket mode.
I have print out the value:

the endpoint has 2rtp channels
the endpoint has sctpconnection null
lastActivity 0, mostRecentChannelCreated is 1572604428109, firstTransferTimeout is 15000, current is 1572604506673