Problem with joining rooms

Hello!

I installed Jitsi-meet on one server and jitsi-videobridge2 on 3 additional servers.

I have around 7 rooms of 6 people at the moment. I would like to support way more rooms on this amount of servers but I already have some problems with this small number of rooms.

At first, everything goes well. However, at some point it becomes very slow to join a conversation. The servers don’t seem to be overloaded at all.

I see this kind of warnings in jicofo logs:

Jicofo 2020-05-03 17:50:13.314 WARNING: [96] org.jitsi.xmpp.component.ComponentBase.log() PROCESSING TIME LIMIT EXCEEDED - it took 2626ms to process: <iq to="focus. ...

Then, at some point it becomes impossible to join a room. I start seeing “OperationFailedException: Failed to join the room” errors in jicofo logs (full log message below). I don’t see anything strange in prosody logs and in jvb logs. If I add one more server with jitsi-videobridge2, everything starts to magically work again.

Full error message from jicofo logs:

    Jicofo 2020-05-03 17:22:18.392 INFO: [92] org.jitsi.jicofo.FocusManager.log() Exception while trying to start the conference
net.java.sip.communicator.service.protocol.OperationFailedException: Failed to join the room
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:298)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:209)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.joinTheRoom(JitsiMeetConferenceImpl.java:578)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.start(JitsiMeetConferenceImpl.java:399)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:446)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:395)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:370)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:421)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSetImpl(FocusComponent.java:259)
	at org.jitsi.xmpp.component.ComponentBase.handleIQSet(ComponentBase.java:362)
	at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
	at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
	at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
	at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
	at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from volt-purple-palace--table-2@conference.table.bar.live/focus: XMPPError: forbidden - cancel
	at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:132)
	at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:263)
	at org.jivesoftware.smackx.muc.MultiUserChat.enter(MultiUserChat.java:355)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:498)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:444)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:240)
	... 17 more
Jicofo 2020-05-03 17:22:18.393 SEVERE: [92] org.jitsi.jicofo.xmpp.FocusComponent.handleIQSetImpl().324 net.java.sip.communicator.service.protocol.OperationFailedException: Failed to join the room
net.java.sip.communicator.service.protocol.OperationFailedException: Failed to join the room
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:298)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.join(ChatRoomImpl.java:209)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.joinTheRoom(JitsiMeetConferenceImpl.java:578)
	at org.jitsi.jicofo.JitsiMeetConferenceImpl.start(JitsiMeetConferenceImpl.java:399)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:446)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:395)
	at org.jitsi.jicofo.FocusManager.conferenceRequest(FocusManager.java:370)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleConferenceIq(FocusComponent.java:421)
	at org.jitsi.jicofo.xmpp.FocusComponent.handleIQSetImpl(FocusComponent.java:259)
	at org.jitsi.xmpp.component.ComponentBase.handleIQSet(ComponentBase.java:362)
	at org.xmpp.component.AbstractComponent.processIQRequest(AbstractComponent.java:515)
	at org.xmpp.component.AbstractComponent.processIQ(AbstractComponent.java:289)
	at org.xmpp.component.AbstractComponent.processQueuedPacket(AbstractComponent.java:239)
	at org.xmpp.component.AbstractComponent.access$100(AbstractComponent.java:81)
	at org.xmpp.component.AbstractComponent$PacketProcessor.run(AbstractComponent.java:1051)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.jivesoftware.smack.XMPPException$XMPPErrorException: XMPP error reply received from volt-purple-palace--table-2@conference.table.bar.live/focus: XMPPError: forbidden - cancel
	at org.jivesoftware.smack.XMPPException$XMPPErrorException.ifHasErrorThenThrow(XMPPException.java:132)
	at org.jivesoftware.smack.StanzaCollector.nextResultOrThrow(StanzaCollector.java:263)
	at org.jivesoftware.smackx.muc.MultiUserChat.enter(MultiUserChat.java:355)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:498)
	at org.jivesoftware.smackx.muc.MultiUserChat.createOrJoin(MultiUserChat.java:444)
	at org.jitsi.impl.protocol.xmpp.ChatRoomImpl.joinAs(ChatRoomImpl.java:240)
	... 17 more

My hardware:

Server for jitsi-meet (jicofo + prosody): 3 cpu cores, 4 GB memory, 300 Mbps bandwidth
Each server for jitsi-videobridge2: 4 cpu cores, 8 GB ram, 400 Mbps bandwidth

Maximum bandwidth usage among servers with jitsi-videobridge2 was 37 Mpbs.
Jicofo server load at the moment exceptions started to appear in the log:

At the moment I don’t know where to start looking. Could you please help me?

It turned out that this exception appears when the XMPP server refused to create a room. Adding a new video-bridge2 server must not have had any effect.
I have to investigate the reason why XMPP suddenly refused to create a room but it’s a different problem.

There is a recent github issue about this exception: