Log Based Monitoring Approach

Hi everyone, first of all thanks for the great work on the Jitsi products!

After getting Jitsi-Meet running on one of our servers, the next obvious challenge was to get some sort of basic monitoring up. Looking through these forums I came across a few topics where people try to achieve the same. It was suggested multiple times to use the muc_size prosody module for this. However, because I’m still at prosody 0.10 and upgrading to 0.11 broke my install until I downgraded again I decided to take a different approach: using the logs.

My question is therefore: for basic monitoring is there something wrong with simply using the nginx logs and the jicofo.log located at “/var/log/nginx/access.log” & “/var/log/jitsi/jicofo.log”? I did some tests with grep searching for “joined|is leaving|@meet.[enteryourdomain]” and it gives me room names, people joining and leaving with a unique id, basically everything I’d need to know how many rooms are open and how many people are joined.

Additional information:

  • I have installed Jitsi-Meet a few days ago through the quick-installer on an Ubuntu 18.04 server and I’m using an nginx webserver.
  • I’m using authentication setup through https://github.com/jitsi/jicofo/README.md “secure domain” section

Edit: I’ll post progress on the bash script soon. Using only the /var/log/jitsi/jicofo.log I think it should be possible to keep track of at least roomcount, roomnames, membercount/room. When using authentication I think it should additionally be possible to keep track of which user authenticated each room.

6 Likes

Does your script display the usernames that people provide when they join? From my inspection of the jicofo.log file, I just see a random identifier string. It would be nice to have both, if that is possible. Like you, I also would be interested to know who successfully used credentials to start/moderate a meeting, or when somebody tries and fails.

So first of all, we have to be specific about what we call “usernames” in this context. When a user gives up a name that is visible to others in the room, I don’t think this pops up anywhere on the jicofo log radar. However, if a user logs in as a host for the first time to authenticate a room there is some predictable behavior, I have managed to relate room name to host that way. The problem arises when the host then opens another room. Because the host is already authenticated, there is no clear single line log message relating the host to the room. There is a predictable pattern when analyzing multiple lines, but obviously that might create reliability problems when there is a lot of simultaneous activity on the server (in other words I’m not sure we can expect the log to be synchronous).

What we perhaps could do is use the log concat script to pull in additional logs (for example the prosody log) and see if we can get more info that way. One problem with that is that from my early inspection the prosody logs use different identifiers than the jicofo logs.

EDIT:
Upon further study of the jicofo log file I think there is actually an easy method of determining the host. It appears that when an authenticated host opens a room they will be assigned an “Authenticated jid” that belongs to a specific AuthSession (like admin@meet.jit.si). So we can link this Authenticated jid to the host’s login. When the host then joins the room, it will say in the jicofo.log “roomname@conference.meet.jit.si/firstPartOfJID joined.” You then know if the host is in the room or not (and by extension if you set up your security that way, you know if the room has been opened by the host)

Why is there no file in my “/var/log/jitsi” directory?I use Docker-jitsi-meet.

This is not really related to this topic, but the answer to your question is the following:
Each docker container has their own file system. When you use the docker-jitsi-meet instal your installation is basically split into multiple containers that run simultaneously and communicate with each other. If you’re looking for the jicofo log, I think you have to “attach” to the jicofo container you’re running with docker and go to the /var/log/jitsi/jicofo.log within that container’s file system (you probably need root privilege).

I don’t find “jicofo.log” file in docker-image.And I have “sudo” privilege ,Still can’t find that file.Do I lack any configration?

I would suggest posting your question in a different thread as this is off topic. I’m not on a docker install so I can’t check for you.

Another quick update with regard to the authentication detection part of the tool. I got it working for all rooms that have roomnames containing no numbers or special characters. Getting it to work with special characters is totally going to work in the future, it just requires some more regex tweaking.

So I now have working:

  • nr of rooms open
  • roomnames of rooms open
  • nr of participants per room
  • host that authenticated room
    Jitsihawk_0_2

The tool runs surprisingly fast (especially considering it is entirely written using Bash). Obviously in the current implementation performance would scale with the size of your log history. Further optimization could be made there if you know that rooms won’t stay open for longer than a few days and you trim your log history accordingly.

4 Likes

Are you willing to share your script?

Looking great. Are you publishing this as opensource or is it only for private use?

Thanks

Wow, that is exactly what I’m looking for. It would be nice to get that script.

Excellent! That’s what I need too. Hope you share it soon :smile:

I can help. to develop this scritp by testing it?

I made a Gitlab repository for the project, I’ll continue development from there. The whole thing is released under an MIT License and if anyone else wishes to contribute with suggestions it would be appreciated and you’d help everyone out.
There are currently the following limitations with the script:

  • only roomnames of format: roomname, roomname-roomname, roomname_roomname are supported. I’ll probably add support for different formatting schemes soon, but for me it had no priority at the time. I’ll also have a look if I can add something in to make editing the regular expressions a little bit easier, so it is easier for others to contribute in the future.
  • currently only the name of the last host entered will be shown, in a future update I’m planning to show an array of hosts instead of only one host names and then this problem should be solved. For now, realize that the name behind the roomname is simply the last host that entered, not necessarily the person that actually authenticated the room.
6 Likes

Thanks @Magic_V !

Do you prefer our feebacks here or on GitLab? My 1st try shows rooms no longer active…

@combnum Thanks for wanting to help out, I think for now the Jitsi forums are a better place because that is where most of the community is at.

With regard to your issue, there are a couple things we could check:

  • The detection part: depends on formatting of the room names in question (like what do the roomNames of the rooms that appear (but shouldn’t) look like). To detect closed rooms the script currently scans the log file for:
    "Disposed conference for room: .*@conference.$DOMAIN"
    and then greps out:
    "\w*\@conference\|\w*\-\w*\@conference\|\w*\_\w*\@conference"
    which means if your room isn’t formatted as
    "word" || "word-word" || "word_word"
    we might get unexpected results.

  • perhaps we are on different versions (not sure if that matters yet)
    My version information:
    OS: Ubuntu Server 18.04 LTS Installed through apt-get.
    running:
    dpkg -l | grep "jitsi-meet\|jicofo"
    gives:
    jicofo 1.0-508-1
    jitsi-meet 1.0.4101-1
    jitsi-meet-prosody 1.0.3729-1
    jitsi-meet-web 1.0.3729-1
    jitsi-meet-web-config 1.0.3729-1

@Magic_V

The room names are all formatted as “word”.

That’s on an Ubuntu 18.04 server too, and we’re using unstable versions of jitsi-meet:
jicofo 1.0-549-1
jitsi-meet 2.0.4473-1
jitsi-meet-prosody 1.0.4027-1
jitsi-meet-web 1.0.4027-1
jitsi-meet-web-config 1.0.4027-1

Thank you very much @Magic_V for this cool approach!

In case anybody is interested I forked it and made some changes to work with a docker-jitsi-meet setup.

@MarcL I have docker setup on my instance. I cloned your repo but it’s showing me 0 rooms but I have created 1 room for testing. Let me know

@MarcL Nice to know you got it working with docker!
I’d be interested in adding in a flag on the main script for supporting a docker install. I’ll wait for some feedback of some other docker users.

@combnum Without a jicofo.log.total file it is a bit hard to assess from my side what could be the issue. If you’re willing to (privately) share some log data I’d be willing to have a deeper look what could be going on.