Host/IP address audit of Authentication failures?

Hi community.

First post… will try to go to the point, hope someone can help me.

I’m a Sys admin, and I’ve been tasked to install a jitsi-meet instance on a publicly reachable server, for public rooms to join, public content, but restricted room creation to certain people… So far, so good, no problem! I’ve followed the secured-domain advices to get that up and running… everybody is happy, but I’m woried…
The problem is that I’m concerned of not knowing how to track source IP address of authentication failures on atempts to create rooms by unauthorized people… by my experience, this is something that tends to happen soon or later.

I use Fail2ban software with incredible success to manage abuse/DOS/DDOS on web applications, i really love Fail2ban! … but I’m failing completely here. I see no source IP at jicofo.log, NGINX access.log seems to be acting just like a proxy to 127.0.0.1 … I’m lost … How do you manage this? how could I trace ‘ugly’ activity down to a source IP … or should I just forget about it?

Thank you very much in advance.
Best regards.

1 Like

Same issue here…

Ubntu Server 18.04 fresh install

tail -f /var/log/prosody/prosody.log

Apr 30 14:32:47 mod_bosh info New BOSH session, assigned it sid ‘ef58ba89-67d5-4a4e-9438-44cc53601151’
Apr 30 14:32:47 boshef58ba89-67d5-4a4e-9438-44cc53601151 info Failed authentication attempt (not-authorized) for user 1@myconf.local from IP: 127.0.0.1

root@webconf:~# tail -f /var/log/fail2ban.log
2020-04-30 14:30:24,637 fail2ban.jail [3044]: INFO Creating new jail ‘prosody’
2020-04-30 14:30:24,637 fail2ban.jail [3044]: INFO Jail ‘prosody’ uses pyinotify {}
2020-04-30 14:30:24,643 fail2ban.jail [3044]: INFO Initiated ‘pyinotify’ backend
2020-04-30 14:30:24,646 fail2ban.filter [3044]: INFO Added logfile: ‘/var/log/prosody/prosody.log’ (pos = 152276, hash = b116607a776494ff6842cc3a6b98c1d8b93dde9c)
2020-04-30 14:30:24,647 fail2ban.filter [3044]: INFO encoding: UTF-8
2020-04-30 14:30:24,647 fail2ban.filter [3044]: INFO maxRetry: 5
2020-04-30 14:30:24,648 fail2ban.filter [3044]: INFO findtime: 600
2020-04-30 14:30:24,648 fail2ban.actions [3044]: INFO banTime: 600
2020-04-30 14:30:24,651 fail2ban.jail [3044]: INFO Jail ‘sshd’ started
2020-04-30 14:30:24,653 fail2ban.jail [3044]: INFO Jail ‘prosody’ started

But not block after 5 wrong pass.

In a default Jitsi installation prosody is handling authentication and is accessed via bosh through a http reverse proxy (usually nginx). So the logfile to be passed to fail2ban is /var/log/prosody/prosody.log. Afaik, to log failed authentication attempts, one needs to enable the module log_auth, at least thats what the log posted above by @rribeirorj looks like. The problem is, that the original remote IP of the autheticating user is replaced by the reverse proxy’s IP which is 127.0.0.1 in this case. The first thing I’d try to solve this would be to set the trusted_proxies option for the bosh module as documented here..

@rribeirorj @Alejandro_Olivan_Alv having any luck with this? I tried adding trusted_proxies as defined here. However , Prosody still recieves the local host 127.0.0.1

Hi! good to know I’m not alone on this.

I think we need a way of getting the public IP address of the participating/involved hosts on the service in a log system, like in any internet service, without tweaks or hacks… Otherwise the project feels to me like (just the sys admin, not final user at all) not a truly
‘production ready’ project, although I’ve been said by users it works pretty well… so it is really close! I really hope this is fixed soon!!!

Thank you very much to all.
Best regards!

I just checked and apparently the bosh module differs between prosody 0.10. and 0.11. In prosody 0.10 we find the following code, which is missing completely in mod_bosh that comes with prosody 0.11:

local trusted_proxies = module:get_option_set("trusted_proxies", { "127.0.0.1", "::1" })._items;

local function get_ip_from_request(request)
	local ip = request.conn:ip();
	local forwarded_for = request.headers.x_forwarded_for;
	if forwarded_for then
		forwarded_for = forwarded_for..", "..ip;
		for forwarded_ip in forwarded_for:gmatch("[^%s,]+") do
			if not trusted_proxies[forwarded_ip] then
				ip = forwarded_ip;
			end
		end
	end
	return ip;
end

Inn addition, mod_bosh of prosody 0.10 assigns the correct external IP in Line 290

ip = get_ip_from_request(request);

All this code is not in mod_bosh as it comes with prosody 0.11 - even the option trusted_proxies does not exist.

I did not check what else changed and did not test this myself but I guess it should work to simply copy-paste the code posted above into your mod_bosh.lua to get the external ip instead of localhost.

EDIT: Found this commit that moves the code in question from mod_bosh to mod_http. So I am back to zero, not sure why it does not work as expected. May be your nginx is not configured to set the X-Forwarded-For header?

Hey!!

Thanks about your observation, but i have X-Forwarded in nginx…

BOSH

location = /http-bind {
    proxy_pass      http://localhost:5280/http-bind;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $http_host;
}

Some idea?

Hi… neither an expert at all on this jitsi internal/under-the-hood services nor an dedicated developer… but I see/wonder something about that code:

Considering that the usual/traditional header that I’ve always seen and worked with, added by proxies/web to the HTTP protocol header section, is named ‘X-Forwarded-For’ (and having as value the original IP address), then

If the code: request.headers.x_forwarded_for is a function/method call (x_forwarded_for being the function/method) that actually returns X-Forwarded-For header value… then it is OK … but, on the other hand, if it somehow refers to the header name directly, and the header at HTTP packet headers has to be named ‘x_forwarded_for’ instead of ‘X-Forwarded-For’ … then that wouldn’t work… it is the only thing I think may fail…

Anyways I got the feeling this is going to be fixed soon :slight_smile:

1 Like

I’m running Prosody 11.5 now. Even though the code was moved to mod_http, I was curious and added lines from the old .10 version (53 through 65 and 290) to mod_bosh, but still only logs the local host.

nginx has x-forwarded-for

location = /http-bind {
    proxy_pass      http://localhost:5280/http-bind;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $http_host;
}

Under /etc/prosody/conf.avail/my.domain.cfg.lua, I have :

cross_domain_bosh = false
consider_bosh_secure = true
trusted_proxies = { "127.0.0.1", "::1" }

At a loss , not sure what else to check

Has anybody tried to use the $proxy_add_x_forwarded_for var instead of $remote_address? This is supposed to append the current $remote_address to the received X-Forwarded-For header, while $remote_address just overwrites potentially existing addresses. This should only be required in cases where an additional reverse proxy sits in front of the jitsi server, so I am not sure if it helps…

The $proxy_add_x_forwarded_for is documente at https://nginx.org/en/docs/http/ngx_http_proxy_module.html#var_proxy_add_x_forwarded_for

I just tested that and also tried

proxy_set_header X-Forwarded-For "$http_x_forwarded_for, $realip_remote_addr";

Restarted nginx and prosody each try, still only seeing 127.0.0.1 in prosody.log :frowning:

May 22 14:26:58 mod_bosh        info    New BOSH session, assigned it sid '6d6cd7c0-a8fc-4bfc-9e0c-4d7a85744f50'
May 22 14:26:58 bosh6d6cd7c0-a8fc-4bfc-9e0c-4d7a85744f50        info    Failed authentication attempt (not-authorized) for user dog@meet.mydomain.com from IP: 127.0.0.1

Mh, now I am also out of ideas. You could try to make a tcpdump on port 5280 and check whether the X-Forwarded-For header is actually set. And you could try to add some logging to the get_ip_from_request() function in mod_http.lua to see what actually arrives.

Thanks for the tip. I ran tcpdump and found the headers being sent are showing 127.0.0.1
So, at least now I have confirmation that it’s not prosody. I’ll keep poking around.

Perhaps authentication is somehow routed through Jicofo and not performed by strophe.js -> prosody directly via the bosh connection. At least that would explain why only localhost is logged in prosody. Unfortunately, I am currently not sure how to check whether that’s the case and if so, what to do about that.

Just did some testing and can confirm that this has nothing to do with Jicofo. As a matter of fact, it just works for me when using docker-jitsi-meet, even if I place an additional nginx proxy in front of the web container.

The prosody version installed in the prosody container was 0.11.5-1~stretch6 and here is what I did:

  1. Change into your prosody-plugins-custom dir in your docker configuration folder (create it if needed) and place the mod_log_auth.lua there:
cd your-docker-config-dir/prosody/prosody-plugins-custom/
wget https://hg.prosody.im/prosody-modules/raw-file/6d1ec8099315/mod_log_auth/mod_log_auth.lua
  1. In the prosody virtualhost config your-docker-config-dir/prosody/conf.d/jitsi-meet.cfg.lua add "log_auth"; to the modules_enabled under the main VirtualHost (meet.jitsi in the default docker install)

  2. In the main prosody config prosody.cfg.lua add the trusted_proxies option and add the docker-gateway as well as the docker IP of the web container (Note that the actual IPs may differ in your setup. Skip the docker IPs if you don’t use docker a tall):

trusted_proxies = { "127.0.0.1", "172.20.0.5", "172.20.0.1"  };
  1. In the web container’s nginx conf, I use the $proxy_add_x_forwarded_for directive (not sure if necessary):
# BOSH
location = /http-bind {
    proxy_pass http://xmpp.meet.jitsi:5280/http-bind;
    #proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host meet.jitsi;
}
  1. Restart containers: docker-compose restart. This should already suffice, however, since I had another proxy in front of the web container I needed to add the $proxy_add_x_forwarded_for; in the respective config as well:
        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass http://localhost:8000;
                proxy_read_timeout 90;
                proxy_intercept_errors on;
                proxy_redirect http://localhost:8000 https://meet.example.org;
        }

That’s it for my test setup. When I enter a wrong password in the authentication dialog of the web UI, prosody logs my actual client IP address, exactly as expected.
So just to state it again: It should just work and there must be something happening in your setup that prevents prosody from seeing the external IP.

1 Like

As mentioned over here, the issue of only getting the localhost address may be caused by the default nginx configuration that uses ALPN multiplexing by means of a stream listener on port 443, while the actual http listener is bound to port 4443. The forwarded TCP stream does not originate from the external IP address, so the http listener can not add the remote IP to the X-Forwarded-For header. You can try to add proxy_bind $remote_addr transparent; to the stream listener. Please report back if that solved your issue.

Solution! @rribeirorj @Alejandro_Olivan_Alv

First , thanks for the deep dive @plokta , your input helped me understand config and proxy relationships.

The overall solution (props to @Neal) for me ended up being:

  1. /etc/nginx/nginx.conf > add the following line:
log_format proxycombined '$proxy_protocol_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';
  1. /etc/nginx/modules-enabled/60-jitsi-meet.conf, add the following line the the server {} stanza:
proxy_protocol on;
  1. /etc/nginx/sites-enabled/your.domain.conf, edit the first 4 lines of the server stanza for port444 as follows:
server {
    listen 4444 ssl http2 proxy_protocol;
    listen [::]:4444 ssl http2 proxy_protocol;
    server_name Your.Domain.Here;
    access_log /var/log/nginx/access.log proxycombined;
  1. Lastly, edit the BOSH location in the same file, replacing $remote_addr replaced with $proxy_protocol_addr
    # BOSH
    location = /http-bind {
       proxy_set_header Host $http_host;
       proxy_set_header X-Forwarded-For $proxy_protocol_addr;
       proxy_pass http://localhost:5280/http-bind;
    }
  • After this , prosody will receive the client IP assuming you’ve already enabled log_auth in your lua config.

How do I enable log_auth? (And which config file?)

I typically grab all of the extra prosody modules using mercurial and then symlink the ones I want to use.

  • Install mercurial
sudo apt install mercurial
  • navigate to : /usr/lib > run the following:
    new directories added called prosody-extra/prosody-modules
sudo hg clone https://hg.prosody.im/prosody-modules/ prosody-extra/prosody-modules
  • Create symlinks for the modules you wish to use
sudo ln -s /usr/lib/prosody-extra/prosody-modules/mod_log_auth /usr/lib/prosody/modules

another good one is

sudo ln -s /usr/lib/prosody-extra/prosody-modules/mod_listusers /usr/lib/prosody/modules
  • edit /etc/prosody/conf.avail/your.domain.cfg.lua , adding the new module location
plugin_paths = { "/usr/share/jitsi-meet/prosody-plugins/", "/usr/local/lib/prosody-extra/prosody-modules" }
  • Next, look for “modules_enabled” in the same file and add the following:
"log_auth"; -- Enables logging of the IP address in a failed authentication attempt.
"listusers"; -- Enables listing of users.
  • Restart prosody
sudo systemctl restart prosody

At this point, any failed login attempts will log the client IP, useful for fail2ban.
And to use the bonus module listing users, run:

sudo prosodyctl mod_listusers
1 Like

Woo hoo! Got it. Thank you, @Craig_Eustice

1 Like