Host/IP address audit of Authentication failures?

plokta · May 22, 2020, 6:07pm

Has anybody tried to use the $proxy_add_x_forwarded_for var instead of $remote_address? This is supposed to append the current $remote_address to the received X-Forwarded-For header, while $remote_address just overwrites potentially existing addresses. This should only be required in cases where an additional reverse proxy sits in front of the jitsi server, so I am not sure if it helps…

The $proxy_add_x_forwarded_for is documente at https://nginx.org/en/docs/http/ngx_http_proxy_module.html#var_proxy_add_x_forwarded_for

Craig_Eustice · May 22, 2020, 6:30pm

I just tested that and also tried

proxy_set_header X-Forwarded-For "$http_x_forwarded_for, $realip_remote_addr";

Restarted nginx and prosody each try, still only seeing 127.0.0.1 in prosody.log

May 22 14:26:58 mod_bosh        info    New BOSH session, assigned it sid '6d6cd7c0-a8fc-4bfc-9e0c-4d7a85744f50'
May 22 14:26:58 bosh6d6cd7c0-a8fc-4bfc-9e0c-4d7a85744f50        info    Failed authentication attempt (not-authorized) for user dog@meet.mydomain.com from IP: 127.0.0.1

plokta · May 22, 2020, 7:18pm

Mh, now I am also out of ideas. You could try to make a tcpdump on port 5280 and check whether the X-Forwarded-For header is actually set. And you could try to add some logging to the get_ip_from_request() function in mod_http.lua to see what actually arrives.

Craig_Eustice · May 22, 2020, 10:14pm

Thanks for the tip. I ran tcpdump and found the headers being sent are showing 127.0.0.1
So, at least now I have confirmation that it’s not prosody. I’ll keep poking around.

plokta · May 23, 2020, 9:38am

Perhaps authentication is somehow routed through Jicofo and not performed by strophe.js -> prosody directly via the bosh connection. At least that would explain why only localhost is logged in prosody. Unfortunately, I am currently not sure how to check whether that’s the case and if so, what to do about that.

plokta · May 23, 2020, 11:28am

Just did some testing and can confirm that this has nothing to do with Jicofo. As a matter of fact, it just works for me when using docker-jitsi-meet, even if I place an additional nginx proxy in front of the web container.

The prosody version installed in the prosody container was 0.11.5-1~stretch6 and here is what I did:

Change into your prosody-plugins-custom dir in your docker configuration folder (create it if needed) and place the mod_log_auth.lua there:

cd your-docker-config-dir/prosody/prosody-plugins-custom/
wget https://hg.prosody.im/prosody-modules/raw-file/6d1ec8099315/mod_log_auth/mod_log_auth.lua

In the prosody virtualhost config your-docker-config-dir/prosody/conf.d/jitsi-meet.cfg.lua add "log_auth"; to the modules_enabled under the main VirtualHost (meet.jitsi in the default docker install)
In the main prosody config prosody.cfg.lua add the trusted_proxies option and add the docker-gateway as well as the docker IP of the web container (Note that the actual IPs may differ in your setup. Skip the docker IPs if you don’t use docker a tall):

trusted_proxies = { "127.0.0.1", "172.20.0.5", "172.20.0.1"  };

In the web container’s nginx conf, I use the $proxy_add_x_forwarded_for directive (not sure if necessary):

# BOSH
location = /http-bind {
    proxy_pass http://xmpp.meet.jitsi:5280/http-bind;
    #proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host meet.jitsi;
}

Restart containers: docker-compose restart. This should already suffice, however, since I had another proxy in front of the web container I needed to add the $proxy_add_x_forwarded_for; in the respective config as well:

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass http://localhost:8000;
                proxy_read_timeout 90;
                proxy_intercept_errors on;
                proxy_redirect http://localhost:8000 https://meet.example.org;
        }

That’s it for my test setup. When I enter a wrong password in the authentication dialog of the web UI, prosody logs my actual client IP address, exactly as expected.
So just to state it again: It should just work and there must be something happening in your setup that prevents prosody from seeing the external IP.

plokta · May 23, 2020, 2:08pm

As mentioned over here, the issue of only getting the localhost address may be caused by the default nginx configuration that uses ALPN multiplexing by means of a stream listener on port 443, while the actual http listener is bound to port 4443. The forwarded TCP stream does not originate from the external IP address, so the http listener can not add the remote IP to the X-Forwarded-For header. You can try to add proxy_bind $remote_addr transparent; to the stream listener. Please report back if that solved your issue.

Craig_Eustice · May 24, 2020, 5:44pm

Solution! @rribeirorj @Alejandro_Olivan_Alv

First , thanks for the deep dive @plokta , your input helped me understand config and proxy relationships.

The overall solution (props to @Neal) for me ended up being:

/etc/nginx/nginx.conf > add the following line:

log_format proxycombined '$proxy_protocol_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';

/etc/nginx/modules-enabled/60-jitsi-meet.conf, add the following line the the server {} stanza:

proxy_protocol on;

/etc/nginx/sites-enabled/your.domain.conf, edit the first 4 lines of the server stanza for port444 as follows:

server {
    listen 4444 ssl http2 proxy_protocol;
    listen [::]:4444 ssl http2 proxy_protocol;
    server_name Your.Domain.Here;
    access_log /var/log/nginx/access.log proxycombined;

Lastly, edit the BOSH location in the same file, replacing $remote_addr replaced with $proxy_protocol_addr

    # BOSH
    location = /http-bind {
       proxy_set_header Host $http_host;
       proxy_set_header X-Forwarded-For $proxy_protocol_addr;
       proxy_pass http://localhost:5280/http-bind;
    }

After this , prosody will receive the client IP assuming you’ve already enabled log_auth in your lua config.

Neal · May 24, 2020, 7:10pm

How do I enable log_auth? (And which config file?)

Craig_Eustice · May 24, 2020, 9:00pm

I typically grab all of the extra prosody modules using mercurial and then symlink the ones I want to use.

Install mercurial

sudo apt install mercurial

navigate to : /usr/lib > run the following:
new directories added called prosody-extra/prosody-modules

sudo hg clone https://hg.prosody.im/prosody-modules/ prosody-extra/prosody-modules

Create symlinks for the modules you wish to use

sudo ln -s /usr/lib/prosody-extra/prosody-modules/mod_log_auth /usr/lib/prosody/modules

another good one is

sudo ln -s /usr/lib/prosody-extra/prosody-modules/mod_listusers /usr/lib/prosody/modules

edit /etc/prosody/conf.avail/your.domain.cfg.lua , adding the new module location

plugin_paths = { "/usr/share/jitsi-meet/prosody-plugins/", "/usr/local/lib/prosody-extra/prosody-modules" }

Next, look for “modules_enabled” in the same file and add the following:

"log_auth"; -- Enables logging of the IP address in a failed authentication attempt.
"listusers"; -- Enables listing of users.

Restart prosody

sudo systemctl restart prosody

At this point, any failed login attempts will log the client IP, useful for fail2ban.
And to use the bonus module listing users, run:

sudo prosodyctl mod_listusers

Neal · May 24, 2020, 9:14pm

Woo hoo! Got it. Thank you, @Craig_Eustice

rribeirorj · May 25, 2020, 10:57pm

PERFECT!

Thank you very much! More security for us!

Alejandro_Olivan_Alv · June 1, 2020, 7:47am

Wow!!! have to try that!!! it looks promising, And I feel It will work!

Thank you very much!!!

masteryoda · June 1, 2020, 11:52am

thank you… this worked like a charm

However guest users who try to join the meeting with wrong passcode isnt logged. can that be enabled too?

i set a meeting up with guest code 12345 & if the guest uses the wrong password, this is the line thats written in the log

Jun 01 11:46:36 general info --------------> user sdkryaiv3m7w5g_a domain guest.meet.mydomain.com res lw4_ksfm pass 12345

here’s the prosody log

Jun 01 11:42:38 bosh5a0c32c5-0063-49ee-9c47-717554fd068e        info    Failed authentication attempt (not-authorized) for user admin@meet.mydomain.com from IP: x.x.x.x
Jun 01 11:42:58 conference.meet.mydomain.com:muc_domain_mapper   warn    Session filters applied
Jun 01 11:42:58 mod_bosh        info    New BOSH session, assigned it sid '7ce132b8-8d9d-4fa1-a074-79b0888baa74'
Jun 01 11:42:58 bosh7ce132b8-8d9d-4fa1-a074-79b0888baa74        info    Authenticated as admin@meet.mydomain.com
Jun 01 11:43:00 general info    --------------> user focus domain auth.meet.mydomain.com res focus951775946518 pass nil
Jun 01 11:43:00 bosh7ce132b8-8d9d-4fa1-a074-79b0888baa74        info    BOSH client disconnected: session close
Jun 01 11:43:00 general info    --------------> user b79ezmigdq5ewkva domain guest.meet.mydomain.com res NFKOge_m pass nil
Jun 01 11:43:23 boshf3f63ab6-51dc-4374-a602-1656f640a170        info    BOSH client disconnected: session close
Jun 01 11:43:23 bosh5a0c32c5-0063-49ee-9c47-717554fd068e        info    BOSH client disconnected: session close
Jun 01 11:43:23 speakerstats.meet.mydomain.com:speakerstats_component    warn    A module has been configured that triggers external events.
Jun 01 11:43:23 speakerstats.meet.mydomain.com:speakerstats_component    warn    Implement this lib to trigger external events.
Jun 01 11:43:25 conference.meet.mydomain.com:muc_domain_mapper   warn    Session filters applied
Jun 01 11:43:25 mod_bosh        info    New BOSH session, assigned it sid '2121eea8-7046-45ac-a3a1-72e206c6f4a8'
Jun 01 11:43:26 bosh2121eea8-7046-45ac-a3a1-72e206c6f4a8        info    Authenticated as gcu28fqctuxtpbkx@guest.meet.mydomain.com
Jun 01 11:43:27 general info    --------------> user focus domain auth.meet.mydomain.com res focus951775946518 pass nil
Jun 01 11:43:27 general info    --------------> user gcu28fqctuxtpbkx domain guest.meet.mydomain.com res 3URu2nRq pass nil
Jun 01 11:44:17 conference.meet.mydomain.com:muc_domain_mapper   warn    Session filters applied
Jun 01 11:44:17 mod_bosh        info    New BOSH session, assigned it sid '6cff02f8-7102-497e-b105-45332b45f31c'
Jun 01 11:44:18 bosh6cff02f8-7102-497e-b105-45332b45f31c        info    Authenticated as recorder@recorder.meet.mydomain.com
Jun 01 11:44:18 general info    --------------> user recorder domain recorder.meet.mydomain.com res B957MRgc pass nil
Jun 01 11:46:00 conference.meet.mydomain.com:muc_domain_mapper   warn    Session filters applied
Jun 01 11:46:00 mod_bosh        info    New BOSH session, assigned it sid '5a6865a0-b384-4116-b018-56221105f9b3'
Jun 01 11:46:01 bosh5a6865a0-b384-4116-b018-56221105f9b3        info    Authenticated as sdkryaiv3m7w5g_a@guest.meet.mydomain.com
Jun 01 11:46:04 general info    --------------> user sdkryaiv3m7w5g_a domain guest.meet.mydomain.com res lw4_ksfm pass 12345
4444Jun 01 11:46:21 general     info    --------------> user sdkryaiv3m7w5g_a domain guest.meet.mydomain.com res lw4_ksfm pass 12345
Jun 01 11:46:36 general info    --------------> user sdkryaiv3m7w5g_a domain guest.meet.mydomain.com res lw4_ksfm pass 12345
Jun 01 11:46:53 general info    --------------> user sdkryaiv3m7w5g_a domain guest.meet.mydomain.com res lw4_ksfm pass 12345
Jun 01 11:47:05 general info    --------------> user sdkryaiv3m7w5g_a domain guest.meet.mydomain.com res lw4_ksfm pass 12345
Jun 01 11:47:11 bosh5a6865a0-b384-4116-b018-56221105f9b3        info    BOSH client disconnected: session close

Alejandro_Olivan_Alv · June 1, 2020, 12:13pm

hi!

That first line on your log looks but, quite promising:

…failed authentication attempt (not-authorized) for user admin@meet.mydomain.com from IP: x.x.x.x

I’m out now, so I can’t still do test by myself, but that IP x.x.x.x wasn’t the real IP from where the illegal action was performed? … If so, That would be enough to taylor a failban pattern/filter to ban x.x.x.x IP address if necessary … looks really good

Regards.

masteryoda · June 1, 2020, 12:17pm

it works just fine if a registered user tries to login & fails.

it doesnt however log invalid login/join attempts of guests with passcode

Alejandro_Olivan_Alv · June 1, 2020, 12:23pm

mmm … now I guet your point… which is very interesting, I agree those events should be logged

Craig_Eustice · June 2, 2020, 12:55pm

Hi @masteryoda, I haven’t noticed that with my setup probably because I have guests join without credentials after a host authenticates and starts the room.

I’ll try to do some testing as I have time, Just so I understand … you have everyone authenticate? meaning a host would authenticate and start a room, but also any guests that join need to authenticate as well?

masteryoda · June 2, 2020, 1:35pm

thanks Craig

I am looking to log the IP addresses for failed attempts by guests who join the room with a PASSCODE, not a prosody user/pass

Craig_Eustice · June 2, 2020, 2:31pm

Ok, I understand now. That’s probably going to need a feature add request. As I understand it, it would need to be Jicofo that logs those attempts.