Page MenuHomePhabricator

Address Gerrit WMCS instance authenticating against LDAP (breaching WMCS policy)
Open, MediumPublicSecurity

Description

The WMCS instance gerrit-prod-1001.devtools.eqiad1.wikimedia.cloud in the devtools project is used by Release-Engineering-Team for preparing Gerrit upgrade or Puppet patches. One can see it as an integration platform before changes reach out to production.

It got noticed that Gerrit was configured to use the WMCS LDAP server has an authentication backend which goes against WMCS Terms of use:

What uses of Cloud Services do we not like?
...
Use of Wikimedia's LDAP server for authentication: Even with end-user notification that is required for any service that handles passwords or other sensitive data, use of any of the Wikimedia LDAP servers which contain Developer account information for password authentication from within the Cloud Services environment is strictly prohibited.

As a result the instance has been shutdown.

The root cause is the instance is provisioned by Puppet using role::gerrit. It thus applies the same configuration used by Production:

modules/gerrit/templates/gerrit.config.erb
[auth]
    type = LDAP
    cookieSecure = true
    cookiePath = /
    registerUrl = https://wikitech.wikimedia.org/w/index.php?title=Special:CreateAccount&returnto=Gerrit/NewUser
    gitBasicAuth = true
    gitBasicAuthPolicy = HTTP_LDAP
[ldap]
    server = ldaps://<%= @ldap_host %>
...

There might have been a local LDAP server at some point, but it is not anymore. Or at least the Gerrit instance does authenticate against WMCS.

We can either:

A) Spin up a local LDAP. @bd808 gave some example such as Striker testing environment using a Docker container. Our scap3-dev does have a LDAP container as well.

B) Use OpenId which is the default for Gerrit. But that means the test instance will be slightly different than the production one. Gerrit auth types: https://gerrit.wikimedia.org/r/Documentation/config-gerrit.html#auth.type

In either case, the All-Users.git Gerrit database will most probably need to be emptied / redone. I guess we can simply remove it and run gerrit init to create a fresh one.

Details

Risk Rating
Low
Author Affiliation
WMF Technology Dept

Event Timeline

For Security-Team I don't think there is any security issue there but I felt it potentially could have been one. The WMCS devtools project is only reachable by members of Release-Engineering-Team and Paladox and we all have NDA + elevated privileged which can already let us do a lot of nasty thing. The fix is to switch the application to use either a local LDAP or a different authentication backend.

For Security-Team I don't think there is any security issue there but I felt it potentially could have been one. The WMCS devtools project is only reachable by members of Release-Engineering-Team and Paladox and we all have NDA + elevated privileged which can already let us do a lot of nasty thing. The fix is to switch the application to use either a local LDAP or a different authentication backend.

Ok, thanks. This definitely could have been a Vuln-Infoleak IIUC. Does the All-Users.git Gerrit database imply any potential issues with user credentials or other potentially sensitive user data?

Or we could just have no auth provider at all.

So far we have not heard of a use case that actually requires that the login works. puppet refactoring, design changes etc can probably all be done without logging in and Hashar has stated he doesn't usually log in.

And the use case from the volunteer who wanted to just try it out and to learn Gerrit, we have declined.

So I am wondering why even worry about that and not just have no logins, while keeping the rest of the instance.

Basically I would prefer any other solution (different auth provider, no auth, move testing instance to production, X) that is NOT having to setup and maintain a local separate LDAP (and Mediawiki).

Ok, thanks. This definitely could have been a Vuln-Infoleak IIUC.

Possibly, the credentials are used to establish a bind with the LDAP user and if successful the user is in. Gerrit does not store those credentials at all. I think the reason for the policy is the application could have been altered to capture the credentials or some sniffer added. That would have been a Vuln-Infoleak indeed. Anyway using the LDAP for authentication definitely allows for that kind of attack hence why the instance got shut down.

Does the All-Users.git Gerrit database imply any potential issues with user credentials or other potentially sensitive user data?

That is the user local database, each user more or less has:

  • a local username
  • a unique account id
  • 1..n mappings to the id of an external authentication as well as related information such as the email address (which is exposed publicly by the web interface).

So I would have a local user hashar, an id such as 42 then a mapping for the LDAP account hashar and my email hashar@example.org (which is already publicly exposed everywhere by Gerrit and in commit message.

Thanks @sbasset!

Hey @hashar et al - Any reason not to make this task public at this point? Thanks.

Agreed, I think it can be public as well. Instance is shut down. I would recommend we remove LDAP from the auth-provider setting and simply have no auth provider. Then we can bring the machine back up and use it for testing purposes as intended again and nothing else will be needed. Also we meanwhile changed the docs so no users should expect this to work anymore and we have no need for the logins to work.

The instance can stay down for now, I will only need it to test the Gerrit 3.5 > 3.6 upgrade. I will address the authentication issue when I start preparing the Gerrit upgrade (at some point over the next few weeks).

sbassett triaged this task as Medium priority.Mar 13 2023, 3:39 PM
sbassett changed Author Affiliation from N/A to WMF Technology Dept.
sbassett changed the visibility from "Custom Policy" to "Public (No Login Required)".
sbassett changed the edit policy from "Custom Policy" to "All Users".
sbassett changed Risk Rating from N/A to Low.
sbassett moved this task from Watching to Our Part Is Done on the Security-Team board.

Change 888808 merged by Dzahn:

[operations/puppet@production] devtools: change gerrit hostname to use wmcloud, not wmflabs

https://gerrit.wikimedia.org/r/888808