Page MenuHomePhabricator

Re-enable use of Gerrit HTTP token to push patchsets
Closed, ResolvedPublic

Description

I believe this was disabled in https://gerrit.wikimedia.org/r/c/operations/puppet/+/497066/1/modules/gerrit/templates/gerrit.config.erb

Long-term this is a feature we want, and has been used by multiple scripts/bots.

Just to be clear, is the removal of the Gerrit HTTP token feature going to be a permanent change or just temporary? If this is a permanent thing I'll have to figure out how to modify libraryupgrader as well...

I hope not. We definitely don't want to lose gerrit rest api in the long run.

Event Timeline

Before we move forward and enable this, let's make sure we have understood the security repercussions and have mitigated them (and if we find it impossible to do so, avoid it).

This is a double edge knife feature as it allows anyone with temporary access to a browser session to set a gerrit password and perform actions on behalf of the user, possibly unnoticed for a long time. It's a secondary authentication method. One, that many new time (and perhaps old time) users are not trained to expect (and don't really expect). It also defeats one of the security practices that users may be following, namely rotating passwords often (and does so in a very unclear way for many people). It's noteworthy that It has been used recently to upload malicious changes to a repo in a case where the attacker had 0 knowledge of the LDAP authentication credentials of the victim. Granted, the attacker had access to the account and the web interface and as such would have been able to create the aforementioned change via the gerrit UI, but they clearly found it easier to just set the HTTP password and use it.

I don't know the gerrit permissions model/system, But could we perhaps only enable it for a small group, so anyone that wants it would need to request access? That should reduce the risk level depending on how many people actually use the feature.

That's not how that works unfortunately, it's either enable it for all, or disable it for all (since this is enabled though a gerrit config).

Before we move forward and enable this, let's make sure we have understood the security repercussions and have mitigated them (and if we find it impossible to do so, avoid it).

+1

This is a double edge knife feature as it allows anyone with temporary access to a browser session to set a gerrit password and perform actions on behalf of the user, possibly unnoticed for a long time. It's a secondary authentication method. One, that many new time (and perhaps old time) users are not trained to expect (and don't really expect). It also defeats one of the security practices that users may be following, namely rotating passwords often (and does so in a very unclear way for many people). It's noteworthy that It has been used recently to upload malicious changes to a repo in a case where the attacker had 0 knowledge of the LDAP authentication credentials of the victim. Granted, the attacker had access to the account and the web interface and as such would have been able to create the aforementioned change via the gerrit UI, but they clearly found it easier to just set the HTTP password and use it.

The same could be said of the ssh key. Temporary access to a browser session would enable someone to add a key to perform actions on behalf of a user. The gerrit command line tools have mostly the same capabilities as the http token.

This is a double edge knife feature as it allows anyone with temporary access to a browser session to set a gerrit password and perform actions on behalf of the user, possibly unnoticed for a long time. It's a secondary authentication method. One, that many new time (and perhaps old time) users are not trained to expect (and don't really expect). It also defeats one of the security practices that users may be following, namely rotating passwords often (and does so in a very unclear way for many people). It's noteworthy that It has been used recently to upload malicious changes to a repo in a case where the attacker had 0 knowledge of the LDAP authentication credentials of the victim. Granted, the attacker had access to the account and the web interface and as such would have been able to create the aforementioned change via the gerrit UI, but they clearly found it easier to just set the HTTP password and use it.

The same could be said of the ssh key. Temporary access to a browser session would enable someone to add a key to perform actions on behalf of a user. The gerrit command line tools have mostly the same capabilities as the http token.

That's really good point.

The same could be said of the ssh key. Temporary access to a browser session would enable someone to add a key to perform actions on behalf of a user. The gerrit command line tools have mostly the same capabilities as the http token.

That's really good point.

It is indeed. Could we perhaps mitigate some of the security ramifications of both of these by having some form of notification? e.g. an email with

  • Your gerrit HTTP password has been changed, report to <INSERT contact point> if it wasn't done by you
  • An SSH key was added/removed to your gerrit account, report to <INSERT contact point> if it wasn't done by you.

would suffice in my mind.

@Paladox any idea if such a feature exists in gerrit?

Gerrit sends a email if a ssh key is added (not sure about if it's removed). Gerrit does not notify if a http password has changed (though that would be great to implement!)

I can speak to upstream about adding a notification for http password changes.

I've done 2 changes https://gerrit-review.googlesource.com/c/gerrit/+/221772/ and https://gerrit-review.googlesource.com/c/gerrit/+/221774/ which will help users know if someone removed their ssh key or changed there http password (not to be confused with auth password).

Paladox removed a subtask: Restricted Task.May 6 2019, 10:21 PM
thcipriani closed subtask Restricted Task as Invalid.May 18 2019, 1:50 PM
akosiaris reopened subtask Restricted Task as Open.May 19 2019, 7:40 AM
Paladox changed the status of subtask Restricted Task from Stalled to Open.Jun 13 2019, 12:28 AM
Paladox triaged this task as Medium priority.Jun 18 2019, 12:08 PM
Paladox closed subtask Restricted Task as Resolved.Jun 24 2019, 7:57 PM

Gerrit has been patched now, so this is now unblocked!

Change 518811 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] gerrit: Re-enable the use of HTTP auth tokens

https://gerrit.wikimedia.org/r/518811

Change 518811 merged by CDanis:
[operations/puppet@production] gerrit: Re-enable the use of HTTP auth tokens

https://gerrit.wikimedia.org/r/518811

thcipriani claimed this task.

Config change has been deployed.

Will announce on Wikitech as well.

Spoke too soon. Gerrit 2.15.14 caused a lot of SendEmail locks (T224448: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely). I have to rollback to 2.15.13. I will update wikitech as well.

thcipriani reopened subtask Restricted Task as Open.Jul 8 2019, 2:59 PM
thcipriani closed subtask Restricted Task as Resolved.Jul 23 2019, 12:17 AM

Change 527596 had a related patch set uploaded (by Paladox; owner: Thcipriani):
[operations/puppet@production] gerrit: Re-enable the use of HTTP auth tokens

https://gerrit.wikimedia.org/r/527596

Change 527596 merged by CDanis:
[operations/puppet@production] gerrit: Re-enable the use of HTTP auth tokens

https://gerrit.wikimedia.org/r/527596

This has been re-enabled since August 8th. I have been waiting to see if this change, once again, caused T224448 to increase in frequency before resolving this task.

Gerrit seems stable (knock on wood), so resolving.

thcipriani closed subtask Restricted Task as Resolved.Oct 21 2019, 5:48 PM