Maybe Release-Engineering-Team has some input / thoughts?

@mseckington Any recommendations? TIA

@Bmueller mentioned that this might also be in the realm of Security-Team. Comments welcome.
I know that nobody is happy about more on their list, but I'd like to reduce single points of failure, and this isn't needed too often.

@Aklapper - Will plan to review with Security-Team leadership this week.

@sbassett: Thanks. Any updates? This is a bottleneck, e.g. I missed T330073 (and nobody else picked it up either)...

In T306708#8657402, @Aklapper wrote:

@sbassett: Thanks. Any updates? This is a bottleneck, e.g. I missed T330073 (and nobody else picked it up either)...

The update is an unfortunate one, in that we do not feel we are currently resourced to take on this additional service/workload.

@sbassett: Thanks for the update! @Bmueller: Please share how to best proceed here. Thanks!

Not sure what the process is now but I created a ticket T334480

https://www.mediawiki.org/wiki/Topic:Xi3nbbjzvuavtbu4 is the most recent example that things do not work currently.

Paraphrasing/quoting a conversation with @acooper so we have thoughts in one place:

• Staff 2FA resets are easier, could streamline and have an official policy.
  • Ideally, verification could be performed by someone (the verifier) who knows the affected person and who is able to identify
  • Ideally the staff member performs the verification over a video call, due to concerns about other potential auth systems having (too) long lived tokens
  • The verifier then updates the ticket (if the request was actually brought up in a ticket (see "Define a place where 2FA reset requests can be brought up when user cannot log into Phab anymore" above)
• For scalability issues with the actual act of enacting the 2FA reset requests, that requires shell/admin permissions (see "That action requires being listed" above)

performs the verification over a video call

To verify someone over video call the verifier would have to know how each given user looks and who they are (and who they report to). I am not sure this is the case (anymore) unless the verifier has Namely access and Namely always has current data and profile photos uploaded by HR?

Namely goes away anyway. :)
It would mean finding a person who both the requestor and the resetter know how they look like, and other way round (latest example).
And that negotiation does not sound like a great use of time for anyone (but sure... security).

In T306708#9318592, @Aklapper wrote:

Namely goes away anyway. :)

I would hope that Dayforce has similar directory searching capabilities. If it does not, that would be very unfortunate, for a number of reasons.

It would mean finding a person who both the requestor and the resetter know how they look like, and other way round (latest example).
And that negotiation does not sound like a great use of time for anyone (but sure... security).

I'd agree that this isn't a very sustainable option, on its face (no pun intended). I think pairing it with asking a live individual certain questions that might only be available from data on officewiki or within Dayforce might be ok though, and would rely less upon verifying someone based upon their current physical appearance.

I was assuming finding the person would be easy - it would be maybe their manager? Which is easily available from the address book.

Oh and all that is required is for the verifier to also post a message authenticated to their accounted confirming they have verified the identity. The verifier in this case is someone from the persons team or management chain. So its not a fixed identity.

The admin would then only need to verify that the verifier posted an authenticated message (e.g. to a phab ticket comment) confirming they verified the user in question.

You could also require messages of acknowledgement to be sent across at least two authentication domains to increase the security. So say a slack message (linked in a phab comment) and a phab comment.

In T306708#9323224, @acooper wrote:

I was assuming finding the person would be easy - it would be maybe their manager? Which is easily available from the address book.

Are you referring to Namely or another address book?

posted an authenticated message (e.g. to a phab ticket comment)

I think part of the problem is that you don't know who actually created the Phab account in the first place, unless it's linked to a wiki user and that wiki user was created by office IT and you follow the whole chain. I could be wrong though.

Namely or just the gmail synced address book.

I think the problem you referred to about unknown usernames to employee mapping, can also be fixed quite easily and without any significant changes to the proposed process. Say a unknown employee user called PACMAN (I'm just making up a name this doesn't refer to anyone specifically). PACMAN knows who they are. They reach out to their manager for video verification. Once that's been done their manager writes a message confirming that they have validated PACMAN's identity and confirming which employee they are.

I have also been thinking about volunteers who are not employees. This is quite a difficult problem as we don't have many trusted means to verify. I have a few ideas I am thinking about:

• Have a WMF staff member who knows the volunteer verify them over a video call. HOWEVER, there seems quite a high likelihood they either don't know a WMF staff member, or even if they do, they wouldn't know what they look like.
• Have 1 or more other community members of good standing (admins?) verify the identity and vouch that it was done.
• Prove access to the email address associated with the account, combined with waiting for a period of time before performing the reset (maybe 30 days) which allows time for a possible account takeover scenario to be detected and remediated.
• Prove access to a wikipedia account with the same username by writing a talk message. I am not sure how common it is that the usernames match though?

Also could require some combination of the above. Has anyone got any other ideas?

If I might insert some input from the Service Desk team, we have established our own SOP for reset requests that perhaps may provide a sort of model here.

Essentially, we have two layers of verification for Foundation staff members requesting Okta password or 2fa resets. Of course, the limitation here is this only works if they're Foundation staff who have Okta accounts.

If a staff member can send the request from their Foundation Slack account, we consider this good enough proof of identity to reset their Okta access. The rationale being that you would have needed prior access to the Okta account in order to sign in to that Slack account and users very often are still signed in to Slack, even after forgetting their password, losing their 2fa method, or otherwise becoming locked out of Okta. However, should they not have access to Slack, our next verification step is to reach out directly to the user's manager. Their manager will in turn verify that the request is legitimate and then provide us secondary contact info to correspond with the user.

My thoughts on how this SOP could be adapted here is perhaps if a staff member needs a phabricator 2fa request and SRE needs identity verification before proceeding, then they can interface directly with the user via Slack or with the user's manager to verify the validity of the request and identity of the requester.

I know some of this may not quite apply the same way, since phabricator is different. Just thought I would chime in with some additional ideas.

@JLam-WMF Thanks for this! One small question, out of curiosity, how do you determine who the manager of a user is? Do you look it up in Namely (soon the system replacing it), LDAP or some other way like an orgchart somewhere?

Ah, there's the rub I guess. We have access to that info via Okta.

I guess that creates something of a stumbling block. Though, until we get a better tooling, maybe a stop gap in the SOP here would be you guys can reach out to techsupport@ requesting that info. Talent and Collaboration would also know.

I have been told that Dayforce should eventually obviate the need for this sort of telephone tag, since it can be hooked to Okta and can provide attributes such as manager info. @bcampbell could comment more.

Proposed new verification option: ssh signatures

Requirements:

• Phabricator account linked with LDAP account.
• LDAP account with an ssh key.

Mechanism:

• User signs dated reset request file and emails/messages/files a task with file + file signature

Instructions for users—Create a signed reset request:

1. Create a reset request text file
2. Create a reset request signature using your LDAP ssh key private key
3. Send your reset request text file and reset request signature to a phab admins (maybe a mailing list would be useful here? There is nothing private in these signatures.)

Commands:

$ # 1. Create a reset request text file
$ echo "$(date -Is) Requesting phabricator.wikimedia.org 2fa reset for YOUR USERNAME" > /tmp/YOUR-USERNAME-phab-reset-request.txt"
$ # 2. Sign reset request with LDAP ssh key
$ ssh-keygen -Y sign -f PATH/TO/LDAP/SSH/PRIVATE.KEY -n file /tmp/YOUR-USERNAME-phab-reset-request.txt
Signing file /tmp/YOUR-USERNAME-phab-reset-request.txt
Write signature to /tmp/YOUR-USERNAME-phab-reset-request.txt.sig

Instructions for admins—Verify a signed reset request:

1. Get user's reset request and reset request signature and store locally.
2. Get user's public key from LDAP; e.g., https://ldap.toolforge.org/user/thcipriani
3. Create a phab-reset-request signers file with the ssh public key
4. Verify signature

$ # 1. Get user's reset request and reset request signature and store locally.
$ ls /tmp/USER-phab-reset-request.txt*
/tmp/USER-phab-reset-request.txt
/tmp/USER-phab-reset-request.txt.sig
$ # 2. Get user's public key from LDAP; e.g., https://ldap.toolforge.org/user/thcipriani
$ export PHABRICATOR_USER_RESET_PUBLIC='ssh-ed25519 etc...'
$ # 3. Create a `phab-reset-request` signers file with the ssh public key
$ `echo "USERNAME $PHABRICATOR_USER_RESET_PUBLIC" >> /tmp/phabricator-reset-signers`
$ # 4. Verify signature
$ ssh-keygen -Y verify -f /tmp/phabricator-reset-signers -I USERNAME -n file -s /tmp/USER-phab-reset-request.txt.sig < /tmp/USER-phab-reset-request.txt
Good "file" signature for USERNAME with RSA key SHA256:KEYHASH

Pros

• proves that user has private ssh key
• private ssh key is a second factor of authentication that is tied to a phabricator account already
• verification can be done by anyone without revealing any private information
• many technical contributors have ldap auth and ssh keys already, likely more than have a committed identity hash
• gives us another option

Cons

• requires ldap account linking
• requires an ssh key
• could be hard to help folks through the process, even with decent docs
• I've left ambiguous how to get the signatures to phab admins
• many users will not have one of
  • an ldap account linked to their phab account
  • an ssh key tied to that ldap account

Current Two-factor Authentication Resets instruction on mw.org

Random thoughts:

1. I have never used the Preferred method to do a reset
   • The preferred method requires four things
     • that a Phabricator user has actually tied their Phabricator account to an MediaWiki SUL account
     • a long-lived committed identity hash
     • an active phab session
     • the actual phrase used to create the committed identity hash
   • All resets have lacked one or more of these things
   • We could remove the requirement for an active phab session if we had another band to send private information. Maybe an email.
2. Committed identity hash is confusing for folks. In short, a committed identity hash is a sha512 hash of some phrase—effectively a password that's hashed and stored in public. The name "committed identity hash" and very long/thorough docs made me think, initially, it was something more complicated than it is.
3. The only other method is "know somebody"

In T306708#11393904, @thcipriani wrote:

Proposed new verification option: ssh signatures

If an attacker has enough access to a developer account to get to the Phabricator 2FA screen, what's stopping them from just adding an another SSH key to that developer account and then using that key to "prove" they are the correct owner of said account?

In T306708#11393919, @taavi wrote:

In T306708#11393904, @thcipriani wrote:

Proposed new verification option: ssh signatures

If an attacker has enough access to a developer account to get to the Phabricator 2FA screen, what's stopping them from just adding an another SSH key to that developer account and then using that key to "prove" they are the correct owner of said account?

That's an excellent question. Nothing, I suppose. I suppose it's similar to the MediaWiki committed identity hash, where we rely on: how long ago was this set?

It looks like we do have a modification timestamp in ldap:

thcipriani@tools-bastion-15:~$ ldapsearch -xxx '(&(objectClass=posixAccount)(sshPublicKey=*)(uid=thcipriani))' sshPublicKey modifyTimestamp modifiersName
...
dn: uid=thcipriani,ou=people,dc=wikimedia,dc=org
sshPublicKey:: <blah-redacted-for-space>
modifyTimestamp: 20250528170616Z
modifiersName: cn=admin,dc=wikimedia,dc=org
...

Comparing that against last successful login, I guess.

Information that seems possible to get:

• When did user login last with 2fa (I don't know, offhand how to get this)
• When did user change their LDAP account last

We could say that the user last controlled the 2fa secret at the time this ssh key existed on their LDAP. BUT I think you're right that that is no guarantee of anything. Their SSH key could have been updated a year ago by a malicious actor who was banking that the user wouldn't notice to get access to phab.

Maybe this is enough validation to reset 2fa for folks without advanced rights in phabricator.

In T306708#11394159, @thcipriani wrote:

In T306708#11393919, @taavi wrote:

If an attacker has enough access to a developer account to get to the Phabricator 2FA screen, what's stopping them from just adding an another SSH key to that developer account and then using that key to "prove" they are the correct owner of said account?

That's an excellent question. Nothing, I suppose. I suppose it's similar to the MediaWiki committed identity hash, where we rely on: how long ago was this set?

This doesn't erase questions, but we did use ssh key control as proof of identity for Developer account 2FA removals for many years: https://wikitech.wikimedia.org/wiki/Password_and_2FA_reset#For_developer_account_users

• When did user login last with 2fa (I don't know, offhand how to get this)

We do not know because https://we.phorge.it/T16310

In T306708#11394508, @Aklapper wrote:

• When did user login last with 2fa (I don't know, offhand how to get this)

We do not know because https://we.phorge.it/T16310

Interesting! I am reading this as: it might be possible to know Soon™, accurate?

In T306708#11394760, @thcipriani wrote:

We do not know because https://we.phorge.it/T16310

Interesting! I am reading this as: it might be possible to know Soon™, accurate?

I updated my upstream patch to pass unit tests. I cannot judge if our use case(s) are convincing enough though to get this merged into upstream. Let's see.