Adding @bd808 as a subscriber—he may have some magic for finding a list of tool maintainers.

In T313359#8089476, @thcipriani wrote:

Adding @bd808 as a subscriber—he may have some magic for finding a list of tool maintainers.

Striker has a local database on the m5 db server. One of the things in that database is a table tracking which tools have used Striker to create differential repos. I can dump that list out and then use it to query the LDAP directory behind our Developer accounts to produce a list of Developer account names (contact via Wikitech talk pages or using sendBulkEmails.php) or raw email addresses.

In T313359#8089476, @thcipriani wrote:

Adding @bd808 as a subscriber—he may have some magic for finding a list of tool maintainers.

Another direction we could come at this from is using conduit to pull the complete list of repos and then examining the ACLs for each repo to get a list of all the Phabricator accounts to contact. Each Phab account is tied to either a SUL account or a Developer account (or both) which could then lead us back to mailing lists of either wiki usernames or raw emails.

I suppose the galaxy brain version would be to produce both lists and contact the distinct union of them.

In T313359#8089535, @bd808 wrote:

In T313359#8089476, @thcipriani wrote:

Adding @bd808 as a subscriber—he may have some magic for finding a list of tool maintainers.

Another direction we could come at this from is using conduit to pull the complete list of repos and then examining the ACLs for each repo to get a list of all the Phabricator accounts to contact. Each Phab account is tied to either a SUL account or a Developer account (or both) which could then lead us back to mailing lists of either wiki usernames or raw emails.

I wonder if this information is available in the database somewhere. @Aklapper do you have thoughts about how easy this query would be?

For git over http one can issue a 301 redirect and avoid breaking the world. For ssh maybe we can have a ssh daemon showing a message before any authentication happens, looks like OpenSsh supports that:

sshd_config(5)

Banner
The contents of the specified file are sent to the remote user before authentication is allowed.
If the argument is none then no banner is displayed.
By default, no banner is displayed.

So potentially keep git-ssh.phabricator.wikimedia.org pointing to some idling ssh daemon that spurts out the banner and disconnect users immediately.

do you have thoughts about how easy this query would be?

Čüş, not really... Haven't played with the Differential DB scheme (and don't have permissions to access the Diffusion scheme) and no idea where ACLs are stored. Took a quick look, have only found reviewer accounts info per repo so far.

In T313359#8092203, @Aklapper wrote:

Čüş

https://en.wiktionary.org/wiki/%C4%8Dus#Etymology ;)

don't have permissions to access the Diffusion scheme)

I have access to the following users:

$fab_user, $metrics_user, $phd_user, $app_user,... I should be able to get the scheme. I'll look.

Of course privileges for you could be changed but not sure if you want that for a one-off and it's a process with DBA.

no idea where ACLs are stored.

ftr, ./modules/profile/templates/mariadb/grants/production-m3.sql.erb but don't worry about it just yet unless you want to

When I used the "metrics_user" I only got a limited set of phabricator databases but it did include phabricator_diffusion. But turns out that the repo info is not in phabricator_diffucion.

Then tried with the not-so-limited app_user and I can see all phabricator databases, including one called phabricator_repository and many others.

So after a use phabricator_repository there are tons of tables but to start with here is the list of repos that have been created by striker:

Now in this table there are also "viewPolicy", "editPolicy" and "pushPolicy" fields. they all are just IDs, names of policies.

There is no obvious "repo owner" field or something like that. I therefore assume what we really want is the list of people who can push, or both edit and push.

Going back to the Phabricator web UI for a moment.. you can just go to https://phabricator.wikimedia.org/diffusion/query/ias.VEMtSCGU/ to get the list of locally hosted repos as well.

If I click through to a random repo from there, for example https://phabricator.wikimedia.org/source/tool-datacompletewiki/ you can see some never had commits / are disabled etc..but some are not.

Actually it looks like a large percentage of them has 0 commits. But let's pick one with commits, like https://phabricator.wikimedia.org/source/tool-nlwiki-bronlinks/

If I go to Manage -> Policies here I can see "Pushable By Custom Policy ".. Then clicking on Custom Policy you get to something like this:

Allow members of any project acl*repository-admins
Allow users valhallasw

So yea.. that's where I will stop for right now but this would mean having to get all the policy IDs and then look into the policies themselves and from there try to find the user names.

total git repos starting with tool- : 445

repos without commits, starting with tool-: 225

I got this via the Phabricator web UI, with search options, sorting by "size"..then some manual formatting., not directly from the database.

https://phabricator.wikimedia.org/diffusion/query/gPlI0Z9TBWkj/

I suggest we first remove the 225 and then get down to 200.

https://etherpad.wikimedia.org/p/1K-YSmk7PCIDjWlEQfMM

Phabricator keeps an history of push actions which can be seen at https://phabricator.wikimedia.org/diffusion/pushlog/ It has the protocol, repo and username among others.

The push logs is kept up to two months, we thus have the history since May 22.

The data can be exported to csv, I have filtered out actions made over https and I end up with 1 bot and 8 users:

@Alexander_Misel
@bd808
@Dsavuljesku
@L10n-bot
@LucasWerkmeister
@Mahir256
@Stang
@Szczot3k
@taavi

There is certainly a lot more users doing pushes, but they are done over https and are thus not concerned by the git-ssh phase out.

I have quickly made a script to generate the list of all for the etherpad. It's running now. I've found at least 1 repo with no users with explicit push perms. That's rPHTO phabricator-Tools

rPHTO is for Phabricator https://phabricator.wikimedia.org/diffusion/PHTO/manage/policies/ says it is pushable by Diffusion-Repository-Administrators . I can't remember the discussion we had in Release-Engineering-Team but I think the idea is to move the Phabricator deployment repository out of Differential toward either Gerrit or Gitlab. Anyway, that repository would in practice only be pushed to by releng people.

Etherpad is updated. Please also see T313487: Some tools have open push policies

Thank you very much to @RhinosF1 for https://www.irccloud.com/pastebin/Rrnxda31/script.py and https://www.irccloud.com/pastebin/zNsHMnJB/

@thcipriani See links above. Now we already have all the Phab user names and all we need is to get the email addresses for them which should be easy.

Someone can start writing the actual email content.

Email draft:

---
Subject: Phabricator's git over ssh will fail after $DATE
---

As of $DATE you will no longer be able to use ssh to interact with git repos
on https://phabricator.wikimedia.org.

You will still be able to use phabricator's git hosting via https[0].

The proposal to use phabricator to host code review was declined in 2018[1].

We're migrating phabricator to new hardware on $DATE. And maintaining ssh access
to repositories requires an ongoing maintenance burden, so we've decided to shut
it down.

You will still be able to use phabricator for git hosting over https. But we
encourage you to consider migration to GitLab[2] in the future.

Thank you!
– Wikimedia Phabricator Maintainers


[0]: <https://wikitech.wikimedia.org/wiki/Phabricator/pushing_over_https>
[1]: <https://phabricator.wikimedia.org/T119908>
[2]: <https://www.mediawiki.org/wiki/GitLab/Hosting_a_project_on_GitLab>

Thoughts?

@thcipriani Sounds good to me. It links to the docs how to push over https, how to move to gitlab and to the reason why Phabricator is not for hosting repos. I like that. Technically we could also untie the 2 $DATES (when phab is switching and when git-ssh is going down) but only if we want to or see an advantage in it.

You will still be able to use phabricator for git hosting over https. But we encourage you to consider migration to GitLab[2] in the future.

@thcipriani: Regarding expectation (or disappointment) management, does the first sentence imply "for many years", or "for a while, but not forever"?
Maybe "strongly encourage"? How long is "in the future"?
I guess I'd prefer a stronger message that having at least four code repository locations (GitLab, Git/Gerrit, Diffusion, GitHub) isn't great for anyone's workflows.

In T313359#8095180, @Dzahn wrote:

Thank you very much to @RhinosF1 for https://www.irccloud.com/pastebin/Rrnxda31/script.py and https://www.irccloud.com/pastebin/zNsHMnJB/

@thcipriani See links above. Now we already have all the Phab user names and all we need is to get the email addresses for them which should be easy.

"easy" 😅

Email doesn't seem to be exposed via any conduit API, so I had to query the database directly.

For reference I used the following query with the list of people provided by @RhinosF1:

SELECT address FROM phabricator_user.user_email ue
    JOIN phabricator_user.user u ON u.phid = ue.userPHID
    WHERE u.userName = "<whatever>";

This netted 347 unique emails for 262 users.

In T313359#8183862, @Aklapper wrote:

You will still be able to use phabricator for git hosting over https. But we encourage you to consider migration to GitLab[2] in the future.

@thcipriani: Regarding expectation (or disappointment) management, does the first sentence imply "for many years", or "for a while, but not forever"?
Maybe "strongly encourage"? How long is "in the future"?
I guess I'd prefer a stronger message that having at least four code repository locations (GitLab, Git/Gerrit, Diffusion, GitHub) isn't great for anyone's workflows.

I'm in favor of "encourage" → "strongly encourage." And s/consider//. So:

For now, you will still be able to use phabricator for git hosting over https. But we strongly encourage you to migrate to GitLab[2].

There are no other plans for diffusion at present. It's close to "for a while, but not forever (probably)"

Now when do I send this thing? :)

@thcipriani I think you can send it right now. It seems just about the value for $DATE in your template. We can just define that as whatever we feel is enough warning time. (2 weeks?) and then turn off git-ssh at that time. The day we shut down git-ssh does not have to correlate much with other migration steps.

Yea, true. I think it did. the few users that were not striker repos but were still using repos on Phabricator I had emailed in the past already.

There is still T308061 about subversion repos, fwiw.

@thcipriani I think it also means I can just shutdown git-ssh and it should not affect anyone. We could stop it before switching servers.. just to proof nobody complains. It doesn't have to be bundled with the server switch but it can be.

In T313359#8230507, @Dzahn wrote:

@thcipriani I think it also means I can just shutdown git-ssh and it should not affect anyone. We could stop it before switching servers.. just to proof nobody complains. It doesn't have to be bundled with the server switch but it can be.

I think that's true. And, yes, stopping it before we switch might give us some extra confidence.