Page MenuHomePhabricator
Create Task
Maniphest T296022

Deprecate git-ssh service on phabricator.wikimedia.org
Closed, ResolvedPublic
Actions

Description

We now have 3 git hosting systems. Phabricator is not the authoritative place to host most repositories.

The git-ssh service on phabricator is barely used and because it uses pybal, it's a significant amount of extra maintenance burden for serviceops every time there is a server or datacenter migration or hardware upgrade.

We should consider deprecating this service. We would still have git over https for projects that still use phabricator.

Before doing this, we should evaluate actual usage and identify any repos / users who are using git-ssh so that we can give notice and help users to migrate before we shut it down.

  • Find users who are still dependent on this service
  • Migrate affected users to https or to gitlab.
    • deactivate active but empty repos
    • deactivate obviously abandoned / test repos with minimal commits
    • migrate iltools repo to gitlab
    • migrate WikiSP: MediaWiki Config to gitlab (T296108)
    • striker repos are migrated or have no more ssh:// cloning
    • mediawiki extension repos, same as above

Details

SubjectRepoBranchLines +/-
phabricator: move enable_vcs parameter from eqiad-only to roleoperations/puppetproduction+2 -1
phabricator: remove sshd config for git-ssh serviceoperations/puppetproduction+0 -48
phabricator: remove support for separate git-ssh IP behind LVSoperations/puppetproduction+0 -15
devtools/phabricator: remove comment about git-ssh vcs serviceoperations/puppetproduction+0 -1
phabricator::vcs: comment out warning about empty listen addressoperations/puppetproduction+1 -1
phabricator: remove vcs_addresses parameter and warningsoperations/puppetproduction+0 -14
remove git-ssh.wikimedia.orgoperations/dnsmaster+0 -3
remove git-ssh from common/service.yamloperations/puppetproduction+0 -28
conftool-data: remove phabricator / git-sshoperations/puppetproduction+0 -4
phabricator: stop ssh-phab serviceoperations/puppetproduction+2 -2
phabricator: move LVS IPs for git-ssh service from role/eqiad to phab1001operations/puppetproduction+5 -6
phabricator: move vcs and LVS settings from common to phab2001operations/puppetproduction+20 -19
phabricator: move lvs::realserver inclusion to profile, create use_lvs parameteroperations/puppetproduction+8 -5
phabricator: fix location of hosts files, ensure services stopped on newoperations/puppetproduction+8 -2
phabricator: allow disabling ssh-phab service except on one hostoperations/puppetproduction+9 -2
phabricator: fix ferm rules for VCS, git-sshoperations/puppetproduction+17 -5
phabricator: move vcs firewall rules to profileoperations/puppetproduction+16 -6
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Dzahn added a comment.Mar 28 2022, 6:26 PM

Thank you @awight :) Gotcha! So for now, even without migrating, it will still be possible to push to the repo, just not via ssh. It's still possible to push via https. That's the only change I will make here for now. Greetings and all the best, Daniel

gerritbot added a comment.Apr 7 2022, 11:28 PM

Change 778366 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: allow disabling ssh-phab service except on one host

https://gerrit.wikimedia.org/r/778366

gerritbot added a project: Patch-For-Review.Apr 7 2022, 11:28 PM
gerritbot added a comment.Apr 8 2022, 8:32 PM

Change 778366 merged by Dzahn:

[operations/puppet@production] phabricator: allow disabling ssh-phab service except on one host

https://gerrit.wikimedia.org/r/778366

Maintenance_bot removed a project: Patch-For-Review.Apr 8 2022, 9:30 PM
Dzahn added a project: collaboration-services.Jun 27 2022, 9:56 PM
Dzahn moved this task from Incoming to Procurement on the collaboration-services board.Jul 9 2022, 3:18 AM
thcipriani removed a subtask: T290337: Migrate from pws gpg to 1password.Jul 19 2022, 10:32 PM
thcipriani removed a subtask: T301170: Change from ssh to pushing via https for "releng-secrets" git repo.
thcipriani edited projects, added Release-Engineering-Team (The Decommission Mission 💀); removed Release-Engineering-Team (Next).Jul 19 2022, 10:35 PM
thcipriani edited projects, added Release-Engineering-Team (Bonus Level 🕹️); removed Release-Engineering-Team (The Decommission Mission 💀).Aug 16 2022, 7:59 PM
gerritbot added a comment.Aug 16 2022, 10:04 PM

Change 823755 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: move lvs::realserver inclusion to profile, depend on vcs_enabled

https://gerrit.wikimedia.org/r/823755

gerritbot added a project: Patch-For-Review.Aug 16 2022, 10:04 PM
gerritbot added a comment.Aug 17 2022, 10:21 PM

Change 823755 merged by Dzahn:

[operations/puppet@production] phabricator: move lvs::realserver inclusion to profile, create use_lvs parameter

https://gerrit.wikimedia.org/r/823755

gerritbot added a comment.Aug 17 2022, 11:52 PM

Change 824319 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: fix location of hosts files, ensure services stopped on new

https://gerrit.wikimedia.org/r/824319

gerritbot added a comment.Aug 17 2022, 11:55 PM

Change 824319 merged by Dzahn:

[operations/puppet@production] phabricator: fix location of hosts files, ensure services stopped on new

https://gerrit.wikimedia.org/r/824319

gerritbot added a comment.Aug 19 2022, 11:23 PM

Change 824798 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: move vcs and LVS settings from common to phab2001

https://gerrit.wikimedia.org/r/824798

gerritbot added a comment.Aug 19 2022, 11:57 PM

Change 824798 merged by Dzahn:

[operations/puppet@production] phabricator: move vcs and LVS settings from common to phab2001

https://gerrit.wikimedia.org/r/824798

thcipriani moved this task from Backlog to Ready on the Release-Engineering-Team (Bonus Level 🕹️) board.Aug 22 2022, 4:07 PM
gerritbot added a comment.Aug 24 2022, 6:18 PM

Change 826360 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: move LVS IPs for git-ssh service from role/eqiad to phab1001

https://gerrit.wikimedia.org/r/826360

gerritbot added a comment.Aug 24 2022, 6:21 PM

Change 826360 merged by Dzahn:

[operations/puppet@production] phabricator: move LVS IPs for git-ssh service from role/eqiad to phab1001

https://gerrit.wikimedia.org/r/826360

thcipriani closed subtask T313359: Email tool maintainers about git-ssh deprecation on phabricator as Invalid.Sep 12 2022, 6:01 PM
gerritbot added a comment.Sep 12 2022, 9:56 PM

Change 831627 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] disable git-ssh.wikimedia.org

https://gerrit.wikimedia.org/r/831627

Dzahn added a comment.Sep 12 2022, 10:14 PM

also T313359#8230507

Dzahn updated the task description. (Show Details)Sep 12 2022, 10:14 PM
Dzahn updated the task description. (Show Details)
Dzahn added a subscriber: bd808.

striker repos have been migrated:) (Thanks @bd808!)

Hmm. now..I still notice the "mediawiki extension repos" check box.

Stashbot mentioned this in T315706: Migrate existing Striker created Diffusion repos to GitLab.Sep 12 2022, 10:23 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-12T22:23:11Z] <mutante> phabricator - disabling repositories: tool-xh-bot, tool-editor-contribution-dashboard, tool-ranker, tool-editor-contribution, tool-mikasa-bot-1, tool-maintun, tool-add-text, tool-wikibookassamese-book.php (none of them had commits) T296022 - T315706

Stashbot added a comment.Sep 12 2022, 10:53 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-12T22:53:29Z] <mutante> phabricator - disabling MediaWiki extension repositories in Diffusion that have 0 commits - T296022 - T315706

Don-vip subscribed.Sep 14 2022, 8:22 PM
Dzahn mentioned this in T191182: Migrate active repositories in Phabricator Differential to GitLab.Sep 14 2022, 11:58 PM
LSobanski moved this task from Procurement to Work in Progress on the collaboration-services board.Sep 20 2022, 3:23 PM
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:20 PM
Hawkeye7 added a comment.Oct 7 2022, 12:41 AM

Something has gone wrong. My attempt to do a git push results in fatal: unable to access 'http://phabricator.wikimedia.org/source/tool-milhistbot.git/': The requested URL returned error: 403.

No prompt is issued for a userid/.password

JJMC89 subscribed.Oct 7 2022, 12:57 AM
In T296022#8296545, @Hawkeye7 wrote:

Something has gone wrong. My attempt to do a git push results in fatal: unable to access 'http://phabricator.wikimedia.org/source/tool-milhistbot.git/': The requested URL returned error: 403.

No prompt is issued for a userid/.password

That is unrelated to this task. tool-milhistbot is now hosted on GitLab, and Diffusion just has a mirror. See T315706: Migrate existing Striker created Diffusion repos to GitLab and the announcment.

Dzahn added a comment.Oct 7 2022, 1:16 AM

Yep, what you can expect to happen soon _for this task_ would be for git-ssh.wikimedia.org to disappear from DNS and simply not exist.

But that would only affect pushing via ssh, not via https.

tool-* repos have moved to Gitlab as JJMC89 said. Your repo is now here: https://gitlab.wikimedia.org/rmallett/tool-milhistbot

Hawkeye7 added a comment.Oct 7 2022, 6:33 AM

Yes, it has moved to Gitlab so now I cannot upload to it. :(

Aklapper added a comment.Oct 7 2022, 7:39 AM

@Hawkeye7: Again, this is unrelated. It's off-topic for this task. See T320216 instead. Thanks.

Hawkeye7 added a comment.Oct 7 2022, 10:20 AM

Does the move to gitlab mean that we can use ssh again instead of https?

Aklapper added a comment.Oct 7 2022, 11:23 AM

See https://www.mediawiki.org/wiki/GitLab/Workflows

Dzahn added a comment.Oct 7 2022, 6:00 PM

You can use either ssh or https with either Gerrit or Gitlab. What's going away is repos hosted on Phabricator.

Stashbot added a comment.Oct 11 2022, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-11T20:27:23Z] <mutante> depooling git-ssh service backends with confctl - T296022

Stashbot added a comment.Oct 11 2022, 9:36 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-11T21:36:53Z] <mutante> phab1001 / phab2001 - temp. disabled puppet; stopped ssh-phab service; scheduled icinga downtimes for ssh-phab pybal backend alerts - effectively "soft shutting down" the service - T296022

gerritbot added a comment.Oct 11 2022, 9:41 PM

Change 841587 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: stop ssh-phab service

https://gerrit.wikimedia.org/r/841587

gerritbot added a comment.Oct 11 2022, 9:42 PM

Change 841587 merged by Dzahn:

[operations/puppet@production] phabricator: stop ssh-phab service

https://gerrit.wikimedia.org/r/841587

gerritbot added a comment.Oct 17 2022, 6:01 PM

Change 843522 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] remove git-ssh from common/service.yaml

https://gerrit.wikimedia.org/r/843522

Dzahn added a comment.Oct 17 2022, 6:40 PM

shutdown was announced in today's SRE meeting

gerritbot added a comment.Oct 17 2022, 6:53 PM

Change 843567 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] conftool-data: remove phabricator / git-ssh

https://gerrit.wikimedia.org/r/843567

gerritbot added a comment.Oct 17 2022, 6:55 PM

Change 843567 merged by Dzahn:

[operations/puppet@production] conftool-data: remove phabricator / git-ssh

https://gerrit.wikimedia.org/r/843567

Jelto awarded a token.Oct 18 2022, 8:20 AM
gerritbot added a comment.Oct 19 2022, 8:14 PM

Change 843522 merged by Dzahn:

[operations/puppet@production] remove git-ssh from common/service.yaml

https://gerrit.wikimedia.org/r/843522

Stashbot added a comment.Oct 19 2022, 8:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-19T20:38:24Z] <mutante> puppetmaster1001/puppetmaster2001 - delete .git-*.err files in /var/run/confd-template T296022

Stashbot added a comment.Oct 19 2022, 8:44 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-19T20:44:55Z] <mutante> lvs1020, lvs1018 - systemctl restart pybal.service ; ipvsadm -Dt '208.80.154.250:22' ; ipvsadm -Dt '[2620:0:861:ed1a::3:16]:22' - T296022

Stashbot added a comment.Oct 19 2022, 8:49 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-19T20:49:35Z] <mutante> lvs2010, lvs2008 - systemctl restart pybal.service ; ipvsadm -Dt '208.80.153.250:22' ; ipvsadm -Dt '[2620:0:860:ed1a::3:fa]:22' - T296022

Dzahn added a comment.Oct 19 2022, 9:08 PM

service has actually been removed from LVS/pybal/conftool now. (thanks bblack for help!)

pending DNS removal, then completed

gerritbot added a comment.Oct 20 2022, 4:16 PM

Change 831627 merged by Dzahn:

[operations/dns@master] remove git-ssh.wikimedia.org

https://gerrit.wikimedia.org/r/831627

Stashbot added a comment.Oct 20 2022, 4:20 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-20T16:20:56Z] <mutante> phab1001 (phabricator) - remove LVS IP from loopback - ip addr del 208.80.154.250 dev lo - T296022

Stashbot added a comment.Oct 20 2022, 5:46 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-20T17:46:13Z] <mutante> phabricator - disabling git-ssh URIs for repo 'phabricator-translations' https://phabricator.wikimedia.org/source/phabricator-translation - T296022

Dzahn closed this task as Resolved.Oct 20 2022, 6:06 PM
hashar mentioned this in T321350: Phabricator translations can no more be received due to decommissioning of git-ssh.Oct 20 2022, 8:07 PM
Dzahn added a subtask: T321350: Phabricator translations can no more be received due to decommissioning of git-ssh.Oct 20 2022, 8:14 PM
Stashbot mentioned this in T322250: decom phab2001 (service owner).Nov 10 2022, 7:54 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T19:54:44Z] <mutante> netbox - deleting special case phab2001-vcs.codfw.wmnet IPv4 (10.192.32.149) and IPv6 (2620:0:860:103:10:192:32:149) - T296022 - T322250

abi_ changed the status of subtask T321350: Phabricator translations can no more be received due to decommissioning of git-ssh from Open to In Progress.Nov 28 2022, 5:19 AM
Stashbot added a comment.Dec 5 2022, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2022-12-05T21:59:32Z] <mutante> deleting special DNS entries for "phab10010-vcs.eqiad.wmnet", IPv4 and IPv6 (Role: VIP), from netbox and syncing netbox data - T296022

Nikerabbit closed subtask T321350: Phabricator translations can no more be received due to decommissioning of git-ssh as Resolved.Dec 8 2022, 12:44 PM
gerritbot added a comment.Jan 5 2023, 12:08 AM

Change 875450 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: remove vcs_addresses parameter and warnings

https://gerrit.wikimedia.org/r/875450

gerritbot added a comment.Jan 5 2023, 5:56 PM

Change 875450 merged by Dzahn:

[operations/puppet@production] phabricator: remove vcs_addresses parameter and warnings

https://gerrit.wikimedia.org/r/875450

gerritbot added a comment.Jan 5 2023, 6:05 PM

Change 875987 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::vcs: comment out warning about empty listen address

https://gerrit.wikimedia.org/r/875987

gerritbot added a comment.Jan 5 2023, 6:13 PM

Change 875987 merged by Dzahn:

[operations/puppet@production] phabricator::vcs: comment out warning about empty listen address

https://gerrit.wikimedia.org/r/875987

Aklapper mentioned this in T347408: Do not expose dead git-ssh.wikimedia.org service as repo Clone URLs (defined in diffusion.ssh-host setting).Sep 26 2023, 3:23 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 26 2023, 3:32 PM
Aklapper mentioned this in T347577: Understand which repositories we mirror, observe, host in Diffusion (and fix some findings).Oct 24 2023, 8:03 PM
Aklapper mentioned this in T244907: Reduce the number of six default URIs in Diffusion.Oct 25 2023, 9:26 AM
Aklapper added a subtask: T301170: Change from ssh to pushing via https for "releng-secrets" git repo.Nov 17 2023, 7:56 PM
Aklapper added a comment.Dec 17 2023, 9:28 AM

The operations/puppet repo still has

./modules/profile/manifests/phabricator/main.pp:         { 'default_value' => 'git-ssh.wikimedia.org' }),
./modules/profile/manifests/phabricator/main.pp:    # This exists to offer git services at git-ssh.wikimedia.org.
./hieradata/cloud/eqiad1/devtools/hosts/phabricator-prod-1001.yaml:# all other IPs are used by phabricator::vcs (aka git-ssh.wikimedia.org)

@Dzahn: If you know, should these entries be removed?

Dzahn added a comment.EditedDec 18 2023, 5:12 PM

@Aklapper Yea, weak "yes". The cloud one, sure, just a comment and not relevant anymore. The other one, we can, but then we need to remove the entire "$enable_vcs" parameter and support. so also:

modules/phabricator/manifests/init.pp:    Boolean                 $enable_vcs         = undef,
modules/phabricator/manifests/init.pp:    if $enable_vcs {

..and then there is more..like $vcs_ip_v4 and $vcs_ip_v6 and $use_lvs...

..and after that you find deployment vars like "vcs" database name and user.. and then need a deployment...

also the entire "phabricator::vcs" class is useless if we do that

Dzahn added a comment.Dec 18 2023, 5:17 PM

it never ends:

data/fixed_settings.yaml:diffusion.ssh-user: 'vcs'
files/phab_deploy_config_deploy.sh:sudo chgrp phd "$SCAP_REV_PATH"/phabricator/conf/local/vcs.json
templates/vcs/phabricator-ssh-hook.sh.erb:VCSUSER=<%= @vcs_user %>
templates/vcs/sshd_config.phabricator.erb:AuthorizedKeysCommandUser <%= @vcs_user %>
templates/vcs/sshd_config.phabricator.erb:AllowUsers <%= @vcs_user %>
gerritbot added a comment.Dec 18 2023, 5:19 PM

Change 983871 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] devtools/phabricator: remove comment about git-ssh vcs service

https://gerrit.wikimedia.org/r/983871

gerritbot added a project: Patch-For-Review.Dec 18 2023, 5:19 PM
gerritbot added a comment.Dec 18 2023, 5:22 PM

Change 983871 merged by Dzahn:

[operations/puppet@production] devtools/phabricator: remove comment about git-ssh vcs service

https://gerrit.wikimedia.org/r/983871

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2023, 5:30 PM
Aklapper added a comment.Dec 18 2023, 11:52 PM

@Dzahn: Eh, sorry if I opened a can of worms. Feel free to leave as-is then?

Dzahn added a comment.Dec 19 2023, 12:05 AM

@Aklapper Well... let me take a look, something in between. We don't have to delete the entire class, but we should clean this thing up:

hieradata/role/eqiad/phabricator.yaml:phabricator::vcs::enable: true

because it makes no sense to have that set only if in eqiad.

And I want to check now if there is any change when removing that.

gerritbot added a comment.Dec 19 2023, 12:07 AM

Change 983955 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: remove enable_vcs parameter set in eqiad-only

https://gerrit.wikimedia.org/r/983955

gerritbot added a project: Patch-For-Review.Dec 19 2023, 12:07 AM
gerritbot added a comment.Dec 19 2023, 5:51 PM

Change 983957 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: remove support for separate git-ssh IP behind LVS

https://gerrit.wikimedia.org/r/983957

gerritbot added a comment.Dec 19 2023, 6:28 PM

Change 983958 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: remove sshd config for git-ssh service

https://gerrit.wikimedia.org/r/983958

gerritbot added a comment.Dec 19 2023, 8:12 PM

Change 983957 merged by Dzahn:

[operations/puppet@production] phabricator: remove support for separate git-ssh IP behind LVS

https://gerrit.wikimedia.org/r/983957

gerritbot added a comment.Dec 19 2023, 8:16 PM

Change 983958 merged by Dzahn:

[operations/puppet@production] phabricator: remove sshd config for git-ssh service

https://gerrit.wikimedia.org/r/983958

Dzahn added a comment.Dec 19 2023, 8:18 PM
In T296022#9411591, @Aklapper wrote:

The operations/puppet repo still has

./modules/profile/manifests/phabricator/main.pp:         { 'default_value' => 'git-ssh.wikimedia.org' }),
./modules/profile/manifests/phabricator/main.pp:    # This exists to offer git services at git-ssh.wikimedia.org.
./hieradata/cloud/eqiad1/devtools/hosts/phabricator-prod-1001.yaml:# all other IPs are used by phabricator::vcs (aka git-ssh.wikimedia.org)

@Dzahn: If you know, should these entries be removed?

@Aklapper Amended to patches, talked to Brennen about it, merged.. complete noop confirmed. The stuff you asked about is gone now.done

gerritbot added a comment.Jan 2 2024, 9:04 PM

Change 983955 merged by Dzahn:

[operations/puppet@production] phabricator: move enable_vcs parameter from eqiad-only to role

https://gerrit.wikimedia.org/r/983955

Maintenance_bot removed a project: Patch-For-Review.Jan 2 2024, 9:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL