Page MenuHomePhabricator
Create Task
Maniphest T314665

Toolforge: Replace all bastion with grid-less bookworm based bastion hosts
Closed, ResolvedPublic
Actions

Description

Currently the Toolforge bastion puppetization is fairly tightly coupled with grid-specific components. Since the grid is eventually going away, we'll want to untangle those dependencies to make it possible to have bastion hosts that don't have grid tooling but do have the various CLI tools ('shell environ') and prevent local crontabs.

Details

Other Assignee
taavi
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
inventory: Remove tools-sgebastion-10cloud/wmcs-cookbooksmain+0 -6
P:toolforge: Remove support for grid bastionsoperations/puppetproduction+9 -1 K
P:toolforge: move webservice CLI to the CLI profileoperations/puppetproduction+29 -49
P:toolforge: ensure new bastions have en_US.UTF-8 localeoperations/puppetproduction+21 -13
O:toolforge: add role for grid-less bastionsoperations/puppetproduction+17 -4
P:toolforge: cleanup bastion grid integrationoperations/puppetproduction+78 -61
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Retire login-buster addressrepos/cloud/toolforge/tofu-provisioning!80taavimain-I6cfa18ae24db8546bef01b51886ef45a6192f350main
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRaymond_NdibeT359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
Restricted Task
 ResolvedSlst2020T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
 Resolved• aborreroT316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
 Resolved• aborreroT307651 Upgrade Toolforge Kubernetes to version 1.24
 ResolvedNoneT360699 Toolsbeta: migrate to Debian Bullseye or later
 ResolvedtaaviT311897 [infra] Toolforge: migrate to Debian Bullseye or later
 ResolvedtaaviT314665 Toolforge: Replace all bastion with grid-less bookworm based bastion hosts
 ResolvedtaaviT314664 [infra] Decommission the Grid Engine infrastructure
 ResolvedkomlaT313405 Migrate remaining tools off Gridengine
 DeclinedNoneT316124 Make Grid Engine tooling emit deprecation warnings
 ResolvedtaaviT329619 Toolforge: decide what to do with tools that have 'stretch' in their service manifests
 ResolvedAndrewT353351 Provide tools for disabling the grid for specific tools
 ResolvedAndrewT353642 Relocate disable-tool-archive-dbs.service
 ResolvedtaaviT358333 Remove toolschecker grid engine checks
 ResolvedtaaviT359934 [infra] Archive grid engine related infrastructure tools
 ResolvedtaaviT359935 [infra] Archive grid engine related Gerrit repositories
 StalleddcaroT356377 [toolforge] simplify calling the different toolforge apis from within the containers
 StalledNoneT356262 [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs
 OpenNoneT356261 [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation
 OpenNoneT398651 [toolforge-cli-gen] review the https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli client as potential consolidation
 ResolvedSlst2020T354745 [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client
 ResolvedSlst2020T358100 [toolforge API] expose all backend APIs OpenAPI specs
 Resolved• aborreroT356523 [jobs-api] introduce OpenAPI to jobs framework
 ResolvedSlst2020T362299 [api-gateway] Add a python server to serve consolidated openapi docs
 DeclinedNoneT369693 [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images
 ResolvedBUG REPORTdcaroT369569 `webservice` requires effective user to be the tool user and listed in NSS passwd data
 ResolvedBUG REPORTdcaroT369573 `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data
 ResolvedBUG REPORTNoneT360488 Missing Perl packages on dev.toolforge.org for anomiebot workflows
 ResolveddcaroT371556 Please re-install "joe"
 ResolvedtaaviT397459 Lock down tools-sgebastion-10 (login-buster.toolforge.org) to only members of tools with known dependencies on it

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dcaro triaged this task as Medium priority.Feb 16 2024, 3:00 PM
dcaro moved this task from Workspace for triaging whenever needed to Ready to be worked on on the Toolforge board.Feb 21 2024, 4:02 PM
• bd808 subscribed.Mar 9 2024, 7:57 PM
[19:47]  <    bd808> the what to keep question is I think the interesting one. Getting rid of lots of language runtime things seems like it will be a good thing and help actually reduce confusion about bastion vs container execution in the future.
[19:55]  <    bd808> This is a rare opportunity to uninstall a lot of packages and then wait for people to ask for specific things to come back, so maybe we should be picky about what we keep without asking for specific use cases.
taavi mentioned this in T314664: [infra] Decommission the Grid Engine infrastructure.Mar 12 2024, 1:27 PM
taavi removed a parent task: T314664: [infra] Decommission the Grid Engine infrastructure.Mar 13 2024, 12:29 PM
taavi added a subtask: T314664: [infra] Decommission the Grid Engine infrastructure.
taavi added a parent task: T311897: [infra] Toolforge: migrate to Debian Bullseye or later.
taavi changed the status of subtask T314664: [infra] Decommission the Grid Engine infrastructure from Stalled to In Progress.Mar 14 2024, 10:55 AM
gerritbot added a comment.Mar 14 2024, 11:51 AM

Change 990703 merged by Majavah:

[operations/puppet@production] O:toolforge: add role for grid-less bastions

https://gerrit.wikimedia.org/r/990703

Maintenance_bot removed a project: Patch-For-Review.Mar 14 2024, 12:31 PM
gerritbot added a comment.Mar 18 2024, 2:42 PM

Change 1012390 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: move webservice CLI to the CLI profile

https://gerrit.wikimedia.org/r/1012390

gerritbot added a project: Patch-For-Review.Mar 18 2024, 2:42 PM
gerritbot added a comment.Mar 19 2024, 10:16 AM

Change 1012615 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: ensure new bastions have en_US.UTF-8 locale

https://gerrit.wikimedia.org/r/1012615

gerritbot added a comment.Mar 19 2024, 10:45 AM

Change 1012615 merged by Majavah:

[operations/puppet@production] P:toolforge: ensure new bastions have en_US.UTF-8 locale

https://gerrit.wikimedia.org/r/1012615

gerritbot added a comment.Mar 19 2024, 10:49 AM

Change 1012390 merged by Majavah:

[operations/puppet@production] P:toolforge: move webservice CLI to the CLI profile

https://gerrit.wikimedia.org/r/1012390

Stashbot added a comment.Mar 19 2024, 11:19 AM

Mentioned in SAL (#wikimedia-cloud) [2024-03-19T11:19:45Z] <taavi> point dev.toolforge.org to tools-bastion-12 T314665

taavi edited projects, added Toolforge (Toolforge iteration 07); removed Toolforge.Mar 19 2024, 11:21 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 19 2024, 11:30 AM
taavi changed the task status from Open to In Progress.Mar 19 2024, 2:07 PM
taavi moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 07) board.
gerritbot added a comment.Mar 19 2024, 9:12 PM

Change 1012752 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: remove support for grid bastions

https://gerrit.wikimedia.org/r/1012752

gerritbot added a project: Patch-For-Review.Mar 19 2024, 9:12 PM
taavi reassigned this task from taavi to dcaro.Apr 2 2024, 1:43 PM
taavi updated Other Assignee, added: taavi.
dcaro added a project: User-dcaro.Apr 3 2024, 8:20 AM
dcaro edited projects, added Toolforge (Toolforge iteration 08); removed Toolforge (Toolforge iteration 07).Apr 3 2024, 4:15 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 08) board.
dcaro edited projects, added Toolforge (Toolforge iteration 09); removed Toolforge (Toolforge iteration 08).Apr 17 2024, 7:48 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 09) board.
dcaro changed the task status from In Progress to Stalled.Apr 17 2024, 2:08 PM
dcaro moved this task from In Progress to Blocked/Paused on the Toolforge (Toolforge iteration 09) board.
dcaro added a comment.Apr 30 2024, 7:48 AM

I'm leaving this open until we remove the old bastions, I'll rephrase the title

dcaro renamed this task from Toolforge: Introduce grid-less bookworm based bastion hosts to Toolforge: Replace all bastion with grid-less bookworm based bastion hosts.Apr 30 2024, 7:49 AM
dcaro changed the status of subtask T314664: [infra] Decommission the Grid Engine infrastructure from In Progress to Stalled.Apr 30 2024, 1:27 PM
dcaro edited projects, added Toolforge (Toolforge iteration 10); removed Toolforge (Toolforge iteration 09).May 22 2024, 4:40 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 10) board.
dcaro edited projects, added Toolforge (Toolforge iteration 11); removed Toolforge (Toolforge iteration 10).Jun 4 2024, 2:55 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 11) board.
dcaro added a subtask: T356377: [toolforge] simplify calling the different toolforge apis from within the containers.Jul 1 2024, 1:50 PM
dcaro edited projects, added Toolforge (Toolforge iteration 12); removed Toolforge (Toolforge iteration 11).Jul 2 2024, 2:01 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 12) board.
dcaro edited projects, added Toolforge (Toolforge iteration 13); removed Toolforge (Toolforge iteration 12).Jul 17 2024, 11:22 AM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 13) board.
fnegri added a subtask: T360488: Missing Perl packages on dev.toolforge.org for anomiebot workflows.EditedJul 18 2024, 4:20 PM
fnegri subscribed.

I've added T360488: Missing Perl packages on dev.toolforge.org for anomiebot workflows as a subtask to remember that anomiebot is currently relying on the login-buster bastion, and if we remove it that tool is likely to break.

• bd808 added a subtask: T371556: Please re-install "joe".Aug 1 2024, 6:11 PM
dcaro edited projects, added Toolforge (Toolforge iteration 14); removed Toolforge (Toolforge iteration 13).Aug 5 2024, 7:39 AM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 14) board.
dcaro closed subtask T371556: Please re-install "joe" as Resolved.Sep 3 2024, 5:22 PM
dcaro edited projects, added Toolforge (Toolforge iteration 15); removed Toolforge (Toolforge iteration 14).Sep 25 2024, 7:55 AM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 15) board.
dcaro edited projects, added Toolforge (Toolforge iteration 16); removed Toolforge (Toolforge iteration 15).Oct 15 2024, 7:32 AM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 16) board.
dcaro edited projects, added Toolforge (Toolforge iteration 17); removed Toolforge (Toolforge iteration 16).Jan 8 2025, 12:48 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 17) board.
dcaro changed the task status from Stalled to In Progress.Jan 21 2025, 2:29 PM
dcaro moved this task from Blocked/Paused to In Progress on the Toolforge (Toolforge iteration 17) board.
dcaro edited projects, added Toolforge (Toolforge iteration 18); removed Toolforge (Toolforge iteration 17).Feb 25 2025, 3:29 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 18) board.
dcaro edited projects, added Toolforge (Toolforge iteration 19); removed Toolforge (Toolforge iteration 18).Mar 18 2025, 3:56 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 19) board.
dcaro edited projects, added Toolforge (Toolforge iteration 20); removed Toolforge (Toolforge iteration 19).May 14 2025, 2:30 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 20) board.
dcaro edited projects, added Toolforge (Toolforge iteration 21); removed Toolforge (Toolforge iteration 20).Jun 11 2025, 10:38 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 21) board.
taavi closed subtask T397459: Lock down tools-sgebastion-10 (login-buster.toolforge.org) to only members of tools with known dependencies on it as Resolved.Jun 23 2025, 9:08 AM
dcaro edited projects, added Toolforge (Toolforge iteration 22); removed Toolforge (Toolforge iteration 21).Jul 14 2025, 9:49 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 22) board.
dcaro edited projects, added Toolforge (Toolforge iteration 23); removed Toolforge (Toolforge iteration 22).Aug 5 2025, 3:45 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 23) board.
dcaro edited projects, added Toolforge (Toolforge iteration 24); removed Toolforge (Toolforge iteration 23).Aug 27 2025, 9:43 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 24) board.
fnegri edited projects, added cloud-services-team (FY2025/26-Q1-Q2); removed cloud-services-team.Sep 5 2025, 8:40 AM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/26-Q1-Q2) board.
taavi claimed this task.Sep 16 2025, 3:23 PM
taavi added a subscriber: dcaro.
Stashbot added a comment.Sep 16 2025, 3:26 PM

Mentioned in SAL (#wikimedia-cloud) [2025-09-16T15:26:20Z] <taavi> shutdown tools-sgebastion-10 T314665

taavi closed subtask T360488: Missing Perl packages on dev.toolforge.org for anomiebot workflows as Resolved.Sep 16 2025, 3:27 PM

I've shut down the bastion, will delete in a few days unless anything urgent pops up.

gerritbot added a comment.Sep 16 2025, 3:28 PM

Change #1012752 merged by Majavah:

[operations/puppet@production] P:toolforge: Remove support for grid bastions

https://gerrit.wikimedia.org/r/1012752

Maintenance_bot removed a project: Patch-For-Review.Sep 16 2025, 3:33 PM
taavi changed the status of subtask T314664: [infra] Decommission the Grid Engine infrastructure from Stalled to Open.Sep 16 2025, 3:44 PM
gerritbot added a comment.Sep 16 2025, 3:44 PM

Change #1188831 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] inventory: Remove tools-sgebastion-10

https://gerrit.wikimedia.org/r/1188831

gerritbot added a project: Patch-For-Review.Sep 16 2025, 3:44 PM
gerritbot added a comment.Sep 16 2025, 3:50 PM

Change #1188831 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] inventory: Remove tools-sgebastion-10

https://gerrit.wikimedia.org/r/1188831

taavi mentioned this in rCCKBafd174543ccc: inventory: Remove tools-sgebastion-10.Sep 16 2025, 3:51 PM
Stashbot added a comment.Sep 16 2025, 3:57 PM

Mentioned in SAL (#wikimedia-cloud) [2025-09-16T15:57:02Z] <taavi> delete tools-sgebastion puppet prefix T314665

CodeReviewBot added a comment.Sep 16 2025, 4:03 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/80

Retire login-buster address

CodeReviewBot added a comment.Sep 18 2025, 6:38 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/80

Retire login-buster address

Stashbot added a comment.Sep 18 2025, 6:47 AM

Mentioned in SAL (#wikimedia-cloud) [2025-09-18T06:46:59Z] <taavi> delete tools-sgebastion-10 T314665

taavi closed this task as Resolved.Sep 18 2025, 6:47 AM
dcaro moved this task from In Progress to Done on the Toolforge (Toolforge iteration 24) board.Sep 18 2025, 1:44 PM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2025/26-Q1-Q2) board.Sep 22 2025, 8:59 AM
taavi closed subtask T314664: [infra] Decommission the Grid Engine infrastructure as Resolved.Sep 23 2025, 3:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits