Page MenuHomePhabricator
Create Task
Maniphest T319401

Improve how we run WMCS cookbooks
Closed, ResolvedPublic
Actions

Description

WMCS cookbooks are currently run by the WMCS team from their own laptops. We would like to run them from a Cumin server in a similar way to production cookbooks.

This is a tracking task of the agreed work between the Cloud-Services team and the Infrastructure-Foundations one.

I've grouped the work based on affected areas. We can open subtasks as needed.

✅ Improve the laptop local environment

T319426 Add ssh socks5 proxy support (blocks removing duplication) (1 day)

✅ New WMCS cookbooks repository

T319436 Import into the new repo the code, splitting libs from cookbooks (½ day)

✅ Setup the production infrastructure

T323516, T323518 Create cloud cumin ganeti hosts, 2 VMs, one per DC (½ day)

  • Setup dedicated SSH config for the double jump in cloud (½ day) (patch)
✅ Spicerack improvements
  • Add module injection support (2 days) (patch)
  • Add register of accessors support (2 days) (patch)
  • Add sudo everywhere (1w, might be less) [not needed for the current setup]

T325168 Load cookbooks from multiple directories (2 days)

✅ Setup the Cloud infrastructure

T323483 Define which SSH key to use to SSH from the new cloud-cumin to hosts
T323484 Fine tune the SSHd config of the restricted bastion for better performances (½ day)

✅ Misc

T325756 Allow wmcs cookbooks running on cloudcuminXXXX to write to the SAL (1~2w)
T325754 Update Spicerack documentation

Postponed to later

We could not complete the following tasks as part of this epic, and they can be postponed as they are not a blocker for the main goal of running WMCS cookbooks from the new cloudcumin hosts.

T319438 Remove code duplication (alertmanager) (1w)
T319450 Move the libs to spicerack modules (2w this might be really easy though, depending on the module solution)
T325067 Decide sudoers rules for users without global root
T325758 Spicerack: Add CI step to test with wmcs cookbooks (1w)
T322511 [spicerack][alertmanager] support silencing alerts without instance label

Details

SubjectRepoBranchLines +/-
cloud cumin: fix ssh config for codf1dev bastionoperations/puppetproduction+1 -1
cloudcumin: fix hieradata for codfw1dev bastionoperations/puppetproduction+0 -0
cloudcumin: add FQDN of the eqiad1 bastionoperations/puppetproduction+1 -1
cloud: authorize cumin from the bastionoperations/puppetproduction+4 -0
cloud cumin: fix authorized keys for cuminoperations/puppetproduction+7 -1
cumin:cloud_master: fix ssh_config for bastionsoperations/puppetproduction+2 -0
cumin::cloud_master: configure openstack backendoperations/puppetproduction+35 -4
cumin::cloud_master: add openstack dependenciesoperations/puppetproduction+5 -0
cloudcumin: improve ssh configoperations/puppetproduction+4 -1
cloudcumin: use the webproxy to connect to Cloudoperations/puppetproduction+1 -5
P:installserver::proxy: add ability to proxy ssh portsoperations/puppetproduction+30 -4
cr-labs: allow SSH from the cloudcumin_groupoperations/homer/publicmaster+1 -1
cloudcumin: actually allow ssh from the mastersoperations/puppetproduction+6 -0
cloudcumin: use the puppetdb microserviceoperations/puppetproduction+5 -1
base::cloud_production: introduce new profileoperations/puppetproduction+82 -15
cumin::cloud_master: introduce new profileoperations/puppetproduction+125 -18
cumin::cloud_target: add a new profileoperations/puppetproduction+24 -0
cloud_management: re-add datacenter-opsoperations/puppetproduction+1 -0
cloud_management: add missing wikimedia_clustersoperations/puppetproduction+5 -0
cloud-cumin: set the installer to use bullseyeoperations/puppetproduction+2 -0
spicerack: add module injection supportoperations/software/spicerackmaster+204 -12
cloudcumin: setup the 2 new VMsoperations/puppetproduction+20 -0
cluster::cloud_management: create new roleoperations/puppetproduction+48 -3
sre.dns.wipe-cache: add sudo to the commandoperations/cookbooksmaster+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fnegri changed the task status from Open to In Progress.Dec 14 2022, 3:04 PM
fnegri updated the task description. (Show Details)
Volans mentioned this in rCCKB8fd38ff03bf4: sre.dns.wipe-cache: add sudo to the command.Dec 14 2022, 3:31 PM
gerritbot added a comment.Dec 14 2022, 4:10 PM

Change 867551 merged by Volans:

[operations/puppet@production] cumin::cloud_master: introduce new profile

https://gerrit.wikimedia.org/r/867551

fnegri updated the task description. (Show Details)Dec 14 2022, 4:41 PM
fnegri closed subtask T323516: eqiad: (1) VM request for cloudcumin1001 as Resolved.Dec 14 2022, 4:44 PM
fnegri closed subtask T323518: codfw: (1) VM request for cumincloud2001 as Resolved.
fnegri closed subtask T323483: Define which SSH key to use to SSH from the new cloud-cumin to hosts as Resolved.Dec 14 2022, 4:47 PM
gerritbot added a comment.Dec 15 2022, 12:02 PM

Change 868372 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] P:installserver::proxy: add ability to proxy ssh ports

https://gerrit.wikimedia.org/r/868372

gerritbot added a comment.Dec 16 2022, 10:11 AM

Change 867169 merged by Volans:

[operations/puppet@production] base::cloud_production: introduce new profile

https://gerrit.wikimedia.org/r/867169

gerritbot added a comment.Dec 16 2022, 10:38 AM

Change 868632 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloudcumin: use the puppetdb microservice

https://gerrit.wikimedia.org/r/868632

gerritbot added a comment.Dec 16 2022, 10:59 AM

Change 868636 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloudcumin: actually allow ssh from the masters

https://gerrit.wikimedia.org/r/868636

gerritbot added a comment.Dec 16 2022, 11:26 AM

Change 868632 merged by Volans:

[operations/puppet@production] cloudcumin: use the puppetdb microservice

https://gerrit.wikimedia.org/r/868632

gerritbot added a comment.Dec 16 2022, 11:36 AM

Change 868636 merged by Volans:

[operations/puppet@production] cloudcumin: actually allow ssh from the masters

https://gerrit.wikimedia.org/r/868636

gerritbot added a comment.Dec 16 2022, 11:48 AM

Change 868646 had a related patch set uploaded (by Volans; author: Volans):

[operations/homer/public@master] cr-labs: allow SSH from the cloudcumin_group

https://gerrit.wikimedia.org/r/868646

gerritbot added a comment.Dec 16 2022, 12:01 PM

Change 868646 merged by jenkins-bot:

[operations/homer/public@master] cr-labs: allow SSH from the cloudcumin_group

https://gerrit.wikimedia.org/r/868646

gerritbot added a comment.Dec 16 2022, 12:29 PM

Change 868372 merged by Jbond:

[operations/puppet@production] P:installserver::proxy: add ability to proxy ssh ports

https://gerrit.wikimedia.org/r/868372

gerritbot added a comment.Dec 16 2022, 1:52 PM

Change 868673 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloudcumin: use the webproxy to connect to Cloud

https://gerrit.wikimedia.org/r/868673

gerritbot added a comment.Dec 16 2022, 3:37 PM

Change 868673 merged by Volans:

[operations/puppet@production] cloudcumin: use the webproxy to connect to Cloud

https://gerrit.wikimedia.org/r/868673

gerritbot added a comment.Dec 16 2022, 4:20 PM

Change 868720 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloudcumin: improve ssh config

https://gerrit.wikimedia.org/r/868720

gerritbot added a comment.Dec 16 2022, 4:36 PM

Change 868720 merged by Volans:

[operations/puppet@production] cloudcumin: improve ssh config

https://gerrit.wikimedia.org/r/868720

Stashbot added a comment.Dec 16 2022, 7:36 PM

Mentioned in SAL (#wikimedia-cloud) [2022-12-16T19:36:40Z] <volans> restarted sshd twice on bastion-restricted-eqiad1-02 to debug SSH connections for T319401

Volans updated the task description. (Show Details)Dec 16 2022, 10:52 PM
gerritbot added a comment.Dec 19 2022, 9:33 AM

Change 869173 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cumin::cloud_master: add openstack dependencies

https://gerrit.wikimedia.org/r/869173

gerritbot added a comment.Dec 19 2022, 12:17 PM

Change 869173 merged by Volans:

[operations/puppet@production] cumin::cloud_master: add openstack dependencies

https://gerrit.wikimedia.org/r/869173

gerritbot added a comment.Dec 19 2022, 1:05 PM

Change 869212 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cumin::cloud_master: configure openstack backend

https://gerrit.wikimedia.org/r/869212

Volans closed subtask T325168: Spicerack: Load cookbooks from multiple directories as Resolved.Dec 19 2022, 2:52 PM
gerritbot added a comment.Dec 19 2022, 4:34 PM

Change 869212 merged by Volans:

[operations/puppet@production] cumin::cloud_master: configure openstack backend

https://gerrit.wikimedia.org/r/869212

gerritbot added a comment.Dec 19 2022, 4:48 PM

Change 869245 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cumin:cloud_master: fix ssh_config for bastions

https://gerrit.wikimedia.org/r/869245

gerritbot added a comment.Dec 19 2022, 4:51 PM

Change 869245 merged by Volans:

[operations/puppet@production] cumin:cloud_master: fix ssh_config for bastions

https://gerrit.wikimedia.org/r/869245

gerritbot added a comment.Dec 19 2022, 4:58 PM

Change 869268 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloud cumin: fix authorized keys for cumin

https://gerrit.wikimedia.org/r/869268

gerritbot added a comment.Dec 19 2022, 5:26 PM

Change 869268 merged by Volans:

[operations/puppet@production] cloud cumin: fix authorized keys for cumin

https://gerrit.wikimedia.org/r/869268

gerritbot added a comment.Dec 19 2022, 6:26 PM

Change 869278 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloud: authorize cumin from the bastion

https://gerrit.wikimedia.org/r/869278

gerritbot added a comment.Dec 20 2022, 1:45 PM

Change 869278 merged by Volans:

[operations/puppet@production] cloud: authorize cumin from the bastion

https://gerrit.wikimedia.org/r/869278

gerritbot added a comment.Dec 20 2022, 3:21 PM

Change 869779 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloudcumin: add FQDN of the eqiad1 bastion

https://gerrit.wikimedia.org/r/869779

gerritbot added a comment.Dec 20 2022, 3:28 PM

Change 869779 merged by Volans:

[operations/puppet@production] cloudcumin: add FQDN of the eqiad1 bastion

https://gerrit.wikimedia.org/r/869779

gerritbot added a comment.Dec 20 2022, 3:40 PM

Change 869782 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloudcumin: fix hieradata for codfw1dev bastion

https://gerrit.wikimedia.org/r/869782

gerritbot added a comment.Dec 20 2022, 3:42 PM

Change 869782 merged by Volans:

[operations/puppet@production] cloudcumin: fix hieradata for codfw1dev bastion

https://gerrit.wikimedia.org/r/869782

gerritbot added a comment.Dec 20 2022, 4:26 PM

Change 869816 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cloud cumin: fix ssh config for codf1dev bastion

https://gerrit.wikimedia.org/r/869816

Volans updated the task description. (Show Details)Dec 20 2022, 4:47 PM
fnegri updated the task description. (Show Details)Dec 21 2022, 4:20 PM
fnegri updated the task description. (Show Details)Dec 21 2022, 4:54 PM
fnegri updated the task description. (Show Details)Dec 21 2022, 4:58 PM
dcaro closed subtask T319436: [cookbooks] Import wmcs branch into the new repo the code, splitting libs from cookbooks as Resolved.Dec 22 2022, 8:14 AM
gerritbot added a comment.Dec 22 2022, 8:34 AM

Change 869167 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] Add moved to wmcs-cookbooks message.

https://gerrit.wikimedia.org/r/869167

fnegri moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Dec 23 2022, 10:04 AM
• nskaggs awarded a token.Jan 11 2023, 10:09 PM
gerritbot added a comment.Jan 12 2023, 12:29 PM

Change 869816 merged by Volans:

[operations/puppet@production] cloud cumin: fix ssh config for codf1dev bastion

https://gerrit.wikimedia.org/r/869816

fnegri changed the status of subtask T323484: Fine tune the SSHd config of the restricted bastion for better performances from Open to In Progress.Jan 13 2023, 6:06 PM
fnegri mentioned this in T326978: [wmcs-cookbooks] Tidy up the new repo.Jan 13 2023, 6:27 PM
fnegri removed subtasks: T319450: [cookbooks] Refactor the specific wmcs libraries into spicerack module, T319438: [cookbooks] Remove all the duplicated code now that we can use spicerack one.
fnegri removed a project: Patch-For-Review.Jan 13 2023, 6:36 PM
fnegri updated the task description. (Show Details)
fnegri removed subtasks: T325758: Spicerack: Add CI step to test with wmcs cookbooks, T325067: cloudcumin: decide sudoers rules for users without global root, T322511: [spicerack][alertmanager] support silencing alerts without instance label.
fnegri updated the task description. (Show Details)
fnegri removed a subtask: Restricted Task.Jan 13 2023, 6:39 PM
fnegri added a comment.Jan 13 2023, 6:49 PM

As Q2 is now over, I suggest that we consider this first iteration complete as soon as we can successfully run WMCS cookbooks from the new cloudcumin hosts. Everything else can be tracked in separate tasks/epics.

I went for a WP:BOLD approach and modified the task description, moving all subtasks that are not strictly blockers to a new section "Postponed to later". Let me know if you disagree and feel free to suggest alternative plans. :)

Aklapper renamed this task from WMCS Cookbook Automation Q2 tracking task to WMCS Cookbook Automation FY2022-23 Q2 tracking task.Jan 13 2023, 8:17 PM
fnegri edited projects, added cloud-services-team (FY2022/2023-Q3); removed cloud-services-team (Kanban).Jan 18 2023, 6:36 PM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2022/2023-Q3) board.
fnegri raised the priority of this task from Medium to High.Apr 12 2023, 2:50 PM
fnegri moved this task from FY2022/2023-Q3 to FY2022/2023-Q4 on the cloud-services-team board.
fnegri edited projects, added cloud-services-team (FY2022/2023-Q4); removed cloud-services-team (FY2022/2023-Q3).
fnegri moved this task from Backlog to Planned Goals on the cloud-services-team (FY2022/2023-Q4) board.
fnegri claimed this task.Apr 14 2023, 1:37 PM
bking subscribed.Apr 14 2023, 1:38 PM
fnegri added a project: Goal.Apr 17 2023, 3:08 PM
fnegri moved this task from Planned Goals to In progress on the cloud-services-team (FY2022/2023-Q4) board.Apr 19 2023, 2:25 PM
fnegri closed subtask T323484: Fine tune the SSHd config of the restricted bastion for better performances as Resolved.Jun 27 2023, 12:03 PM
fnegri changed the status of subtask T325756: Allow wmcs cookbooks running on cloudcuminXXXX to write to the SAL from Open to In Progress.Jun 27 2023, 12:20 PM
fnegri renamed this task from WMCS Cookbook Automation FY2022-23 Q2 tracking task to Improve how we run WMCS cookbooks.Jul 10 2023, 5:31 PM
fnegri updated the task description. (Show Details)
fnegri updated the task description. (Show Details)
fnegri edited projects, added cloud-services-team (FY2023/2024-Q1-Q2); removed cloud-services-team (FY2022/2023-Q4).Jul 26 2023, 4:04 PM
fnegri added a parent task: T325067: cloudcumin: decide sudoers rules for users without global root.Jul 26 2023, 4:06 PM
fnegri mentioned this in T325067: cloudcumin: decide sudoers rules for users without global root.Jul 26 2023, 4:09 PM
fnegri moved this task from Backlog to Planned Goals on the cloud-services-team (FY2023/2024-Q1-Q2) board.Jul 26 2023, 4:11 PM
fnegri moved this task from Planned Goals to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.Jul 26 2023, 5:47 PM
fnegri moved this task from In progress to Planned Goals on the cloud-services-team (FY2023/2024-Q1-Q2) board.
fnegri mentioned this in rCCKBb8b571884f0d: Add project name to Spicerack's sal_logger.Jul 31 2023, 2:41 PM
fnegri removed a parent task: T325067: cloudcumin: decide sudoers rules for users without global root.Aug 2 2023, 1:40 PM
fnegri closed subtask T325756: Allow wmcs cookbooks running on cloudcuminXXXX to write to the SAL as Resolved.Aug 2 2023, 2:24 PM
fnegri added a parent task: T276440: wmcs.spicerack: Setup a host to run cookbooks from prod network.Aug 4 2023, 12:13 PM
fnegri mentioned this in T276440: wmcs.spicerack: Setup a host to run cookbooks from prod network.Aug 4 2023, 12:16 PM
fnegri added a project: Cloud-VPS.Aug 4 2023, 12:34 PM
fnegri moved this task from Backlog to wmcs-cookbooks on the Cloud-VPS board.Aug 4 2023, 12:36 PM
fnegri moved this task from Planned Goals to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.Aug 7 2023, 2:53 PM
fnegri changed the status of subtask T325754: Update Spicerack documentation from Open to In Progress.Aug 7 2023, 2:57 PM
fnegri closed subtask T325754: Update Spicerack documentation as Resolved.Aug 9 2023, 2:42 PM
fnegri closed this task as Resolved.Aug 9 2023, 2:46 PM
fnegri added a project: Epic.
fnegri updated the task description. (Show Details)
fnegri moved this task from In progress to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.Aug 10 2023, 9:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL