Page MenuHomePhabricator
Create Task
Maniphest T347624

Refactor sre.wdqs.data-transfer to use new spicerack class api
Closed, ResolvedPublic
Actions

Description

See https://doc.wikimedia.org/spicerack/master/introduction.html#class-interface

The most tangible benefit is getting better logging via overriding the runtime_description property.

AC

  • Cookbook split into CookbookRunner and CookbookRunnerBase classes
  • New runtime_description that includes phab ticket, args called with, etc

Details

SubjectRepoBranchLines +/-
wdqs.data-transfer: fix netbox object not callableoperations/cookbooksmaster+4 -4
wdqs.data_transfer: refactor spicerack class apioperations/cookbooksmaster+178 -133
Customize query in gerrit

Event Timeline

RKemper created this task.Sep 28 2023, 7:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 28 2023, 7:04 PM
RKemper moved this task from Incoming to In Progress on the Data-Platform-SRE board.Sep 28 2023, 7:04 PM
gerritbot added a comment.Sep 28 2023, 7:05 PM

Change 961878 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] wdqs.data_transfer: refactor spicerack class api

https://gerrit.wikimedia.org/r/961878

gerritbot added a project: Patch-For-Review.Sep 28 2023, 7:05 PM
Stashbot added a comment.Sep 28 2023, 7:55 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-28T19:55:43Z] <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer (T347624, testing new cookbook changes) xfer categories => wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling both afterwards

Stashbot added a comment.Sep 28 2023, 8:04 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-28T20:04:29Z] <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T347624, testing new cookbook changes) xfer categories => wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling both afterwards

Stashbot added a comment.Oct 2 2023, 4:30 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-02T16:30:29Z] <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer (T347624, testing new cookbook changes) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling both afterwards w/ encryption

Stashbot added a comment.Oct 2 2023, 4:39 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-02T16:39:15Z] <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T347624, testing new cookbook changes) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling both afterwards w/ encryption

Gehel triaged this task as Medium priority.Nov 3 2023, 10:28 AM
Gehel edited projects, added Data-Platform-SRE (2023.12.01 - 2023.12.31); removed Patch-For-Review, Data-Platform-SRE.Dec 6 2023, 9:27 AM
Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.Dec 6 2023, 9:28 AM
Volans subscribed.Dec 6 2023, 9:30 AM
Gehel edited projects, added Data-Platform-SRE (2024.01.01 - 2024.01.21); removed Data-Platform-SRE (2023.12.01 - 2023.12.31).Dec 31 2023, 9:09 PM
Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.01.01 - 2024.01.21) board.Dec 31 2023, 9:10 PM
Gehel edited projects, added Data-Platform-SRE (2024.01.22 - 2024.02.11); removed Data-Platform-SRE (2024.01.01 - 2024.01.21).Jan 22 2024, 1:42 PM
Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.01.22 - 2024.02.11) board.Jan 22 2024, 1:43 PM
RKemper moved this task from In Progress to Needs Review on the Data-Platform-SRE (2024.01.22 - 2024.02.11) board.Jan 22 2024, 7:47 PM
Stashbot added a comment.Jan 23 2024, 7:49 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T19:49:53Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T347624, test data xfer) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling both afterwards

Stashbot added a comment.Jan 23 2024, 7:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T19:57:36Z] <bking@cumin2002> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T347624, test data xfer) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet, repooling both afterwards

Stashbot added a comment.Jan 23 2024, 7:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T19:57:54Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T347624, test data xfer) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Jan 23 2024, 8:08 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T20:08:56Z] <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T347624, test data xfer) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Jan 23 2024, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T20:12:08Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T347624, test data xfer) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Jan 23 2024, 8:23 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-23T20:23:12Z] <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T347624, test data xfer) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Feb 7 2024, 10:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T22:46:12Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.Feb 7 2024, 10:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T22:46:32Z] <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Gehel edited projects, added Data-Platform-SRE (2024.02.12 - 2024.03.03); removed Data-Platform-SRE (2024.01.22 - 2024.02.11).Feb 9 2024, 10:45 AM
Gehel moved this task from Backlog to Needs Review on the Data-Platform-SRE (2024.02.12 - 2024.03.03) board.
Stashbot added a comment.Feb 20 2024, 7:48 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-20T19:48:32Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.Feb 20 2024, 7:48 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-20T19:48:38Z] <ryankemper@cumin2002> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.Feb 27 2024, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-27T19:36:10Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.Feb 27 2024, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-27T19:47:22Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

gerritbot added a comment.Feb 27 2024, 7:51 PM

Change 961878 merged by Ryan Kemper:

[operations/cookbooks@master] wdqs.data_transfer: refactor spicerack class api

https://gerrit.wikimedia.org/r/961878

Gehel edited projects, added Data-Platform-SRE (2024.03.04 - 2024.03.24); removed Data-Platform-SRE (2024.02.12 - 2024.03.03).Mar 1 2024, 3:34 PM
Gehel moved this task from Backlog to Needs Review on the Data-Platform-SRE (2024.03.04 - 2024.03.24) board.Mar 1 2024, 3:35 PM
bking closed this task as Resolved.Mar 1 2024, 3:48 PM
bking moved this task from Needs Review to Done on the Data-Platform-SRE (2024.03.04 - 2024.03.24) board.

Per Gerrit patches and our test, I believe this is done. Closing...

gerritbot added a comment.Thu, Apr 18, 10:52 PM

Change #1021588 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] wdqs.data-transfer: fix netbox object not callable

https://gerrit.wikimedia.org/r/1021588

gerritbot added a project: Patch-For-Review.Thu, Apr 18, 10:52 PM
gerritbot added a comment.Fri, Apr 19, 6:29 PM

Change #1021588 merged by Ryan Kemper:

[operations/cookbooks@master] wdqs.data-transfer: fix netbox object not callable

https://gerrit.wikimedia.org/r/1021588

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL