Page MenuHomePhabricator
Create Task
Maniphest T259516

DBA python layout
Closed, ResolvedPublic
Actions

Description

Current issues:

  • wmfmariadbpy contains some outdated scripts (e.g. check_health.py which is an outdated version of check_mariadb.py in the puppet repo)
  • it also contains some unfinished/unused scripts (e.g. sql.py)
  • it contains some duplication of libraries with transferpy (e.g. CuminExecution)
  • it mixes scripts and libraries in the same directory
  • it mixed backup and db code in the same directory

Proposal:

  • Move all of the remote-execution libraries into wmfmariadbpy, and have transferpy depend on python3-wmfmariadb.
  • Delete unfinished/unused code. Can always be retrieved from git history later if we decide we want to work on it.
  • Separate out code by package; and package-level scripts can live in packagename/cli/ in the repo.
  • Migrate check_mariadb.py to wmfmariadb, and update puppet to use it from there.

Details

SubjectRepoBranchLines +/-
Remove unused sql.py and check_private_data.pyoperations/software/wmfmariadbpymaster+0 -394
Fix references to now removed transferpy.RemoteExecution moduleoperations/software/transferpymaster+1 -2
Move RemoteExecution library to wmfmariadbpyoperations/software/transferpymaster+4 -407
mariadb: Update name of wmfmariadbpy packageoperations/puppetproduction+1 -1
mariadb: Drop check_mariadb.py in favour of packaged versionoperations/puppetproduction+3 -723
Update remote execution libraries from transferpyoperations/software/wmfmariadbpymaster+263 -20
Split utilities into separate packagesoperations/software/wmfmariadbpymaster+69 -51
check_health: update from check_mariadb.py in puppet repooperations/software/wmfmariadbpymaster+164 -29
Customize query in gerrit

Related Objects

Event Timeline

Kormat created this task.Aug 3 2020, 2:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 3 2020, 2:25 PM
Kormat triaged this task as Medium priority.Aug 3 2020, 2:25 PM
• Marostegui moved this task from Triage to Backlog on the DBA board.Aug 4 2020, 5:14 AM
Kormat moved this task from Unsorted 💣 to Active 🚁 on the User-Kormat board.Aug 4 2020, 7:31 AM
gerritbot added a comment.Aug 4 2020, 1:27 PM

Change 618310 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] check_health: update from check_mariadb.py in puppet repo

https://gerrit.wikimedia.org/r/618310

gerritbot added a project: Patch-For-Review.Aug 4 2020, 1:27 PM
gerritbot added a comment.Aug 4 2020, 3:34 PM

Change 618310 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] check_health: update from check_mariadb.py in puppet repo

https://gerrit.wikimedia.org/r/618310

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2020, 4:10 PM
Kormat updated the task description. (Show Details)Aug 5 2020, 12:28 PM
gerritbot added a comment.Aug 5 2020, 12:28 PM

Change 618513 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] Split utilities into separate packages

https://gerrit.wikimedia.org/r/618513

gerritbot added a project: Patch-For-Review.Aug 5 2020, 12:28 PM
gerritbot added a comment.Aug 7 2020, 2:18 PM

Change 618513 merged by Kormat:
[operations/software/wmfmariadbpy@master] Split utilities into separate packages

https://gerrit.wikimedia.org/r/618513

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2020, 3:11 PM
gerritbot added a comment.Aug 11 2020, 9:28 AM

Change 619443 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Drop check_mariadb.py in favour of packaged version

https://gerrit.wikimedia.org/r/619443

gerritbot added a project: Patch-For-Review.Aug 11 2020, 9:28 AM
gerritbot added a comment.Aug 11 2020, 2:14 PM

Change 619476 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] Update remote execution libraries from transferpy

https://gerrit.wikimedia.org/r/619476

gerritbot added a comment.Aug 13 2020, 8:13 AM

Change 619959 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/transferpy@master] Move RemoteExecution library to wmfmariadbpy

https://gerrit.wikimedia.org/r/619959

Kormat claimed this task.Aug 13 2020, 8:22 AM
Kormat moved this task from Backlog to In progress on the DBA board.
gerritbot added a comment.Aug 13 2020, 3:33 PM

Change 619476 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] Update remote execution libraries from transferpy

https://gerrit.wikimedia.org/r/619476

Stashbot added a comment.Aug 18 2020, 1:04 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-18T13:04:19Z] <kormat> disabling puppet on all db machines T259516

gerritbot added a comment.Aug 18 2020, 1:09 PM

Change 619443 merged by Kormat:
[operations/puppet@production] mariadb: Drop check_mariadb.py in favour of packaged version

https://gerrit.wikimedia.org/r/619443

gerritbot added a comment.Aug 18 2020, 1:16 PM

Change 620931 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Update name of wmfmariadbpy package

https://gerrit.wikimedia.org/r/620931

gerritbot added a comment.Aug 18 2020, 1:18 PM

Change 620931 merged by Kormat:
[operations/puppet@production] mariadb: Update name of wmfmariadbpy package

https://gerrit.wikimedia.org/r/620931

Stashbot added a comment.Aug 20 2020, 8:52 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-20T08:52:37Z] <kormat> removing /usr/bin/check_mariadb.py from all db hosts T259516

gerritbot added a comment.Aug 21 2020, 9:33 AM

Change 619959 merged by Jcrespo:
[operations/software/transferpy@master] Move RemoteExecution library to wmfmariadbpy

https://gerrit.wikimedia.org/r/619959

Kormat updated the task description. (Show Details)Aug 21 2020, 9:34 AM
gerritbot added a comment.Aug 21 2020, 9:40 AM

Change 621692 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/transferpy@master] Fix references to now removed transferpy.RemoteExecution module

https://gerrit.wikimedia.org/r/621692

gerritbot added a comment.Aug 21 2020, 11:05 AM

Change 621692 merged by jenkins-bot:
[operations/software/transferpy@master] Fix references to now removed transferpy.RemoteExecution module

https://gerrit.wikimedia.org/r/621692

gerritbot added a comment.Aug 25 2020, 8:24 AM

Change 622310 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] Remove unused sql.py and check_private_data.py

https://gerrit.wikimedia.org/r/622310

gerritbot added a comment.Aug 25 2020, 10:03 AM

Change 622310 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] Remove unused sql.py and check_private_data.py

https://gerrit.wikimedia.org/r/622310

Kormat closed this task as Resolved.Aug 25 2020, 1:39 PM
Kormat updated the task description. (Show Details)

Mission accomplished.

Kormat mentioned this in rOSWBa869b2b63752: check_health: update from check_mariadb.py in puppet repo.Sep 1 2020, 8:36 AM
Kormat mentioned this in rOSWBc93094990027: Split utilities into separate packages.
Kormat mentioned this in rOSWB4a6a29f7a204: Update remote execution libraries from transferpy.
Kormat mentioned this in rOSWB0a49876a195c: Remove unused sql.py and check_private_data.py.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL