Page MenuHomePhabricator

DBA python layout
Closed, ResolvedPublic

Description

Current issues:

  • wmfmariadbpy contains some outdated scripts (e.g. check_health.py which is an outdated version of check_mariadb.py in the puppet repo)
  • it also contains some unfinished/unused scripts (e.g. sql.py)
  • it contains some duplication of libraries with transferpy (e.g. CuminExecution)
  • it mixes scripts and libraries in the same directory
  • it mixed backup and db code in the same directory

Proposal:

  • Move all of the remote-execution libraries into wmfmariadbpy, and have transferpy depend on python3-wmfmariadb.
  • Delete unfinished/unused code. Can always be retrieved from git history later if we decide we want to work on it.
  • Separate out code by package; and package-level scripts can live in packagename/cli/ in the repo.
  • Migrate check_mariadb.py to wmfmariadb, and update puppet to use it from there.

Event Timeline

Kormat created this task.Aug 3 2020, 2:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 3 2020, 2:25 PM
Kormat triaged this task as Medium priority.Aug 3 2020, 2:25 PM
Marostegui moved this task from Triage to Backlog on the DBA board.Aug 4 2020, 5:14 AM
Kormat moved this task from Unsorted 💣 to Active 🚁 on the User-Kormat board.Aug 4 2020, 7:31 AM

Change 618310 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] check_health: update from check_mariadb.py in puppet repo

https://gerrit.wikimedia.org/r/618310

Change 618310 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] check_health: update from check_mariadb.py in puppet repo

https://gerrit.wikimedia.org/r/618310

Kormat updated the task description. (Show Details)Aug 5 2020, 12:28 PM

Change 618513 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] Split utilities into separate packages

https://gerrit.wikimedia.org/r/618513

Change 618513 merged by Kormat:
[operations/software/wmfmariadbpy@master] Split utilities into separate packages

https://gerrit.wikimedia.org/r/618513

Change 619443 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Drop check_mariadb.py in favour of packaged version

https://gerrit.wikimedia.org/r/619443

Change 619476 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] Update remote execution libraries from transferpy

https://gerrit.wikimedia.org/r/619476

Change 619959 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/transferpy@master] Move RemoteExecution library to wmfmariadbpy

https://gerrit.wikimedia.org/r/619959

Kormat claimed this task.Aug 13 2020, 8:22 AM
Kormat moved this task from Backlog to In progress on the DBA board.

Change 619476 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] Update remote execution libraries from transferpy

https://gerrit.wikimedia.org/r/619476

Mentioned in SAL (#wikimedia-operations) [2020-08-18T13:04:19Z] <kormat> disabling puppet on all db machines T259516

Change 619443 merged by Kormat:
[operations/puppet@production] mariadb: Drop check_mariadb.py in favour of packaged version

https://gerrit.wikimedia.org/r/619443

Change 620931 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Update name of wmfmariadbpy package

https://gerrit.wikimedia.org/r/620931

Change 620931 merged by Kormat:
[operations/puppet@production] mariadb: Update name of wmfmariadbpy package

https://gerrit.wikimedia.org/r/620931

Mentioned in SAL (#wikimedia-operations) [2020-08-20T08:52:37Z] <kormat> removing /usr/bin/check_mariadb.py from all db hosts T259516

Change 619959 merged by Jcrespo:
[operations/software/transferpy@master] Move RemoteExecution library to wmfmariadbpy

https://gerrit.wikimedia.org/r/619959

Kormat updated the task description. (Show Details)Aug 21 2020, 9:34 AM

Change 621692 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/transferpy@master] Fix references to now removed transferpy.RemoteExecution module

https://gerrit.wikimedia.org/r/621692

Change 621692 merged by jenkins-bot:
[operations/software/transferpy@master] Fix references to now removed transferpy.RemoteExecution module

https://gerrit.wikimedia.org/r/621692

Change 622310 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] Remove unused sql.py and check_private_data.py

https://gerrit.wikimedia.org/r/622310

Change 622310 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] Remove unused sql.py and check_private_data.py

https://gerrit.wikimedia.org/r/622310

Kormat closed this task as Resolved.Tue, Aug 25, 1:39 PM
Kormat updated the task description. (Show Details)

Mission accomplished.