Page MenuHomePhabricator

Spin off common Spicerack modules into a standalone Python library importable anywhere
Closed, ResolvedPublic

Description

As part of Q1 OKRs, we want to spin off from Spicerack some of its modules and functionalities that can be useful to any other script/framework/tool.
The idea is to have a standalone library that can be included anywhere, potentially even installing it in our base system directly.
It should be available both as a Debian package and a PyPI module.
It should not required specific configuration files for the library, just the ones eventually needed by the specific modules, eventually with a sane default for the path to the configuration file (TBD where to put them).

The final list of functionalities/modules that should be spin off is TBD, this below is an initial draft:

  • actions
  • administrative (decided to skip this one)
  • config
  • decorators? (has some Spicerack-specific logic about dry-run detection that will be left in Spicerack)
  • dns
  • interactive (all but the mgmt password util function that will stay in spicerack)
  • the SAL logging part of the log module
  • phabricator
  • prometheus

Details

Event Timeline

Volans triaged this task as Medium priority.Jul 14 2020, 11:12 AM
Volans created this task.

I think it would be nice if there was a small helper function to set User-Agent headers per our policy.

The most difficult step is of course coming up with a name for the lib :)

The most straightforward combination in my opinion is py-wmflib for the Pypi/Debian package name, and wmflib for the python module. I woudn't use any other name along the line of cumin, but I don't have strong opinions.

On the naming part I'm open for everything, my only 2 cents are that usually having PyPi and python module with 2 different names causes small weirdness sometimes so I'd prefer to avoid it.

The name appears at least in those different environments:

  • Gerrit/GitHub repo name: would probably be wise to refer to Python in some way here
  • PyPI package: many packages have 'py' in the name but it usually feels weird, if we could avoid it here even better
  • Debian package: like Gerrit above, but that's easy because being a Python library it should be called python3-$name
  • The actual module to import in Python, would be nice to have wmflib here

So, using your proposal @elukey I'd say that we could do something like:

  • Gerrit repo: any of pywmflib, python-wmflib, wmflibpy, etc...
  • PyPi: wmflib
  • Debian package: python3-wmflib
  • Python module: wmflib

Change 617403 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Initial release of wmflib

https://gerrit.wikimedia.org/r/617403

Change 617408 had a related patch set uploaded (by Elukey; owner: Elukey):
[integration/config@master] Add settings for operations/software/pywmflib

https://gerrit.wikimedia.org/r/617408

Change 617408 merged by jenkins-bot:
[integration/config@master] Add settings for operations/software/pywmflib

https://gerrit.wikimedia.org/r/617408

Change 617403 merged by jenkins-bot:
[operations/software/pywmflib@master] Initial release of wmflib

https://gerrit.wikimedia.org/r/617403

Mentioned in SAL (#wikimedia-releng) [2020-09-01T11:04:39Z] <hashar> Manually triggering doc generation for operations/software/pywmflib: zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/pywmflib --change 617403,6 # T257905

Change 623576 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] Add link to pywmflib

https://gerrit.wikimedia.org/r/623576

Change 623576 merged by jenkins-bot:
[integration/docroot@master] Add link to pywmflib

https://gerrit.wikimedia.org/r/623576

Change 626378 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Remove some "Spicerack" references from the doc strings

https://gerrit.wikimedia.org/r/626378

Change 626380 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Add basic debian packaging

https://gerrit.wikimedia.org/r/626380

Change 626378 merged by Elukey:
[operations/software/pywmflib@master] Remove some "Spicerack" references from the doc strings

https://gerrit.wikimedia.org/r/626378

Change 626380 abandoned by Elukey:
[operations/software/pywmflib@master] Add basic debian packaging

Reason:
We decided to go for the separate debian branch

https://gerrit.wikimedia.org/r/626380

Version 0.0.2 has been released to APT wikimedia for buster (python3-wmflib) and on PyPI (wmflib).

Change 629686 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] dns: replace module with the one in wmflib

https://gerrit.wikimedia.org/r/629686

Change 629692 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.hosts.decommission: import from new library

https://gerrit.wikimedia.org/r/629692

Change 629686 merged by jenkins-bot:
[operations/software/spicerack@master] dns: replace module with the one in wmflib

https://gerrit.wikimedia.org/r/629686

Change 631792 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Import the action module from spicerack

https://gerrit.wikimedia.org/r/631792

Change 631909 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Import the config module from Spicerack

https://gerrit.wikimedia.org/r/631909

Change 631910 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Import the phabricator module from Spicerack

https://gerrit.wikimedia.org/r/631910

Change 631792 merged by Elukey:
[operations/software/pywmflib@master] Import the action module from spicerack

https://gerrit.wikimedia.org/r/631792

Change 631909 merged by Elukey:
[operations/software/pywmflib@master] Import the config module from Spicerack

https://gerrit.wikimedia.org/r/631909

Change 631910 merged by Elukey:
[operations/software/pywmflib@master] Import the phabricator module from Spicerack

https://gerrit.wikimedia.org/r/631910

Change 633774 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] Temporary limit spicerack dependency

https://gerrit.wikimedia.org/r/633774

Change 633774 merged by jenkins-bot:
[operations/cookbooks@master] Temporary limit spicerack dependency

https://gerrit.wikimedia.org/r/633774

Mentioned in SAL (#wikimedia-operations) [2020-10-23T09:09:28Z] <volans> upgrading spicerack to 0.0.44 on cumin hosts - T257905

Mentioned in SAL (#wikimedia-operations) [2020-10-23T10:56:14Z] <volans> uploaded python3-wmflib_0.0.3 to apt.wikimedia.org buster-wikimedia - T257905

Change 636000 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] Remove modules migrated to wmflib

https://gerrit.wikimedia.org/r/636000

Mentioned in SAL (#wikimedia-operations) [2020-10-28T07:53:14Z] <volans> upgraded python3-wmflib to 0.0.3 on the cumin hosts - T257905

Change 636000 merged by jenkins-bot:
[operations/software/spicerack@master] Remove modules migrated to wmflib

https://gerrit.wikimedia.org/r/636000

Change 629692 merged by jenkins-bot:
[operations/cookbooks@master] sre.hosts.decommission: import from new library

https://gerrit.wikimedia.org/r/629692

Change 647189 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Import prometheus and constants module from spicerack

https://gerrit.wikimedia.org/r/647189

Change 647189 merged by Elukey:
[operations/software/pywmflib@master] Import prometheus and constants module from spicerack

https://gerrit.wikimedia.org/r/647189

Change 650518 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Port the decorators.py module from Spicerack

https://gerrit.wikimedia.org/r/650518

Change 650518 merged by Elukey:
[operations/software/pywmflib@master] Port the decorators.py module from Spicerack

https://gerrit.wikimedia.org/r/650518

Change 650546 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/software/pywmflib@master] Port IRCSocketHandler from Spickerack and create irc_utils.py

https://gerrit.wikimedia.org/r/650546

Change 650546 merged by Elukey:
[operations/software/pywmflib@master] Port IRCSocketHandler from Spickerack and create irc.py

https://gerrit.wikimedia.org/r/650546

So pywmflib is feature-complete, before closing this task we should port the remaining parts of Spicerack to it and then we are done :)

Change 651765 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] interactive: migrate from spicerack to wmflib

https://gerrit.wikimedia.org/r/651765

Change 651805 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] Use newly migrated code from wmflib

https://gerrit.wikimedia.org/r/651805

Change 651805 merged by Volans:
[operations/software/spicerack@master] Use newly migrated code from wmflib

https://gerrit.wikimedia.org/r/651805

Mentioned in SAL (#wikimedia-operations) [2021-01-05T15:41:19Z] <volans> upgraded wmflib to 0.0.6 on all hosts where it's installed - T257905

Change 651765 merged by jenkins-bot:
[operations/cookbooks@master] interactive: migrate from spicerack to wmflib

https://gerrit.wikimedia.org/r/651765

This comment was removed by Volans.

Mentioned in SAL (#wikimedia-operations) [2021-01-13T15:05:57Z] <volans> upgraded spicerack to 0.0.47-1+deb10u1 on cumin2001 - T257905

Change 655909 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] wmf-auto-reimage: fix Netbox update for 2.9 upgrade

https://gerrit.wikimedia.org/r/655909

Mentioned in SAL (#wikimedia-operations) [2021-01-13T15:56:52Z] <volans> upgraded spicerack to 0.0.47-1+deb10u1 on cumin1001 - T257905

Change 704345 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/spicerack@master] decorators: migrate to the wmflib version

https://gerrit.wikimedia.org/r/704345

Change 704345 merged by jenkins-bot:

[operations/software/spicerack@master] decorators: migrate to the wmflib version

https://gerrit.wikimedia.org/r/704345

And with the last patch merged that finally makes use of the wmflib version of the @retry decorator also in spicerack this can be called completed. IT will be released in the next few days.