Page MenuHomePhabricator

Create a repository for sharing ad-hoc local development tools
Closed, ResolvedPublic

Description

Background

Wikimedia developers occasionally create small scripts or tools which serve a specialized purpose and often remain in ~/bin before eventually getting shared via ad-hoc methods (code dumped on wiki, paste[1], gist, etc.)

Proposal:

In order to encourage Developer Productivity through the sharing of time-saving tools and scripts, we should create a LocalDevTools[2] repository that will be a home to various small tools and utilities that are specifically useful Wikimedia developers and site reliability engineers, and serve as a catch-all for tools that do not have another more appropriate home.

  • A single git repository of scripts and tools
  • Contributor-friendly, Inclusive contribution conditions
    • Accept just about anything in order to encourage sharing small tools that don't have a more appropriate home.
      • I expect that this will be mostly bash and python, ideally written to work across recent versions of Debian and Mac OS X but I don't think ruby, php or your favorite scripting language should be necessarily excluded.
  • Developer-friendly setup and dependency management:
    • Developers should be able to just clone the repository and run any script within it without too much fuss.
    • We could have a one time setup process that requires making appropriate symlinks or configuring a few environment variables, just as long as it's easy and only needs to be done once.
    • Include most small dependencies in-repo as much as possible when they aren't available from common os repositories.
Footnotes
  1. The example that motivated me to finally create this task was P5608 but onboarding two new members of the RelEng team has recently brought the issue to light a few times - there are a ton of useful tools, snippets, and other small utilities that are shared among wikimedia developers through very inefficient means and newcomers are severely disadvantaged by reinventing a lot of wheels. This leads to a lot of lost productivity, especially during the first few months of work.
  2. The name LocalDevTools is just my proposal, feel free to bike-shed about it in the comments 😄

Event Timeline

mmodell updated the task description. (Show Details)

cc'ing the rest of the releng team plus the DeviantArt cabal.

Thanks @mmodell for opening the task for this wish I had since a while.
I'm not sure how to solve the team-specific tools/script we might need, if this should just include very broad scripts and each team should be responsible to create their own team-specific repos or if we should support that use case in this single repo.

I think one repo should be good enough for most cases. Teams could have a team specific directory inside of the repo if we wanted to organize it that way. Release engineering already has the rMREL MediaWiki Release Tools repo where we stick some random scripts but I think a cross-team repo would benefit everyone the most.

I was made aware today of the existence of a wmf-utils repository that might have been created with that in mind but doesn't seem very used:
https://gerrit.wikimedia.org/g/wmf-utils/+/refs/heads/master

Traditionally we have the tools in standalone repositories under Gerrit mediawiki/tools hierarchy. There are a few gems here and there:

mediawiki/tools/cookiecutter-libraryEasy bootstrap of a new project
mediawiki/tools/schrootA secure chroot by Tim Starling
mediawiki/tools/git-remoteTeach git to handle mediawiki:// urls (by Ori Livneh iirc)

Then there are bunch of project all around. wmf-utils is one of them, ./utils in the operations/puppet.git repo, mediawiki/extensions/BoilerPlate which can be used to create a new extension.... And of course MediaWiki Vagrant which must have a lot of local development tools.

Beware though, repositories that serves multiple purposes might end up being unmanageable later on. We have several examples (eventlogging and poolcounter daemons were developed in the repositories of their MediaWiki extensions), operations/software which quickly became some mono repo until some softwares got split, puppet which serves to host and ship software (recently keyholder got extracted to a new repo).

Anyway, that should not stop us. If some script becomes to big, it can always be promoted to its own little git repository :-]

Be bold and create yet another repo in Gerrit, it is cheap.

I forgot. mediawiki/tools/code-utils at https://phabricator.wikimedia.org/diffusion/MCUT/ which states:

# MediaWiki code-utils
	
A collections of code utilities for MediaWiki.
``

Yeah maybe it'd be better to reuse one of these repos than to create yet another.

Change 524755 had a related patch set uploaded (by CDanis; owner: CDanis):
[wmf-utils@master] wmf-utils: add wmf-update-prod-known-hosts

https://gerrit.wikimedia.org/r/524755

Change 524754 had a related patch set uploaded (by CDanis; owner: CDanis):
[wmf-utils@master] wmf-clone: fix flake8

https://gerrit.wikimedia.org/r/524754

Change 524757 had a related patch set uploaded (by CDanis; owner: CDanis):
[wmf-utils@master] wmf-utils: add .gitignore for editor backup files and .tox

https://gerrit.wikimedia.org/r/524757

Change 524754 merged by jenkins-bot:
[wmf-utils@master] wmf-clone: fix flake8

https://gerrit.wikimedia.org/r/524754

Change 524757 merged by jenkins-bot:
[wmf-utils@master] wmf-utils: add .gitignore for editor backup files and .tox

https://gerrit.wikimedia.org/r/524757

Change 524755 merged by jenkins-bot:
[wmf-utils@master] wmf-utils: add wmf-update-prod-known-hosts

https://gerrit.wikimedia.org/r/524755

Which project tag to add for tasks about this repository wmf-utils such as T275806: wmf-utils has an outdated script to update known hosts files?

In the interest of seeing how it works, I wound up using wmf-utils for a recent helper script for some docker work: https://gerrit.wikimedia.org/r/c/wmf-utils/+/670320

Marking it as resolved since we have wmf-utils.git now. It might need some marketing and a few volunteers to babysit incoming patches though ;]

DLynch added a subscriber: Krinkle.

@Krinkle points out that the current contents of wmf-utils were somewhat SRE-focused, and suggests that code-utils would be the better one to use for general purpose tools that don't e.g. expect root. I have thus reopened this ticket.

Also note that the current repo is only writable by SRE (Antoine was able to merge this patch as Gerrit admin via RelEng membership), and the name suggests its contents are WMF-specific, whereas the mediawiki-code-utils repo is a virally maintained and open to MediaWiki developer tooling applicable by all MediaWiki contributors (incl. volunteer maintainers of core and extensions).

Chiming in as the person who created wmf-utils initially: the vision I had for the repository is to collect wmf-specific utilities. In other words anything that interacts with the Foundation' systems (e.g. wmf-clone). The SRE-only membership is an historical artifact and I think the repository should be open to wmf (and nda ?) to enable collaborative maintenance.

Having said all that, I don't feel strongly about code-utils vs wmf-utils vs something else, hope that helps!

This might be a good candidate for early adoption / migration to gitlab?

cc'ing @Addshore as the mwcli project may also be a good home for MediaWiki local development scripts (or maybe it already replaces other bash/python scripts that are around)

cc'ing @Addshore as the mwcli project may also be a good home for MediaWiki local development scripts (or maybe it already replaces other bash/python scripts that are around)

It would be relatively easy to pull small useful scripts or functionality into the mwcli project
Do we have any shortlist candidates?

Looks like https://gerrit.wikimedia.org/g/wmf-utils is the way to go.

There is a similar one https://gerrit.wikimedia.org/g/mediawiki/tools/code-utils which are tools specific to MediaWiki development.