Page MenuHomePhabricator

WIP: PSHH - Parallel SH Hell
AbandonedPublic

Authored by mmodell on Jan 8 2016, 3:52 AM.

Details

Reviewers
thcipriani
Group Reviewers
Release-Engineering-Team
Patch without arc
git checkout -b D86 && curl -L https://phabricator.wikimedia.org/D86?download=true | git apply
Summary

Eventually this will replace make-wmf-branch.

The idea here is to execute arbitrary commands once for each submodule within a repository. Eventually this will be done in parallel for maximum performance. I might be able to abstract this into a framework that will be generally useful for executing many copies of the same job in parallel. For now it's not very general and not yet actually parallel.

Diff Detail

Repository
rMREL MediaWiki Release Tools
Branch
pshh (branched from master)
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 307
Build 417: arc lint + arc unit

Event Timeline

mmodell updated this revision to Diff 246.Jan 8 2016, 3:52 AM
mmodell retitled this revision from to PSHH - Paralell SH Hell.
mmodell updated this object.
mmodell added a reviewer: thcipriani.
Restricted Application added a reviewer: Release-Engineering-Team. · View Herald TranscriptJan 8 2016, 3:52 AM

notes:

Not currently parallel ;)

mmodell retitled this revision from PSHH - Paralell SH Hell to WIP: PSHH - Paralell SH Hell.Jan 8 2016, 5:01 PM
mmodell updated this object.
mmodell edited edge metadata.
mmodell updated this revision to Diff 247.Jan 8 2016, 5:38 PM
  • use prompt_toolkit
mmodell updated this revision to Diff 251.EditedJan 10 2016, 1:39 AM
  • Better use of prompt toolkit
  • Better tracking of success / failure
  • Starting to abstract this into a generally useful framework.

This thing is starting to get good.

"No Linters Available" - maybe time to add a .arclint file? :)

mmodell updated this revision to Diff 252.Jan 10 2016, 6:35 AM
  • now with moar parallel.

"No Linters Available" - maybe time to add a .arclint file? :)

Indeed, but this repo is a mixture of languages, I need to look into linters for php, python, etc. Also, I hate PEP8 so I'm in no rush to have that linter here.

mmodell updated this revision to Diff 253.Jan 10 2016, 4:42 PM
  • store results in sqlite
mmodell planned changes to this revision.Jan 10 2016, 4:46 PM

Next steps:

  • support sql on the command line, so that we can easily manipulate the the lists with sql queries
  • allow rerunning commands on arbitrary lists (mainly for re-running commands on the failed items from a previous run)

It turns out that python concurrent futures work really well. I wish scap could use python 3.5 :(

mmodell updated this revision to Diff 254.Jan 11 2016, 5:36 PM
  • refactored the list handling code to be more general (and object oriented)
mmodell updated this revision to Diff 255.Jan 11 2016, 10:18 PM
  • pass arbitrary arguments to the final command handler function
  • reduce the list of items to just the failed items after each batfch runs.
mmodell updated this revision to Diff 305.Jan 24 2016, 7:28 PM
  • refactored code for cleaner organization
  • Read lists of paths from stdin
mmodell retitled this revision from WIP: PSHH - Paralell SH Hell to WIP: PSHH - Parallel SH Hell.Jan 25 2016, 9:31 AM
mmodell updated this revision to Diff 321.Jan 28 2016, 5:49 PM
  • better handling of system commands
mmodell updated this revision to Diff 322.Jan 28 2016, 6:03 PM
  • handle piped list + command passed from the calling shell
mmodell abandoned this revision.May 2 2017, 7:45 PM