This is urgent, blocking work so please help limit scope. This task is finished when the following features are verified working:
Job migration status:
https://docs.google.com/spreadsheets/d/1qfXaBmhW45qSbFRqgJs_zpeEt6ZIo2hrZcBtRhVXS9w/edit#gid=0
process-control
-----
* [x] Ops can package and deploy the tool.
* [x] stdout logfiles are written one file per job run. (T161155)
* [x] Devs can run jobs one-off.
* [x] Runs jobs according to a code-generated crontab.
* [x] Never drop logs even (especially!) if the process is killed unexpectedly. (T161571)
* [x] Failmail when job exits with non-zero return code--let not perfect be thine enemy.
* [x] Nobody can accidentally run the script as their own user.
* [ ] Working workaround for specific chained jobs. (T161035)
puppet
-----
* [x] Jobs configuration is sync'ed to /var/lib/process-control with localsettings. Read-only.
* [x] Global configuration file is synced to /etc/process-control.yaml
* [x] Devs have sudo access to the scripts and can pass any CLI params.
* [x] jenkins g+ws /var/log/process-control
* [x] cron-generate can somehow write to /etc/cron.d/process-control
Not in MVP scope
-----
* Devs can kill jobs.
* Log actions and errors to syslog. Echo to console when os.isatty()
* script to list all jobs and statuses (T161584)
* should be able to disable groups of jobs (T160699)
* repeated failure handling (T161567)
* Turn process-control lock module into a context manager (T161536)
* Clean up deb packaging once we're on Jessie.
* 100% test coverage coziness.