This is urgent, blocking work so please help limit scope. This task is finished when the following features are verified working:

Job migration status:
https://docs.google.com/spreadsheets/d/1qfXaBmhW45qSbFRqgJs_zpeEt6ZIo2hrZcBtRhVXS9w/edit#gid=0

process-control

• Ops can package and deploy the tool.
• stdout logfiles are written one file per job run. (T161155)
• Devs can run jobs one-off.
• Runs jobs according to a code-generated crontab.
• Never drop logs even (especially!) if the process is killed unexpectedly. (T161571)
• Failmail when job exits with non-zero return code--let not perfect be thine enemy.
• Nobody can accidentally run the script as their own user.
• Working workaround for specific chained jobs. (T161035)

puppet

• Jobs configuration is sync'ed to /var/lib/process-control with localsettings. Read-only.
• Global configuration file is synced to /etc/process-control.yaml
• Devs have sudo access to the scripts and can pass any CLI params.
• jenkins g+ws /var/log/process-control
• cron-generate can somehow write to /etc/cron.d/process-control

Not in MVP scope

• Devs can kill jobs.
• Log actions and errors to syslog. Echo to console when os.isatty()
• script to list all jobs and statuses (T161584)
• should be able to disable groups of jobs (T160699)
• repeated failure handling (T161567)
• Turn process-control lock module into a context manager (T161536)
• Clean up deb packaging once we're on Jessie.
• 100% test coverage coziness.