Page MenuHomePhabricator

Migrate tools-submit to tools-cron-01/-02
Closed, ResolvedPublic

Description

And move bigbrother to tools-services. The two cron machines will be in cold-standby-failover mode (aka, someone has to explicitly start them to perform failover). They'll also be trusty and have more RAM / CPU than current machine.

Event Timeline

yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Toolforge.
yuvipanda added a subscriber: yuvipanda.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

Change 265183 had a related patch set uploaded (by Yuvipanda):
toollabs: Add cronrunner role

https://gerrit.wikimedia.org/r/265183

Change 265183 merged by Yuvipanda:
toollabs: Add cronrunner role

https://gerrit.wikimedia.org/r/265183

Change 265192 had a related patch set uploaded (by Yuvipanda):
toollabs: Use hiera to figure out where cron runner host is

https://gerrit.wikimedia.org/r/265192

Change 265192 merged by Yuvipanda:
toollabs: Use hiera to figure out where cron runner host is

https://gerrit.wikimedia.org/r/265192

Change 265193 had a related patch set uploaded (by Yuvipanda):
toollabs: Move bigbrother to services nodes

https://gerrit.wikimedia.org/r/265193

So failover should be:

  1. Stop cron in active (or shut it down)
  2. Clean out / Backup all the active crontabs in active (note that there's also a puppet initiated backup onto NFS every 20mins)
  3. Copy crontabs from backup to new host
  4. Set hiera variable
  5. Run puppet on the bastions

Change 265194 had a related patch set uploaded (by Yuvipanda):
toollabs: Backup crontabs from cronrunner

https://gerrit.wikimedia.org/r/265194

Change 265195 had a related patch set uploaded (by Yuvipanda):
toollabs: Move updatetools to run on services host

https://gerrit.wikimedia.org/r/265195

Alright, I think that moves them all off to their appropriate places. I'll switchover tomorrow.

The cron runner hosts do not have the whole exec_environ anymore. No idea why they did - they shouldn't.

Change 265194 merged by Yuvipanda:
toollabs: Backup crontabs from cronrunner

https://gerrit.wikimedia.org/r/265194

Change 265206 had a related patch set uploaded (by Yuvipanda):
toollabs: Move toolwatcher to services

https://gerrit.wikimedia.org/r/265206

Change 265193 merged by Yuvipanda:
toollabs: Move bigbrother to services nodes

https://gerrit.wikimedia.org/r/265193

Change 265195 merged by Yuvipanda:
toollabs: Move updatetools to run on services host

https://gerrit.wikimedia.org/r/265195

Change 265206 merged by Yuvipanda:
toollabs: Move toolwatcher to services

https://gerrit.wikimedia.org/r/265206

This is all done now! \o/

I've left tools-submit as is (with crontabs moved to crontabs.bak so they don't accidentally fire) in case we need to recover anything in the next few days.