And move bigbrother to tools-services. The two cron machines will be in cold-standby-failover mode (aka, someone has to explicitly start them to perform failover). They'll also be trusty and have more RAM / CPU than current machine.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T123186 tools.taxonbot and tools.giftbot cronjobs not firing | |||
| Resolved | yuvipanda | T123873 Migrate tools-submit to tools-cron-01/-02 |
Event Timeline
Comment Actions
Change 265183 had a related patch set uploaded (by Yuvipanda):
toollabs: Add cronrunner role
Comment Actions
Change 265192 had a related patch set uploaded (by Yuvipanda):
toollabs: Use hiera to figure out where cron runner host is
Comment Actions
Change 265192 merged by Yuvipanda:
toollabs: Use hiera to figure out where cron runner host is
Comment Actions
Change 265193 had a related patch set uploaded (by Yuvipanda):
toollabs: Move bigbrother to services nodes
Comment Actions
So failover should be:
- Stop cron in active (or shut it down)
- Clean out / Backup all the active crontabs in active (note that there's also a puppet initiated backup onto NFS every 20mins)
- Copy crontabs from backup to new host
- Set hiera variable
- Run puppet on the bastions
Comment Actions
Change 265194 had a related patch set uploaded (by Yuvipanda):
toollabs: Backup crontabs from cronrunner
Comment Actions
Change 265195 had a related patch set uploaded (by Yuvipanda):
toollabs: Move updatetools to run on services host
Comment Actions
Alright, I think that moves them all off to their appropriate places. I'll switchover tomorrow.
Comment Actions
The cron runner hosts do not have the whole exec_environ anymore. No idea why they did - they shouldn't.
Comment Actions
Change 265206 had a related patch set uploaded (by Yuvipanda):
toollabs: Move toolwatcher to services
Comment Actions
Change 265195 merged by Yuvipanda:
toollabs: Move updatetools to run on services host
Comment Actions
I've left tools-submit as is (with crontabs moved to crontabs.bak so they don't accidentally fire) in case we need to recover anything in the next few days.