Page MenuHomePhabricator

Decommission script race condition
Open, NormalPublic

Description

The wmf_decommission_host.py script has a clear race condition given that the puppet crontab that runs every 30 minutes will run apt-get update in any case (also if puppet is disabled) that will re-populate DebMonitor for that host.
Given the frequency of this crontab is pretty easy to incur into this race condition.
The same race condition exists also for the daily debmonitor cron, but with a much smaller propability of hitting it.

Opening this task to find a clean solution.

Event Timeline

Volans created this task.Oct 8 2018, 8:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2018, 8:04 AM

The decom script could create some file like /var/run/decom-in-progress and then we patch the 30m cron run to bail out if that file is found.

colewhite triaged this task as Normal priority.Dec 17 2018, 10:11 PM