What is this task about? (Some conference presentation? Or just Epic with a corresponding epic title? :P )

@Aklapper: yeah it's epic

@dduvall, @thcipriani, @demon: This is a fairly helpful high level overview/comparison of salt and ansible, maybe worth a read: http://jensrantil.github.io/salt-vs-ansible.html

btw, see also: T45338: Trebuchet blockers for MediaWiki (tracking)

In order to depool a server (and not trigger false alarms in the alerting system) we would like to have a deployment flag (a lock file or something) that causes the pybal check to fail but in a way that lets pybal know that this is a temporary and expected downtime. Then some custom logic in pybal would temporarily depool the server but continue checking for the status to return to normal (and bypass alerting)

So, in our current configuration, pybal checks the individual servers by ssh, which uses a "forced command" on the target server to run this:

uptime; touch /var/tmp/pybal-check.stamp

It also checks http but only on the varnish proxy not on the individual apache nodes.

Changing the force command to something like this would probably do the trick:

[ -f "/tmp/deploy.lock" ] && stat --format=deployment:%Y /tmp/deploy.lock && exit 123 || uptime;

This will exit with status 123 when the lock file exists, and outputs the last modified time of the lock file as "deployment:timestamp" so that pybal can know a) that it's down for deployment and b) when the deployment lock was last updated. We could then add some intelligence to pybal to alert if a host is stuck in deployment...the deployment process could periodically touch the lock file to indicate that it is in fact progressing and not hung somewhere.

Sound good?

So pybal already has a built in (and configurable) threshold limiting how many servers can be depooled at any given time. This addresses @chasemp's concern about silently depooling most of the cluster if something goes wrong with a deploy.

it looks like all we need is the change above to implement the depool during deploy via creating a simple lock file and removing it at the end of the deployment process.

We should have some sort of self-checks to be sure that everything is kosher before removing the lock.

I just put this as a session for tomorrow at 2pm Lyon time per https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2015/Program

@mmodell, re monitoring / depooling: You could consider directly checking etcd for this. See T100793.