Page MenuHomePhabricator

Install conftool on deployment masters
Closed, ResolvedPublic

Description

We need conftool (rOSCT operations-software-conftool), or at least the python-conftool library, on deployment masters.

This is needed in order for us to implement pooling/depooling from scap as in D600: Create a wrapper around conftool for our pooling/depooling needs.

Event Timeline

mmodell created this task.Apr 21 2017, 5:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 21 2017, 5:15 PM
mmodell triaged this task as Medium priority.Apr 21 2017, 5:18 PM
mmodell added a project: SRE-tools.
mmodell added a subscriber: Joe.
Volans edited projects, added Operations; removed SRE-tools.Apr 21 2017, 5:38 PM

Change 349498 had a related patch set uploaded (by 20after4):
[operations/puppet@production] Include ::profile::conftool::client on deployment servers

https://gerrit.wikimedia.org/r/349498

mmodell updated the task description. (Show Details)Apr 21 2017, 6:31 PM
mmodell updated the task description. (Show Details)
Joe added a comment.Apr 26 2017, 6:53 AM

I don't think scap should interact with conftool by itself, unless it reproduces what the restart-<service> scripts are doing right now, which is:

  • depool in conftool
  • verify that all LVS servers have actually depooled the host, in case they didn't, wait until they're able to depool the server
  • restart
  • do the same thing when repooling

basically, this is to ensure that we don't hit the depool threshold of pybal by doing e.g. a rolling restart that runs too fast.

@Joe: That all seems reasonable. I don't particularly want to duplicate logic in scap unless it makes the most sense for that logic to live in scap.

This task is mostly concerned with implementing deplooling for mediawiki deployments but the functionality in scap could handle the process with other services as well. If this isn't the right approach then I think we could use some guidance from you on how to get this working.

demon added a comment.Apr 27 2017, 6:07 PM

So, our original usecase for conftool was T125629: Depool proxies temporarily while scap is ongoing to avoid taxing those nodes -- we shouldn't have to worry much about depooling the whole cluster.

But then T162209: Figure out how node limitation interacts with proxies came along, which makes me wonder if we actually want proxies to be normal apaches, or if they'd be better as a kind of "deployment server lite" that serves as proxies for **all* targets, not just MW (and so we don't have to pick proxies for each service, all would immediately benefit). If this is the case, T125629 is moot.

demon added a comment.May 10 2017, 4:15 PM

Oh, duh other total obvious usecase I forgot: being able to pull our target list from etcd directly, instead of having to write the dsh files as an intermediate step.

Joe added a comment.May 11 2017, 6:05 AM

@Joe: That all seems reasonable. I don't particularly want to duplicate logic in scap unless it makes the most sense for that logic to live in scap.

This task is mostly concerned with implementing deplooling for mediawiki deployments but the functionality in scap could handle the process with other services as well. If this isn't the right approach then I think we could use some guidance from you on how to get this working.

Fair enough, this is a legitimate problem (depooling scap proxies).

I added some comments on the patch, and merging the change you have here since it makes sense to have conftool configured on deployment servers.

Change 349498 merged by Giuseppe Lavagetto:
[operations/puppet@production] role::deployment::mediawiki: include ::profile::conftool::client

https://gerrit.wikimedia.org/r/349498

Joe closed this task as Resolved.May 11 2017, 6:25 AM
Joe claimed this task.