Page MenuHomePhabricator

Create in-cloud, cloud-vps-wide cumin masters
Closed, ResolvedPublic

Description

Once we have these we can decom or reclaim labpuppetmaster1001/1002

Event Timeline

Andrew created this task.Sep 10 2019, 3:24 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptSep 10 2019, 3:25 AM

Change 535670 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud cumin: don't use a bastion if cumin is already running in the cloud

https://gerrit.wikimedia.org/r/535670

Change 535677 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud cumin: add cloud-cumin-01.cloudinfra as a cumin master

https://gerrit.wikimedia.org/r/535677

Change 535670 merged by Andrew Bogott:
[operations/puppet@production] cloud cumin: don't use a bastion if cumin is already running in the cloud

https://gerrit.wikimedia.org/r/535670

Change 535677 merged by Andrew Bogott:
[operations/puppet@production] cloud cumin: add cloud-cumin-01.cloudinfra as a cumin master

https://gerrit.wikimedia.org/r/535677

Andrew closed this task as Resolved.Sep 10 2019, 8:20 PM

There's now a cumin master on cloud-cumon-01.cloudinfra.eqiad.wmflabs that seems to work just fine.

Krenair reopened this task as Open.Sep 10 2019, 10:53 PM

It seems to work within cloudinfra but I think we have a little bit of config tweaking to do.

Change 535727 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Labs cumin masters: Only set project filter if we're a project-specific cumin master

https://gerrit.wikimedia.org/r/535727

Change 535733 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Labs cumin masters: Remove config associated with proxying via bastion

https://gerrit.wikimedia.org/r/535733

Krenair added a comment.EditedSep 10 2019, 10:59 PM

https://gerrit.wikimedia.org/r/535727 should make it behave like the existing cumin master, https://gerrit.wikimedia.org/r/535733 is tidyup of historical stuff

(I tried applying the first of those manually on the new instance, ran cumin '*' id and saw 814 hosts will be targeted so that looks good - 664 of those hosts even respond how you'd expect with uid=0(root) gid=0(root) groups=0(root), we should go through the ones that don't later in a different task)

Change 535727 merged by Andrew Bogott:
[operations/puppet@production] Labs cumin masters: Only set openstack project filter if we're project-specific

https://gerrit.wikimedia.org/r/535727

Change 535866 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud cumin: add a second cumin master

https://gerrit.wikimedia.org/r/535866

Change 535866 merged by Andrew Bogott:
[operations/puppet@production] cloud cumin: add a second cumin master

https://gerrit.wikimedia.org/r/535866

I built a second cumin host, cloud-cumin-02.cloudinfra.eqiad.wmflabs. It's partly for backup, and partly because I wanted to confirm that the existing puppetization is sufficient. It turns out that it is! The new host just required a reboot to get keyholder on board.

-02 doesn't reach quite as many hosts since I haven't hand-tuned the exceptions, but as puppet does its work it should eventually be equivalent to -01.

Change 535733 merged by Andrew Bogott:
[operations/puppet@production] Labs cumin masters: Remove config associated with proxying via bastion

https://gerrit.wikimedia.org/r/535733

Andrew reassigned this task from Andrew to Krenair.Sep 11 2019, 3:57 PM

I think this task is done but I'll let @Krenair comment and close :)

Krenair closed this task as Resolved.Sep 11 2019, 8:18 PM
Krenair reassigned this task from Krenair to Andrew.

with the merging of https://gerrit.wikimedia.org/r/535727 I think it's done :)