Page MenuHomePhabricator

Deploy restbase to restbase202[123]
Closed, ResolvedPublic

Description

A recent expansion added 3 new hosts in codfw, restbase2021 & restbase2022 & restbase2023. RESTBase needs to be deployed to each of these.


See also: T243000: Bootstrap new Cassandra instances: restbase202[123]-{a,b,c}

Event Timeline

Eevans triaged this task as Medium priority.Feb 3 2020, 9:09 PM
Pchelolo added subscribers: hnowlan, Clarakosi, Pchelolo.

Since @Clarakosi is now a RESTBase deployer, we can let her do it for the sake of learning. Please coordinate with me when/what will be happening, I'm not entirely sure the steps below are a complete list of steps.

Steps:

  1. Add restbase202[123] to scap deployment targets for scap (in gerrit ofc)
  2. Add restbase202[123] to hiera and run puppet.
  3. Do the scap deploy.
  4. Add new hosts to conftool-data and pool the new hosts ( SRE help will be needed, cc @hnowlan )

All these steps should be done after Cassandra is bootstrapped. See T219404 for the ticket where fresh deploy was done the previous time.

All these steps should be done after Cassandra is bootstrapped. See T219404 for the ticket where fresh deploy was done the previous time.

Cassandra has been bootstrapped.

Change 570091 had a related patch set uploaded (by Clarakosi; owner: Clarakosi):
[mediawiki/services/restbase/deploy@master] Add Add restbase202[123] to scap deployment targets

https://gerrit.wikimedia.org/r/570091

Change 570094 had a related patch set uploaded (by Clarakosi; owner: Clarakosi):
[operations/puppet@production] Add restbase202[123] to hiera

https://gerrit.wikimedia.org/r/570094

Is it safe to add the hosts to conftool and to hiera at the same time? Seems like we should add them to hiera in order to receive deployments via scap and then configure them in conftool, but I'm still getting familiar with how interactions between puppet, conftool and scap work and are ordered.

Change 570094 merged by Hnowlan:
[operations/puppet@production] hiera: Add restbase202[123] to hiera

https://gerrit.wikimedia.org/r/570094

Change 570091 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Add restbase202[123] to scap deployment targets

https://gerrit.wikimedia.org/r/570091

Mentioned in SAL (#wikimedia-operations) [2020-03-25T18:39:37Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@a1c3be4]: Add restbase202[123] T244178

RESTBase has been deployed, so we need to pull those servers now?

Mentioned in SAL (#wikimedia-operations) [2020-03-25T18:53:36Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@a1c3be4]: Add restbase202[123] T244178 (duration: 14m 00s)

restbase202[123] have been pooled and appear healthy in pybal so far. I'll keep an eye