Page MenuHomePhabricator

Replace deployment-memc[04-05] with Buster hosts
Closed, ResolvedPublic

Description

deployment-memc[04-05] are running Jessie while deployment-memc08 is running Buster. Note that 04-05 have role::mediawiki::memcached while 08 has role::memcached. 04-05 have been configured to MediaWikis redis_lock service, which seems to be used for MW locks, I'm not sure if it's required to be persistent or can the data not be transferred when creating new Buster hosts.

Event Timeline

taavi triaged this task as Medium priority.Mar 7 2021, 12:42 PM
taavi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
taavi renamed this task from Decomission deployment-memc[04-05] to Replace deployment-memc[04-05] with Buster hosts.Mar 7 2021, 1:00 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-07T13:14:13Z] <Majavah> create deployment-memc09 on Buster T276707

Mentioned in SAL (#wikimedia-releng) [2021-03-07T15:28:06Z] <Majavah> remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, T276707 T250585

Mentioned in SAL (#wikimedia-releng) [2021-03-07T15:44:52Z] <Majavah> create deployment-memc10 on Buster T276707, beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted

Mentioned in SAL (#wikimedia-releng) [2021-03-07T15:52:52Z] <Majavah> redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 T276707

Redis/MainStash is now running on deployment-memc[08-10], redis-lock and Memcached are still on [04-05,08] and I'll try to move then shortly

Change 669436 had a related patch set uploaded (by Majavah; owner: Majavah):
[operations/puppet@production] wancache: change deployment-prep to new Buster hosts

https://gerrit.wikimedia.org/r/669436

Change 669439 had a related patch set uploaded (by Majavah; owner: Majavah):
[operations/mediawiki-config@master] betacluster: Use new memc* hosts for redis_lock

https://gerrit.wikimedia.org/r/669439

Change 669439 merged by jenkins-bot:
[operations/mediawiki-config@master] betacluster: Use new memc* hosts for redis_lock

https://gerrit.wikimedia.org/r/669439

Mentioned in SAL (#wikimedia-releng) [2021-03-07T16:51:31Z] <Majavah> cherry pick 669436 and 669436 to deployment-puppetmaster04 T276707

Mentioned in SAL (#wikimedia-releng) [2021-03-07T17:18:50Z] <Majavah> shutdown deployment-memc[04-05] T276707

Change 669436 merged by Legoktm:
[operations/puppet@production] wancache: change deployment-prep to new Buster hosts

https://gerrit.wikimedia.org/r/669436

The old VMs were deleted.