Decommission clouddb1013-clouddb1020
Open, MediumPublic
Actions

Assigned To

Authored By

	Marostegui
	Wed, Aug 5, 7:48 AM

Project Tags

Referenced Files

None

Subscribers

Description

These hosts have been freshed by clouddb1022-clouddb1033 (this includes the expansion) see https://phabricator.wikimedia.org/T409557

The old ones should be depooled and leave without traffic for a month, so we can test the new ones on its own and then once that has gone well, we can decommission them:

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	clouddb1013-clouddb1020: Disable notifications	Marostegui	operations/puppet	production	+7 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		Marostegui	T434048 Decommission clouddb1013-clouddb1020
Resolved		Marostegui	T409557 Productionize new clouddb* hosts (clouddb1022-1033)
Open		Marostegui	T404715 Setup x4 section
Resolved		Marostegui	T406550 Productionize db126[0-3]
Resolved		Marostegui	T406551 Productionize db224[5-8]
Open		Marostegui	T431309 Setup x4 on sanitarium hosts
Open		None	T431984 dbctl instance edit crashes with "exceptions must derive from BaseException" instead of showing real validation errors
Resolved	PRODUCTION ERROR	Zabe	T431989 PHP Warning: Undefined array key "x4"
Resolved		Marostegui	T410365 SystemdUnitFailed: generate-mysqld-exporter-config.service on prometheus100[56]:9100
Resolved		cmooney	T430132 cloudlb100X cannot talk to new host clouddb1026

Event Timeline

Marostegui created this task.Wed, Aug 5, 7:48 AM

Restricted Application added a project: tools-platform-team. · View Herald TranscriptWed, Aug 5, 7:48 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Marostegui claimed this task.Wed, Aug 5, 7:49 AM

Marostegui triaged this task as Medium priority.

Marostegui added a subtask: T409557: Productionize new clouddb* hosts (clouddb1022-1033).

Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2026-08-05T07:53:44Z] <marostegui> Depool clouddb1013:s1 T434048

Mentioned in SAL (#wikimedia-operations) [2026-08-05T07:54:33Z] <marostegui> Depool clouddb1014 (s2,s7) T434048

Mentioned in SAL (#wikimedia-operations) [2026-08-05T07:57:56Z] <marostegui> Depool clouddb1015 (s4,s6) T434048

Mentioned in SAL (#wikimedia-operations) [2026-08-05T07:59:19Z] <marostegui> Depool clouddb1016 (s5,s8) T434048

Mentioned in SAL (#wikimedia-operations) [2026-08-05T08:01:30Z] <marostegui> Depool clouddb1017 (s1) T434048

Mentioned in SAL (#wikimedia-operations) [2026-08-05T08:01:58Z] <marostegui> Depool clouddb1018 (s2,s7) T434048

Mentioned in SAL (#wikimedia-operations) [2026-08-05T08:02:21Z] <marostegui> Depool clouddb1020 (s5,s8) T434048

All those clouddb hosts have been depooled. Only the new ones are serving traffic now fyi @taavi @fnegri

Change #1321531 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1013-clouddb1020: Disable notifications

https://gerrit.wikimedia.org/r/1321531

Change #1321531 merged by Marostegui: