⚓ T405446 Decommission druid100[7-8].eqiad.wmnet

		Status	Subtype	Assigned	Task
		Resolved	Request	VRiley-WMF	T403801 decommission druid100[7-8].eqiad.wmnet
		Resolved		• Stevemunene	T405446 Decommission druid100[7-8].eqiad.wmnet

• Stevemunene created this task.Sep 24 2025, 7:18 AM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptSep 24 2025, 7:18 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• Stevemunene moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.Sep 24 2025, 7:18 AM

Mentioned in SAL (#wikimedia-analytics) [2025-09-24T07:21:16Z] <stevemunene> change the Druid public (AQS) connection string to druid1011 as we decommission druid1007 T405446

• Stevemunene updated the task description. (Show Details)Sep 24 2025, 9:42 AM

• Stevemunene updated the task description. (Show Details)Sep 24 2025, 11:47 AM

hosts set to decommissioning

Druid hosts are done decommissioning, next is removing them from LVS

Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.

Depooled and removed the hosts from LVS

stevemunene@puppetserver1001:~$ sudo confctl select 'service=(druid-public-broker),name=druid1007.eqiad.wmnet' set/pooled=no
eqiad/druid-public/druid-public-broker/druid1007.eqiad.wmnet: pooled changed yes => no
WARNING:conftool.announce:conftool action : set/pooled=no; selector: service=(druid-public-broker),name=druid1007.eqiad.wmnet
stevemunene@puppetserver1001:~$ sudo confctl select 'service=(druid-public-broker),name=druid1008.eqiad.wmnet' set/pooled=no
eqiad/druid-public/druid-public-broker/druid1008.eqiad.wmnet: pooled changed yes => no
WARNING:conftool.announce:conftool action : set/pooled=no; selector: service=(druid-public-broker),name=druid1008.eqiad.wmnet
stevemunene@puppetserver1001:~$ sudo confctl select dc=eqiad,cluster=druid-public,service=druid-public-broker get
{"druid1009.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1010.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1011.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1012.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1013.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1007.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1008.eqiad.wmnet": {"weight": 10, "pooled": "no"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}

stevemunene@puppetserver1001:~$ sudo confctl select dc=eqiad,cluster=druid-public,service=druid-public-broker get
{"druid1009.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1010.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1011.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1012.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}
{"druid1013.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=druid-public,service=druid-public-broker"}

• Stevemunene updated the task description. (Show Details)Sep 29 2025, 3:48 PM

Icinga downtime and Alertmanager silence (ID=ef39a3c3-8f18-4c7a-a254-bfa590b603bc) set by stevemunene@cumin1003 for 2 days, 0:00:00 on 2 host(s) and their services with reason: Decommissioning druid_public hosts

druid[1007-1008].eqiad.wmnet

@Stevemunene is blocked on access keys, someone else has to run the decom cookbook.

Gehel mentioned this in T403801: decommission druid100[7-8].eqiad.wmnet.Oct 1 2025, 9:41 AM

Icinga downtime and Alertmanager silence (ID=42df5935-dfe8-456f-a594-adc05d19a266) set by stevemunene@cumin1003 for 2 days, 0:00:00 on 2 host(s) and their services with reason: Decommissioning druid_public hosts

druid[1007-1008].eqiad.wmnet

removed the keytabs

stevemunene@krb1002:~$ sudo manage_principals.py list *druid1007*
druid/an-druid1007.eqiad.wmnet@WIKIMEDIA
druid/druid1007.eqiad.wmnet@WIKIMEDIA
stevemunene@krb1002:~$ sudo manage_principals.py list *druid1008*
druid/druid1008.eqiad.wmnet@WIKIMEDIA
stevemunene@krb1002:~$ sudo manage_principals.py delete druid/druid1007.eqiad.wmnet@WIKIMEDIA
Principal successfully deleted.
stevemunene@krb1002:~$ sudo manage_principals.py delete druid/druid1008.eqiad.wmnet@WIKIMEDIA
Principal successfully deleted.
stevemunene@krb1002:~$ sudo manage_principals.py list *druid1008*
stevemunene@krb1002:~$ sudo manage_principals.py list *druid1007*
druid/an-druid1007.eqiad.wmnet@WIKIMEDIA

• Stevemunene updated the task description. (Show Details)Oct 3 2025, 1:45 PM

• Stevemunene closed this task as Resolved.Oct 3 2025, 1:50 PM

• Stevemunene moved this task from In Progress to Done on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 10 2025, 7:24 AM