Just realized we (still) have 4 people* VMs and we have not finished the upgrade yet.
people[2003-2004].codfw.wmnet,people[1004-1005].eqiad.wmnet
At least the old hosts are not decom'ed.
Double check that is all that is left and do the proper decoms.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| site: remove decom'ed people bookworm machines | operations/puppet | production | +1 -6 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | RLazarus | T380211 Upgrade Envoy to >= 1.24 | |||
| Resolved | RLazarus | T402584 Upgrade Envoy to v1.26.8 and drop buster | |||
| Resolved | Dzahn | T402596 upgrade people servers to trixie | |||
| Resolved | Dzahn | T408713 decom old people VMs / finish people host upgrade |
Mentioned in SAL (#wikimedia-operations) [2025-10-29T16:18:36Z] <mutante> shutting down people1004.eqiad.wmnet, people2003.codfw.wmnet - T408713 T402596
hosts shut down after sending a warning message via wall - but VMs not destroyed just yet
someone or something booted the old VMs again that I had shut down.. about 8 days ago. unclear why
Mentioned in SAL (#wikimedia-operations) [2025-11-06T17:38:37Z] <mutante> shutting down people1004 and people2003 - had already shut them down on Oct 29 but someone or something booted them again T408713
The decom cookbook warned me that IPs of these machines appear in:
deployment-charts/helmfile.d/services/machinetranslation/values.yaml.
There was another long-term ticket (where was it?) about this and people hosts being used to dump data for a production service.
We really need to stop that. Especially with hardcoded IPs in other repos.
Aborted the decom cookbook and started to ask in #wikimedia-ml
cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: people1004.eqiad.wmnet
Change #1203502 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] site: remove decom'ed people bookworm machines
cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: people2003.codfw.wmnet
Change #1203502 merged by Dzahn:
[operations/puppet@production] site: remove decom'ed people bookworm machines