Page MenuHomePhabricator
Create Task
Maniphest T402596

upgrade people servers to trixie
Closed, ResolvedPublic
Actions

Description

people.wikimedia.org backends have traditionally been a good candidate to upgrade early when a new distro version is released

check if the install of trixie goes well

they are (mostly) just a simple apache and not critical. good way to see if any adjustments are needed before doing more important services.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site: add peopleweb role to new peopleweb hosts againoperations/puppetproduction+1 -1
switch people service aliases in eqiad and codfw to new trixie hostsoperations/dnsmaster+4 -4
peopleweb: make people2004 the new rsync sourceoperations/puppetproduction+1 -3
bacula: Ignore backup failures from people1005 & people2004operations/puppetproduction+4 -0
peopleweb: allow multiple rsync destination hostsoperations/puppetproduction+5 -2
site: add regex to include people2004 with insetup roleoperations/puppetproduction+2 -2
site: add people1005 with insetup roleoperations/puppetproduction+5 -0
site: add peopleweb role to people1005operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Aug 21 2025, 9:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 21 2025, 9:08 PM
Dzahn updated the task description. (Show Details)Aug 21 2025, 9:10 PM
gerritbot added a comment.Aug 21 2025, 9:20 PM

Change #1180977 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: add people1005 with insetup role

https://gerrit.wikimedia.org/r/1180977

gerritbot added a project: Patch-For-Review.Aug 21 2025, 9:20 PM

Change #1180977 merged by Dzahn:

[operations/puppet@production] site: add people1005 with insetup role

https://gerrit.wikimedia.org/r/1180977

ops-monitoring-bot added a comment.Aug 21 2025, 9:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host people1005.eqiad.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Aug 21 2025, 9:30 PM
ops-monitoring-bot added a comment.Aug 21 2025, 10:04 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host people1005.eqiad.wmnet with OS trixie completed:

  • people1005 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508212153_dzahn_2239507_people1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Aug 21 2025, 10:13 PM

Change #1180990 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: add peopleweb role to people1005

https://gerrit.wikimedia.org/r/1180990

gerritbot added a project: Patch-For-Review.Aug 21 2025, 10:13 PM
gerritbot added a comment.Aug 22 2025, 5:07 PM

Change #1180990 merged by Dzahn:

[operations/puppet@production] site: add peopleweb role to people1005

https://gerrit.wikimedia.org/r/1180990

Dzahn changed the task status from Open to Stalled.Aug 22 2025, 5:18 PM

currently blocked on T402668

Dzahn mentioned this in T402669: ProbeDown - people1005.Aug 22 2025, 5:29 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 22 2025, 5:30 PM
Dzahn added a subtask: T402669: ProbeDown - people1005.Aug 22 2025, 5:31 PM
Stashbot added a comment.Aug 22 2025, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2025-08-22T17:32:42Z] <dzahn@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on people1005.eqiad.wmnet with reason: T402596

gerritbot added a comment.Aug 22 2025, 8:38 PM

Change #1181194 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: add regex to include people2004 with insetup role

https://gerrit.wikimedia.org/r/1181194

gerritbot added a project: Patch-For-Review.Aug 22 2025, 8:38 PM

Change #1181194 merged by Dzahn:

[operations/puppet@production] site: add regex to include people2004 with insetup role

https://gerrit.wikimedia.org/r/1181194

ops-monitoring-bot added a comment.Aug 22 2025, 8:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host people2004.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Aug 22 2025, 9:23 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host people2004.codfw.wmnet with OS trixie completed:

  • people2004 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508222108_dzahn_3903597_people2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Aug 22 2025, 9:30 PM
Stashbot added a comment.Aug 26 2025, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-08-26T17:44:39Z] <dzahn@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on people1005.eqiad.wmnet with reason: T402596

Dzahn added a subtask: T402953: SystemdUnitFailed - envoyproxy on people1005.Aug 26 2025, 5:54 PM
Dzahn added a parent task: T402584: Upgrade Envoy to v1.26.8 and drop buster.
Dzahn closed subtask T402953: SystemdUnitFailed - envoyproxy on people1005 as Resolved.Aug 26 2025, 5:57 PM
gerritbot added a comment.Sep 3 2025, 3:17 PM

Change #1184553 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: add peopleweb role to new peopleweb hosts again

https://gerrit.wikimedia.org/r/1184553

gerritbot added a project: Patch-For-Review.Sep 3 2025, 3:17 PM
gerritbot added a comment.Sep 3 2025, 3:21 PM

Change #1184553 merged by Dzahn:

[operations/puppet@production] site: add peopleweb role to new peopleweb hosts again

https://gerrit.wikimedia.org/r/1184553

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2025, 3:31 PM
Dzahn changed the task status from Stalled to In Progress.Sep 3 2025, 4:09 PM
Dzahn added a subtask: T403526: SystemdUnitFailed - people1005 - envoyproxy.

We have envoy now and after some debugging in the linked subtask.. envoy and apache are working now.

gerritbot added a comment.Sep 3 2025, 7:04 PM

Change #1184573 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] peopleweb: add additional rsync destination hosts

https://gerrit.wikimedia.org/r/1184573

gerritbot added a project: Patch-For-Review.Sep 3 2025, 7:04 PM
gerritbot added a comment.Sep 3 2025, 10:54 PM

Change #1184573 merged by Dzahn:

[operations/puppet@production] peopleweb: allow multiple rsync destination hosts

https://gerrit.wikimedia.org/r/1184573

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2025, 11:30 PM
LSobanski assigned this task to Dzahn.Sep 5 2025, 8:38 AM
LSobanski triaged this task as Low priority.
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.
gerritbot added a comment.Sep 9 2025, 1:40 PM

Change #1186509 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] bacula: Ignore backup failures from people1005 & people2004

https://gerrit.wikimedia.org/r/1186509

gerritbot added a project: Patch-For-Review.Sep 9 2025, 1:40 PM
Dzahn changed the task status from In Progress to Stalled.Sep 9 2025, 5:57 PM

Putting this on pause because backups were failing and that turned out to be T404114.

gerritbot added a comment.Sep 9 2025, 5:59 PM

Change #1186509 merged by Dzahn:

[operations/puppet@production] bacula: Ignore backup failures from people1005 & people2004

https://gerrit.wikimedia.org/r/1186509

Maintenance_bot removed a project: Patch-For-Review.Sep 9 2025, 6:31 PM
jcrespo closed subtask T404114: Trixie bacula-fd package incompatible with our bacula installation as Resolved.Sep 11 2025, 4:28 PM
Dzahn changed the task status from Stalled to Open.Sep 11 2025, 4:29 PM

unstalled. bacula issue was resolved

gerritbot added a comment.Sep 12 2025, 6:50 PM

Change #1187884 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] peopleweb: make people2004 the new rsync source

https://gerrit.wikimedia.org/r/1187884

gerritbot added a project: Patch-For-Review.Sep 12 2025, 6:50 PM
gerritbot added a comment.Sep 12 2025, 6:54 PM

Change #1187885 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] switch people service aliases in eqiad and codfw to new trixie hosts

https://gerrit.wikimedia.org/r/1187885

gerritbot added a comment.Sep 15 2025, 5:52 PM

Change #1187885 merged by Dzahn:

[operations/dns@master] switch people service aliases in eqiad and codfw to new trixie hosts

https://gerrit.wikimedia.org/r/1187885

gerritbot added a comment.Sep 15 2025, 5:53 PM

Change #1187884 merged by Dzahn:

[operations/puppet@production] peopleweb: make people2004 the new rsync source

https://gerrit.wikimedia.org/r/1187884

Dzahn added a subtask: T404631: ProbeDown - people2004.Sep 15 2025, 6:27 PM
Dzahn closed subtask T404631: ProbeDown - people2004 as Resolved.
Maintenance_bot removed a project: Patch-For-Review.Sep 15 2025, 6:31 PM
Dzahn closed this task as Resolved.Sep 15 2025, 7:10 PM

rsynced data again, checked httpbb tests, monitoring fixed.

Sent mail to sre-at-large to announce this.

Dzahn added a subtask: T408713: decom old people VMs / finish people host upgrade.Oct 29 2025, 4:15 PM
Stashbot mentioned this in T408713: decom old people VMs / finish people host upgrade.Oct 29 2025, 4:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-29T16:18:36Z] <mutante> shutting down people1004.eqiad.wmnet, people2003.codfw.wmnet - T408713 T402596

Dzahn changed the status of subtask T408713: decom old people VMs / finish people host upgrade from Open to In Progress.Oct 29 2025, 4:20 PM
Dzahn changed the status of subtask T408713: decom old people VMs / finish people host upgrade from In Progress to Stalled.Nov 6 2025, 6:00 PM
Dzahn changed the status of subtask T408713: decom old people VMs / finish people host upgrade from Stalled to In Progress.Nov 10 2025, 6:39 PM
Dzahn closed subtask T408713: decom old people VMs / finish people host upgrade as Resolved.Nov 10 2025, 6:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits