We can have a go at migrating the mwdebug* hosts to PHP8.1, until mw-experimental is ready, so to unblock any folks who may need them
- mwdebug1001
- mwdebug1002
- mwdebug2001
- mwdebug2002
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | Request | None | T421484 Reclaim parsoidtest1001 and testreduce1001 | ||
| Resolved | jijiki | T421483 parsoidtest1001.eqiad.wmnet retirement plan | |||
| Resolved | Jgiannelos | T420336 mw-parsoid improvements | |||
| Resolved | jijiki | T386246 Migrate parsoidtest functionality to kubernetes (mw-parsoid) | |||
| Resolved | None | T290536 Serve production traffic via Kubernetes | |||
| Resolved | MLechvien-WMF | T324003 mwdebug: people in the "deployment" group should be able to launch 'experimental' instances for testing purposes | |||
| Resolved | jijiki | T397498 Deprecate mwdebugXXXX hosts | |||
| Resolved | jijiki | T276994 Provide an mwdebug functionality on kubernetes (mw-experimental) | |||
| Resolved | None | T391452 Migrate mwdebug* hosts to PHP8.1 |
Event Timeline
Comment Actions
Change #1135020 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] logging: add support for php 8.1
Comment Actions
Change #1135020 merged by Effie Mouzeli:
[operations/puppet@production] logging: add support for php 8.1
Comment Actions
Change #1135021 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] switch mwdebug1002 to php8.1
Comment Actions
Change #1135021 merged by Effie Mouzeli:
[operations/puppet@production] switch mwdebug1002 to php8.1
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mwdebug1002.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mwdebug1002.eqiad.wmnet with OS bullseye completed:
- mwdebug1002 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"mwdebug1002.eqiad.wmnet": {"weight": 10, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=testserver,service=apache2"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504091247_jiji_633535_mwdebug1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Failed to run httpbb tests
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- No changes in confctl are needed to restore the previous state.
- Updated Netbox data from PuppetDB
Comment Actions
Change #1135698 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] php8.1: minor fixes
Comment Actions
Change #1135698 merged by Effie Mouzeli:
[operations/puppet@production] php8.1: minor fixes
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mwdebug1002.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mwdebug1002.eqiad.wmnet with OS bullseye completed:
- mwdebug1002 (PASS)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"mwdebug1002.eqiad.wmnet": {"weight": 10, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=testserver,service=apache2"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504101252_jiji_1006618_mwdebug1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Run of httpbb tests was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- No changes in confctl are needed to restore the previous state.
- Updated Netbox data from PuppetDB
Comment Actions
mwdebug1002 is done. I will reimage mwdebug2002 today as well, and if nothing comes up until next week, I will move forward with the remaining ones.
Comment Actions
Change #1135736 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] switch mwdebug2002 to php8.1
Comment Actions
Change #1135736 merged by Effie Mouzeli:
[operations/puppet@production] switch mwdebug2002 to php8.1
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mwdebug2002.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mwdebug2002.codfw.wmnet with OS bullseye completed:
- mwdebug2002 (PASS)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"mwdebug2002.codfw.wmnet": {"weight": 10, "pooled": "inactive"}, "tags": "dc=codfw,cluster=testserver,service=apache2"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504101552_jiji_1195674_mwdebug2002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Run of httpbb tests was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- No changes in confctl are needed to restore the previous state.
- Updated Netbox data from PuppetDB
Comment Actions
Change #1137290 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] switch mwdebug2001 to php8.1
Comment Actions
Change #1137290 merged by Effie Mouzeli:
[operations/puppet@production] switch mwdebug2001 to php8.1
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mwdebug2001.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mwdebug2001.codfw.wmnet with OS bullseye completed:
- mwdebug2001 (PASS)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"mwdebug2001.codfw.wmnet": {"weight": 10, "pooled": "inactive"}, "tags": "dc=codfw,cluster=testserver,service=apache2"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504171457_jiji_620454_mwdebug2001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Run of httpbb tests was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- No changes in confctl are needed to restore the previous state.
- Updated Netbox data from PuppetDB
Comment Actions
Change #1138363 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] switch mwdebug1001 to php8.1
Comment Actions
Change #1138363 merged by Effie Mouzeli:
[operations/puppet@production] switch mwdebug1001 to php8.1
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mwdebug1001.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mwdebug1001.eqiad.wmnet with OS bullseye completed:
- mwdebug1001 (PASS)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"mwdebug1001.eqiad.wmnet": {"weight": 10, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=testserver,service=apache2"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Set boot media to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504231437_jiji_1919800_mwdebug1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Run of httpbb tests was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- No changes in confctl are needed to restore the previous state.
- Updated Netbox data from PuppetDB