Page MenuHomePhabricator
Create Task
Maniphest T309789

[ceph] Upgrade hosts to bullseye
Open, In Progress, HighPublic
Actions

Description

This was blocked by not having the packages, but with T309786 we unblock this one.

codfw

  • cloudcephmon2004-dev
  • cloudcephmon2005-dev
  • cloudcephmon2006-dev
  • cloudcephosd2001-dev
  • cloudcephosd2002-dev
  • cloudcephosd2003-dev

eqiad

  • cloudcephosd1006
  • cloudcephosd1007
  • cloudcephosd1008
  • cloudcephosd1009
  • cloudcephosd1011
  • cloudcephosd1012
  • cloudcephosd1013
  • cloudcephosd1014
  • cloudcephosd1015
  • cloudcephosd1016
  • cloudcephosd1017
  • cloudcephosd1018
  • cloudcephosd1019
  • cloudcephosd1020
  • cloudcephosd1022
  • cloudcephosd1023
  • cloudcephosd1024
  • cloudcephmon1001
  • cloudcephmon1002
  • cloudcephmon1003

Details

SubjectRepoBranchLines +/-
cloudcephosd1006: update interface namesoperations/puppetproduction+2 -2
Use our own alert managingoperations/cookbookswmcs+154 -30
Customize query in gerrit

Related Objects
Search...

Event Timeline

dcaro triaged this task as High priority.Jun 2 2022, 2:38 PM
dcaro created this task.
dcaro moved this task from To refine to Today on the User-dcaro board.
dcaro added a parent task: T309786: ceph: upgrad to latest v15 before upgrade to v16.
Stashbot added a comment.Jun 13 2022, 9:15 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-06-13T09:15:09Z] <wm-bot2> Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Jun 13 2022, 11:03 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-06-13T11:03:26Z] <wm-bot2> Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Jun 13 2022, 11:04 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-06-13T11:04:19Z] <wm-bot2> Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Jun 13 2022, 11:05 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-06-13T11:05:25Z] <wm-bot2> Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Jun 13 2022, 11:07 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-06-13T11:07:48Z] <wm-bot2> Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Jun 13 2022, 11:08 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-06-13T11:08:34Z] <wm-bot2> Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

Stashbot added a comment.Jun 13 2022, 11:14 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2022-06-13T11:14:52Z] <wm-bot2> Finished rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

gerritbot added a comment.Jun 13 2022, 11:31 AM

Change 805108 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] Use our own alert managing

https://gerrit.wikimedia.org/r/805108

gerritbot added a project: Patch-For-Review.Jun 13 2022, 11:31 AM
gerritbot added a comment.Jul 4 2022, 9:34 AM

Change 805108 merged by jenkins-bot:

[operations/cookbooks@wmcs] Use our own alert managing

https://gerrit.wikimedia.org/r/805108

dcaro moved this task from Today to Refined on the User-dcaro board.Aug 23 2022, 8:12 AM
dcaro lowered the priority of this task from High to Medium.Aug 26 2022, 10:28 AM
dcaro raised the priority of this task from Medium to High.
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Sep 27 2022, 9:29 PM
dcaro mentioned this in rCCKBeb2cb8957790: Use our own alert managing.Dec 14 2022, 3:33 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 7:25 PM
fnegri moved this task from Kanban to Doing? (legacy column) on the cloud-services-team board.
fnegri moved this task from Doing? (legacy column) to Inbox on the cloud-services-team board.Jan 19 2023, 12:55 PM
fnegri edited projects, added cloud-services-team (FY2022/2023-Q3); removed cloud-services-team.Jan 19 2023, 2:31 PM
dcaro changed the task status from Open to In Progress.Jan 31 2023, 9:57 AM
dcaro moved this task from Backlog to In progress on the cloud-services-team (FY2022/2023-Q3) board.
dcaro moved this task from In progress to Blocked on the cloud-services-team (FY2022/2023-Q3) board.Feb 7 2023, 3:11 PM
• nskaggs mentioned this in T332400: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster .Mar 17 2023, 2:53 PM
• nskaggs edited projects, added cloud-services-team (FY2022/2023-Q4); removed cloud-services-team (FY2022/2023-Q3).Apr 11 2023, 9:18 PM
fnegri moved this task from Backlog to Planned Goals on the cloud-services-team (FY2022/2023-Q4) board.Apr 12 2023, 2:40 PM
fnegri added a project: Goal.Apr 19 2023, 2:28 PM
fnegri moved this task from Planned Goals to In progress on the cloud-services-team (FY2022/2023-Q4) board.
dcaro moved this task from In progress to Planned Goals on the cloud-services-team (FY2022/2023-Q4) board.Apr 26 2023, 2:42 PM
fnegri edited projects, added cloud-services-team (FY2023/2024-Q1-Q2); removed cloud-services-team (FY2022/2023-Q4).Jul 26 2023, 4:03 PM
fnegri moved this task from Backlog to Planned Goals on the cloud-services-team (FY2023/2024-Q1-Q2) board.Jul 26 2023, 4:11 PM
fnegri moved this task from Planned Goals to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.Aug 7 2023, 2:53 PM
dcaro added a parent task: T349619: Migrate roles to puppet7.Dec 7 2023, 3:59 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 7 2023, 4:10 PM
fnegri renamed this task from ceph: upgrade hosts to bullseye to [ceph] Upgrade hosts to bullseye.Jan 22 2024, 5:34 PM
fnegri added a project: Cloud-VPS.
fnegri edited projects, added cloud-services-team (FY2023/2024-Q3-Q4); removed cloud-services-team (FY2023/2024-Q1-Q2).Feb 1 2024, 11:13 AM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q3-Q4) board.
taavi moved this task from Unsorted to Storage on the Cloud-VPS board.Tue, Jun 4, 3:57 PM
dcaro updated the task description. (Show Details)Wed, Jun 26, 9:55 AM
Stashbot added a comment.Wed, Jun 26, 9:59 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T09:59:53Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)

Stashbot added a comment.Wed, Jun 26, 10:00 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T10:00:01Z] <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T309789)

Stashbot added a comment.Wed, Jun 26, 10:00 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T10:00:28Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)

Stashbot added a comment.Wed, Jun 26, 11:16 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T11:16:29Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789)

ops-monitoring-bot added a comment.Wed, Jun 26, 11:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host cloudcephosd1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, Jun 26, 12:30 PM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host cloudcephosd1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudcephosd1006.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Wed, Jun 26, 1:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host cloudcephosd1006.eqiad.wmnet with OS bullseye

Stashbot added a comment.Wed, Jun 26, 2:45 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T14:45:37Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)

Stashbot added a comment.Wed, Jun 26, 2:45 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T14:45:43Z] <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)

Stashbot added a comment.Wed, Jun 26, 2:45 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T14:45:46Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)

Stashbot added a comment.Wed, Jun 26, 2:46 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T14:45:59Z] <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)

gerritbot added a comment.Wed, Jun 26, 3:05 PM

Change #1049964 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] cloudcephosd1006: update interface names

https://gerrit.wikimedia.org/r/1049964

gerritbot added a project: Patch-For-Review.Wed, Jun 26, 3:05 PM

Change #1049964 merged by David Caro:

[operations/puppet@production] cloudcephosd1006: update interface names

https://gerrit.wikimedia.org/r/1049964

Stashbot added a comment.Wed, Jun 26, 3:09 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T15:09:32Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)

ops-monitoring-bot added a comment.Wed, Jun 26, 3:15 PM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host cloudcephosd1006.eqiad.wmnet with OS bullseye completed:

  • cloudcephosd1006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202406261409_root_1179650_cloudcephosd1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Wed, Jun 26, 3:18 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T15:18:00Z] <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)

Maintenance_bot removed a project: Patch-For-Review.Wed, Jun 26, 3:31 PM
Stashbot added a comment.Wed, Jun 26, 3:46 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T15:46:48Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)

Stashbot added a comment.Wed, Jun 26, 7:18 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-26T19:18:17Z] <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)

dcaro updated the task description. (Show Details)Thu, Jun 27, 6:48 AM
Stashbot added a comment.Thu, Jun 27, 12:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-27T12:07:16Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)

Stashbot added a comment.Thu, Jun 27, 1:44 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-06-27T13:44:00Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789)

ops-monitoring-bot added a comment.Thu, Jun 27, 2:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by dcaro@cumin1002 for host cloudcephosd1007.eqiad.wmnet with OS bullseye

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL