Page MenuHomePhabricator
Create Task
Maniphest T399858

Cloud Ceph misbehaving on Debian Bookworm
Closed, ResolvedPublic
Actions

Assigned To
Andrew
Authored By
fnegri
Jul 17 2025, 2:45 PM
Tags
Referenced Files
F65560091: Screenshot 2025-07-22 at 10.46.51.png
Jul 22 2025, 8:48 AM
F65196721: Screenshot 2025-07-18 at 16.50.52.png
Jul 18 2025, 2:51 PM
F65141387: Screenshot 2025-07-18 at 10.15.36.png
Jul 18 2025, 8:16 AM
F65061658: Screenshot 2025-07-17 at 17.50.09.png
Jul 17 2025, 3:50 PM
F65061579: Screenshot 2025-07-17 at 17.48.44.png
Jul 17 2025, 3:50 PM
F65060728: Screenshot 2025-07-17 at 17.39.15.png
Jul 17 2025, 3:39 PM
F65060249: Screenshot 2025-07-17 at 17.32.35.png
Jul 17 2025, 3:35 PM
F65056781: image4.png
Jul 17 2025, 2:45 PM
View All 12 Files
Subscribers
Aklapper
Alien333
Andrew
dcaro
fnegri
• joanna_borun

Description

While working on T306820: [ceph] Upgrade to v16, we observed the following:

This task is to investigate what is the issue that caused Ceph to misbehave on the upgraded hosts.

Some graphs from the incident doc:

CPU usage on the affected hosts is high. This is depicted both in the percentiles graph as well individually

image5.png (1×1 px, 192 KB)

image8.png (365×1 px, 650 KB)

Memory usage in also high on the affected hosts, explaining swap usage and md resync (which happens on first access)

image1.png (359×1 px, 320 KB)

Running processes are really weird for the affected hosts:

image2.png (340×1 px, 462 KB)

And similarly Disk utilization

image4.png (406×1 px, 647 KB)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ceph codfw1dev: revert back to pacificoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

fnegri created this task.Jul 17 2025, 2:45 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptJul 17 2025, 2:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
fnegri added a subtask: T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures.Jul 17 2025, 2:45 PM
fnegri triaged this task as High priority.Jul 17 2025, 2:47 PM
fnegri edited projects, added cloud-services-team (FY2025/2026-Q1-Q2); removed cloud-services-team.
fnegri updated the task description. (Show Details)Jul 17 2025, 3:33 PM

cloudcephosd1006 was reimaged again to bookworm on 2025-07-16 at 18:21 UTC. The disk utilization graph shows a similar dip:

Screenshot 2025-07-17 at 17.32.35.png (1×2 px, 2 MB)

fnegri added a comment.Jul 17 2025, 3:39 PM

Same thing for node_procs_running:

Screenshot 2025-07-17 at 17.39.15.png (1×1 px, 255 KB)

fnegri added a comment.Jul 17 2025, 3:50 PM

Memory utilization on cloudcephosd1006 is increasing and it looks like it might crash the server in a few hours:

Screenshot 2025-07-17 at 17.48.44.png (1×2 px, 350 KB)

This is how it looks like on cloudcephosd1007:

Screenshot 2025-07-17 at 17.50.09.png (1×2 px, 398 KB)

fnegri added a project: Sustainability (Incident Followup).Jul 17 2025, 4:11 PM
fnegri added a project: SRE-OnFire.
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/2026-Q1-Q2) board.Jul 18 2025, 8:11 AM
fnegri changed the task status from Open to In Progress.Jul 18 2025, 8:16 AM
fnegri claimed this task.

Memory usage on cloudcephosd1006 did reset at 18:00 UTC yesterday (I think it was due to @dcaro tweaking a setting on the host). Now it's growing again... I'll keep an eye on it today.

Screenshot 2025-07-18 at 10.15.36.png (888×1 px, 96 KB)

I'm claiming this task temporarily as @dcaro is out today.

fnegri reassigned this task from fnegri to Andrew.Jul 18 2025, 2:51 PM

The growth rate is slowing, but it's not flatlining as I hoped... So the server is likely to crash in a few hours.

Screenshot 2025-07-18 at 16.50.52.png (1×2 px, 334 KB)

Re-assigning to @Andrew as I will log off shortly.

Andrew added a comment.Jul 20 2025, 10:44 AM

cloudcephosd1006 alerted over night; I'm going to reboot it so that we get another 36 or so hours of good behavior.

Stashbot added a comment.Jul 20 2025, 10:44 AM

Mentioned in SAL (#wikimedia-cloud) [2025-07-20T10:44:19Z] <andrewbogott> rebooting cloudcephosd1006 to give us another few days before the memory runs out. T399858

Stashbot added a comment.Jul 21 2025, 1:59 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T13:59:08Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T399858)

Stashbot added a comment.Jul 21 2025, 3:04 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:04:02Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T399858)

Stashbot added a comment.Jul 21 2025, 3:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:07:36Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T399858)

Stashbot added a comment.Jul 21 2025, 3:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:07:42Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T399858)

Stashbot added a comment.Jul 21 2025, 3:51 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:51:03Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T399858)

Stashbot added a comment.Jul 21 2025, 3:51 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:51:10Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T399858)

Stashbot added a comment.Jul 21 2025, 3:53 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:53:08Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T399858)

Stashbot added a comment.Jul 21 2025, 3:53 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:53:14Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T399858)

Stashbot added a comment.Jul 21 2025, 3:56 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T15:56:34Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T399858)

Stashbot added a comment.Jul 21 2025, 7:44 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-07-21T19:44:22Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) (T399858)

ops-monitoring-bot added a comment.Jul 22 2025, 8:35 AM

Cookbook cookbooks.sre.hosts.reimage was started by dcaro@cumin1003 for host cloudcephosd1006.eqiad.wmnet with OS bullseye

fnegri added a comment.Jul 22 2025, 8:48 AM

cloudcephosd1006 was reimaged again on 2025-07-21, but this time without keeping the data.

This made it consume all the memory even faster:

Screenshot 2025-07-22 at 10.46.51.png (1×2 px, 401 KB)

fnegri mentioned this in T399281: 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures.Jul 22 2025, 8:57 AM
Alien333 subscribed.Jul 22 2025, 8:58 AM
ops-monitoring-bot added a comment.Jul 22 2025, 9:55 AM

Cookbook cookbooks.sre.hosts.reimage started by dcaro@cumin1003 for host cloudcephosd1006.eqiad.wmnet with OS bullseye completed:

  • cloudcephosd1006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507220855_dcaro_2825422_cloudcephosd1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Andrew added a comment.Jul 22 2025, 2:42 PM

Our issue resembles this upstream report: https://serverfault.com/questions/1172161/osds-stability-issues-post-upgrade-to-ceph-quincy-17-2-8

• joanna_borun subscribed.Jul 24 2025, 8:52 AM
gerritbot added a comment.Aug 13 2025, 6:36 PM

Change #1178598 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] ceph codfw1dev: revert back to pacific

https://gerrit.wikimedia.org/r/1178598

gerritbot added a project: Patch-For-Review.Aug 13 2025, 6:36 PM
gerritbot added a comment.Aug 13 2025, 6:40 PM

Change #1178598 merged by Andrew Bogott:

[operations/puppet@production] ceph codfw1dev: revert back to pacific

https://gerrit.wikimedia.org/r/1178598

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2025, 7:31 PM
Andrew mentioned this in T402190: [ceph,eqiad1] upgrade from pacific->quincy.Aug 18 2025, 2:30 PM
Andrew mentioned this in T404249: [ceph,eqiad1] upgrade from quincy->reef (and bookworm).Sep 10 2025, 7:01 PM
Andrew closed subtask T400334: [ceph,codfw1dev] upgrade the hosts from pacific->quincy as Resolved.
taavi moved this task from Unsorted to Storage on the Cloud-VPS board.Sep 12 2025, 7:14 AM
Andrew closed this task as Resolved.Sep 23 2025, 8:39 PM

I am now upgrading the cluster to Bookworm + Reef (18.x) and that seems to bypass this issue.

fnegri moved this task from In progress to Done on the cloud-services-team (FY2025/2026-Q1-Q2) board.Sep 29 2025, 9:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits