Page MenuHomePhabricator

wmcs-backup fails to create differential when the data changed too much and fails the whole run
Open, Needs TriagePublic

Description

The parent tasks's T428867#12013155 uncovered a backy2 bug where a differential backup fails if it can't find enough matching pre-existing data in the last snapshot from which calculating the diff.
Instead of falling back to a full backup an exception is raised.
In this case wmcs-backup retries 3 times, but this is a deterministic failure, so it failed all 3 times and then made the whole backup run fail.

This should ideally be fixed in backy2 (that is mostly unmaintained) and also in the way wmcs-backup handles the failure. Ideally it should continue with the other volumes and fail the whole unit only if the % of failures is over a certain threshold. It could still exit with a failed status (to trigger monitoring once T428873 is fixed) for a single failed backup, but it should continue to backup all the others.

Event Timeline

Mentioned in SAL (#wikimedia-cloud) [2026-06-12T14:00:42Z] <volans> clearing cloudback-original snapshots of cinder volumes created in 2026 to allow the backups to not fail for T428995