Not sure if related, but just saw that puppet started failing in a bunch of nodes a few hours ago: https://tools-prometheus.wmflabs.org/tools/graph?g0.range_input=6h&g0.stacked=0&g0.expr=sum(puppet_agent_failed)&g0.tab=0&g1.range_input=6h&g1.expr=sum(puppet_agent_enabled)&g1.tab=0

In T259192#6346755, @Bstorm wrote:

Thre are some, largely unhelpful, recent discussions about that talk (as well as the link to the video) here https://www.reddit.com/r/ceph/comments/cznqoz/ceph_whole_cluster_backuprestore/

The basic long and short of it is that there's some things people are doing to save a bit by using radosgw, which doesn't give you as fast a hotswap cluster, but there it is. This also helps emphasize the not-that-great state of backups in ceph.