Change Details

I noticed the "thanos compact errors" alert firing (and recovering shortly after): ``` PROBLEM - Thanos compact has high percentage of failures on alert1001 is CRITICAL: job=thanos-compact ``` And indeed there have been sporadic errors the last couple of days: https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact?viewPanel=12&orgId=1&from=1624729924435&to=1625040554813 The timing nicely aligns with `sdf` failing on thanos-be1003 (T285664 + T285662) The error on the thanos-compact side has to do with multipart uploads, e.g. ``` Jun 29 20:55:36 thanos-fe2001 thanos-compact[5024]: level=error ts=2021-06-29T20:55:36.083160442Z caller=compact.go:386 msg="retriable error" err="compaction: group 0@10531109435386935375: upload of 01F9CRN1CWYQJVEV8Q86EHRX4V failed: upload chunks: upload file /srv/thanos-compact/compact/0@10531109435386935375/01F9CRN1CWYQJVEV8Q86EHRX4V/chunks/000001 as 01F9CRN1CWYQJVEV8Q86EHRX4V/chunks/000001: upload s3 object: One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag. ``` And I found this bug (fixed in swift 2.21, we're running 2.19 on buster) which sounds promising: https://bugs.launchpad.net/swift/+bug/1636663 Things to do: * [ ] Remove `sdf` from swift ring while the disk is being ordered/replace/etc, this will mitigate the errors * [ ] The thanos compact errors alert should be more "persistent" I think, and not flap * [ ] Investigate if the bug above does indeed fix this problem

I noticed the "thanos compact errors" alert firing (and recovering shortly after): ``` PROBLEM - Thanos compact has high percentage of failures on alert1001 is CRITICAL: job=thanos-compact ``` And indeed there have been sporadic errors the last couple of days: https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact?viewPanel=12&orgId=1&from=1624729924435&to=1625040554813 The timing nicely aligns with `sdf` failing on thanos-be1003 (T285664 + T285662) The error on the thanos-compact side has to do with multipart uploads, e.g. ``` Jun 29 20:55:36 thanos-fe2001 thanos-compact[5024]: level=error ts=2021-06-29T20:55:36.083160442Z caller=compact.go:386 msg="retriable error" err="compaction: group 0@10531109435386935375: upload of 01F9CRN1CWYQJVEV8Q86EHRX4V failed: upload chunks: upload file /srv/thanos-compact/compact/0@10531109435386935375/01F9CRN1CWYQJVEV8Q86EHRX4V/chunks/000001 as 01F9CRN1CWYQJVEV8Q86EHRX4V/chunks/000001: upload s3 object: One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag. ``` And I found this bug (fixed in swift 2.21, we're running 2.19 on buster) which sounds promising: https://bugs.launchpad.net/swift/+bug/1636663 Things to do: * [x] Remove `sdf` from swift ring while the disk is being ordered/replace/etc * [ ] Verify that removing `sdf` does indeed fix the errors * [ ] The thanos compact errors alert should be more "persistent" I think, and not flap * [ ] Investigate if the bug above does indeed fix this problem

I noticed the "thanos compact errors" alert firing (and recovering shortly after): ``` PROBLEM - Thanos compact has high percentage of failures on alert1001 is CRITICAL: job=thanos-compact ``` And indeed there have been sporadic errors the last couple of days: https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact?viewPanel=12&orgId=1&from=1624729924435&to=1625040554813 The timing nicely aligns with `sdf` failing on thanos-be1003 (T285664 + T285662) The error on the thanos-compact side has to do with multipart uploads, e.g. ``` Jun 29 20:55:36 thanos-fe2001 thanos-compact[5024]: level=error ts=2021-06-29T20:55:36.083160442Z caller=compact.go:386 msg="retriable error" err="compaction: group 0@10531109435386935375: upload of 01F9CRN1CWYQJVEV8Q86EHRX4V failed: upload chunks: upload file /srv/thanos-compact/compact/0@10531109435386935375/01F9CRN1CWYQJVEV8Q86EHRX4V/chunks/000001 as 01F9CRN1CWYQJVEV8Q86EHRX4V/chunks/000001: upload s3 object: One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag. ``` And I found this bug (fixed in swift 2.21, we're running 2.19 on buster) which sounds promising: https://bugs.launchpad.net/swift/+bug/1636663 Things to do: * [ * [x] Remove `sdf` from swift ring while the disk is being ordered/replace/etc, this will mitigate * [ ] Verify that removing `sdf` does indeed fix the errors * [ ] The thanos compact errors alert should be more "persistent" I think, and not flap * [ ] Investigate if the bug above does indeed fix this problem