Thanos compactor is failing to compact some blocks with this error:
/var/log/syslog:Jun 9 03:33:57 thanos-fe2001 thanos[7657]: level=warn ts=2020-06-09T03:33:57.045927309Z caller=intrumentation.go:54 msg="changing probe status" status=not-ready reason="error executing compaction : first pass of downsampling failed: downsampling to 5 min: open block 01EABJ7NFEJDDPS0TFHTVN70H2: invalid magic number 5b7b2268"
Upon inspection, these are the contents of the block's chunks (unsurprisingly 5b7b2268 is ascii for [{"h)
root@thanos-fe2001:~# s3cmd get s3://thanos/01EABJ7NFEJDDPS0TFHTVN70H2/chunks/000001 - | jq .
[
{
"hash": "f0db4e4b923918fbc3053ec871197ccd",
"last_modified": "2020-06-09T03:33:35.000000",
"bytes": 134217728,
"name": "/thanos+segments/01EABJ7NFEJDDPS0TFHTVN70H2/chunks/000001/OTMxZDA0OWYtZmIyOC00YmYxLThiZWUtY2RhZGQ4MGUyYmJh/1",
"content_type": "application/octet-stream"
},
{
"hash": "d3c66bcb5d80ecedc004663cb1698738",
"last_modified": "2020-06-09T03:33:35.000000",
"bytes": 118687527,
"name": "/thanos+segments/01EABJ7NFEJDDPS0TFHTVN70H2/chunks/000001/OTMxZDA0OWYtZmIyOC00YmYxLThiZWUtY2RhZGQ4MGUyYmJh/2",
"content_type": "application/octet-stream"
}
]Meaning that instead of the file's content we have downloaded the file's manifest for Swift's static large file support. In other words a list of pointers to actual Swift objects, as if Swift wasn't recognizing the upload as a SLO.