Page MenuHomePhabricator
Create Task
Maniphest T410152

Disk space saturation (/srv) on Titan hosts
Open, Needs TriagePublic
Actions

Assigned To
None
Authored By
tappof
Nov 14 2025, 4:21 PM
Project Tags
Referenced Files
F71626623: image.png
Jan 29 2026, 9:47 PM
F71626530: image.png
Jan 29 2026, 9:47 PM
F71626485: image.png
Jan 29 2026, 9:47 PM
Restricted File
Jan 19 2026, 6:16 PM
Restricted File
Jan 19 2026, 6:15 PM
Restricted File
Jan 16 2026, 11:23 AM
Restricted File
Jan 16 2026, 11:23 AM
Restricted File
Jan 16 2026, 8:56 AM
View All 19 Files
Subscribers
Aklapper
andrea.denisse
Blake
Clement_Goubert
dr0ptp4kt
herron
hnowlan
View All 9 Subscribers

Description

We received alerts related to /srv on hosts titan1001, titan1002, and titan2002 (titan2001 has a larger RAID).

They started increasing around October 3rd.

image.png (2,972×797 px, 75 KB)

A couple of weeks earlier, we also noticed an increase in the bytes transferred from Swift, reasonably due to: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1184566

image.png (2,969×762 px, 94 KB)

https://w.wiki/G5qB

We tried deleting older files on /srv/thanos-store (after depooling host titan1001 and stopping thanos-store), but the cache was immediately repopulated to exactly the same size as before, in just a few minutes…

image.png (2,970×787 px, 66 KB)

Moreover, we noticed some inconsistencies in the storage configuration between the titan hosts:

  • titan1001 and titan1002 have some spare disks already present but they are missing on titan2002;
  • titan2001 is striping multiple partitions on the same disk.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
thanos/compact: increase meta-fetch goroutines to fix compactor inactivityTiziano Foglioperations/puppetproduction+1 -1
thanos/compact: revert concurrency to 2Tiziano Foglioperations/puppetproduction+1 -1
thanos/compact: increase meta-fetch goroutines to fix compactor inactivityTiziano Foglioperations/puppetproduction+16 -10
thanos-compact: reduce concurrency to 2Tiziano Foglioperations/puppetproduction+1 -1
prometheus/k8s-pods-metrics: move drop job to metric_relabel_configsCwhiteoperations/puppetproduction+3 -1
prometheus/k8s-pods-metrics: drop mediawiki_action_api_modules_latency metricCwhiteoperations/puppetproduction+6 -6
prometheus/k8s-pod: drop mediawiki_action_api_modules_lantecy metricTiziano Foglioperations/puppetproduction+6 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Nov 17 2025, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2025-11-17T08:32:27Z] <tappof> titan1002: switch /srv mount from /dev/md2 to /dev/vg0/srv T410152

tappof added a comment.Nov 17 2025, 9:11 AM

/srv was also moved to the VG on titan1002.

tappof added a parent task: T361229: titan200[12] RAM/SSD upgrade coordination.Nov 17 2025, 1:22 PM
Stashbot added a comment.Nov 17 2025, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-17T13:53:10Z] <tappof> titan2002: switch /srv mount from /dev/md2 to /dev/vg0/srv T410152

tappof added a comment.Nov 17 2025, 1:58 PM

/srv was also moved to the VG on titan2002.

tappof added a comment.EditedNov 17 2025, 4:49 PM

Just a heads-up: on hosts titan1002 and titan2002, the disks previously associated with the software RAID0 have not yet been added to the new VG. With the current settings, we’ve gained roughly 10 days to further explore the issue.

How changing the cutoff has affected disk utilization on /srv/thanos-store (tentative analysis)

  1. We changed the --max-time parameter of Thanos Store from -15d to -1d.
  2. This effectively caused a 5x increase in the amount of data transferred from the object store.
  3. One compactor cycle takes roughly 2 weeks.
  4. Just considering point 1, we are potentially increasing the amount of data that can reside under /srv/thanos-store.
  5. Every day, the compactor creates new fresh blocks. Blocks are considered for downsampling only when they are older than 2 days ("All raw resolution metrics that are older than 40 hours are downsampled at a 5m resolution").
  6. Over time, with a cutoff of -1d, Thanos Store will constantly cache the new blocks created by the compactor (compacted and/or downsampled).
  7. In the short term, however, the blocks already present in the store have not yet been processed as deletable by the compactor, so they are effectively still valid. The store stops using them until they are no longer valid (i.e., removable), after which it starts requesting new blocks — but this time much more frequently, since previously a block remained valid for 2 weeks (with --max-time -15d and a compaction cycle duration of ~14d, blocks were probabilistically replaced almost simultaneously). Today, a block remains valid for about one day because it is then compacted or downsampled, or both (and therefore effectively becomes a new block with its data merged with other blocks).

I’d like to revert the patch to see if restoring the old cutoff value will reverse the trend in disk utilization. If I’m correct, it will take some time to return to the old pattern.

We also need to align the storage configurations between hosts, taking into account T396862: Improve titan hosts stateless-ness.

herron added a comment.Nov 17 2025, 6:20 PM

Thanks for the summary, before we pull the trigger on the revert could we try a couple alternatives?

a) Try setting the download strategy to lazy, and pair this with a frequent cleanup job on the data-dir. In theory this will prevent the data-dir from immediately refilling after pruning old/large blocks and only keep actively used index headers on disk.

--store.index-header-lazy-download-strategy=eager
                                 Strategy of how to download index headers lazily. Supported values: eager, lazy. If eager, always download index header during initial load. If lazy, download index header during query time.

b) Test the impact of disabling index header cache and relying solely on memcached - This may be too slow, but if performance was ok it would remove the on disk header cache requirements entirely.

--cache-index-header       Cache TSDB index-headers on disk to reduce startup time. When set to true, Thanos Store will download index headers from remote object storage on startup and create a header file on disk. Use --data-dir to set the directory in which index headers will be downloaded.

B should be fairly straightforward to test, we could set header cache to false on a single host while watching overall thanos performance and depool+revert at the first sign of trouble.

tappof added a comment.Nov 18 2025, 7:47 AM

Sure, I think we can explore any route that will fix our scenario. That said, we’re quite happy with the current Thanos/Prometheus performance, so I’d like to better understand the real needs behind having such a short cutoff window of just one day.

I was looking at the documentation and noticed that for backfilling metrics we request data from the Thanos Querier. Isn’t it able to request data from the sidecar as well when the query comes from promtool?

hnowlan moved this task from Inbox to FY2025/2026-Q2 on the SRE Observability board.Nov 19 2025, 3:28 PM
hnowlan edited projects, added: SRE Observability (FY2025/2026-Q2); removed: SRE Observability.
tappof added a comment.Nov 20 2025, 12:04 PM

It is now affecting the compactor as well.

11:23:34 jinxer-wm │ FIRING: DiskSpace: Disk space titan2001:9100:/srv 0.3483% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
11:38:34 jinxer-wm │ RESOLVED: DiskSpace: Disk space titan2001:9100:/srv 0.004537% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
12:25:12 jinxer-wm │ FIRING: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted

image.png (927×1,111 px, 113 KB)

tappof added a comment.Nov 20 2025, 3:12 PM

I’d suggest reverting the patch, as the compactor is currently unable to do its job. This could lead to a thrashing situation that would be harder to recover from than the one we’re experiencing now. Once we’ve confirmed we’re no longer in troubled waters, we can investigate why backfilling metrics was difficult without such a cutoff and eventually evaluate alternatives.
I’m not entirely sure this is the root cause, but I prefer to give it a try before changing other configurations that could impact performance.

Stashbot added a comment.Nov 24 2025, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-24T14:22:19Z] <tappof> Remove unused md2 and add its devices to vg0 on titan1002 T410152

Stashbot added a comment.Nov 24 2025, 2:42 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-24T14:42:31Z] <tappof> Remove unused md2 and add its devices to vg0 on titan2002 T410152

hnowlan subscribed.Nov 26 2025, 11:24 AM
tappof mentioned this in T411273: Thanos (store|query-frontend) memcached cache in bad status.Nov 28 2025, 2:46 PM
tappof added a comment.Nov 28 2025, 4:10 PM

To avoid a revert on Friday and to be in the driver’s seat during the weekend, 100 GB were added to the VGs on titan1001, titan1002, and titan2002.

image.png (1,254×328 px, 53 KB)

Stashbot added a comment.Nov 28 2025, 4:17 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-28T16:17:01Z] <tappof> Added 100 GB to /srv LV on titan1001/1002/2002 (T410152)

fgiunchedi mentioned this in T411343: thanos-store OOMing on titan eqiad.Dec 1 2025, 9:26 AM
Stashbot added a comment.Dec 1 2025, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2025-12-01T15:56:40Z] <tappof> "thanos-store: set cutoff days to 1" reverted on titan1001 (1/4) T410152

Stashbot added a comment.Dec 1 2025, 4:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-12-01T16:28:35Z] <tappof> "thanos-store: set cutoff days to 1" reverted on titan1002 (2/4) T410152

Stashbot added a comment.Dec 1 2025, 5:17 PM

Mentioned in SAL (#wikimedia-operations) [2025-12-01T17:17:16Z] <tappof> "thanos-store: set cutoff days to 1" reverted on titan2002 (3/4) T410152

tappof added a comment.Dec 1 2025, 5:32 PM

Another side effect:

image.png (951×1,206 px, 116 KB)

Stashbot added a comment.Dec 1 2025, 5:39 PM

Mentioned in SAL (#wikimedia-operations) [2025-12-01T17:39:16Z] <tappof> "thanos-store: set cutoff days to 1" reverted on titan2001 (4/4) T410152

tappof added a parent task: T349521: Prometheus/Pyrra: establish backfill process for recording rules.Dec 2 2025, 10:20 AM
tappof mentioned this in T349521: Prometheus/Pyrra: establish backfill process for recording rules.Dec 2 2025, 2:19 PM
andrea.denisse subscribed.Dec 2 2025, 8:17 PM
tappof added a comment.Dec 4 2025, 10:56 AM

The trend has changed after the revert.

image.png (1,211×1,013 px, 53 KB)

dr0ptp4kt subscribed.Dec 4 2025, 9:30 PM
tappof added a comment.Dec 18 2025, 2:06 PM

We’re still in troubled waters with this issue. After the revert, an initial cleanup happened twice (which was promising, since I had expected the cleanup to happen gradually), but the filesystem usage started increasing again.

I found something interesting:

  1. A couple of init commands which give some information related to the blocks:
> thanos tools bucket --objstore.config-file /etc/thanos-compact/objstore.yaml ls -o json > /tmp/bucket
> cat  /tmp/bucket | jq  '{udli: .ulid, minTime: .minTime, maxTime: .maxTime, labels: .thanos.labels}' > /tmp/bucket_short
  1. Then I started looking at when the files started growing in size:
> find . -maxdepth 1 -mindepth 1 -type d -mtime -200 -exec du -sh {} \; | grep -E '^[0-9]+(\.[0-9]+)?G' | wc -l
210
...
> find . -maxdepth 1 -mindepth 1 -type d -mtime -77 -exec du -sh {} \; | grep -E '^[0-9]+(\.[0-9]+)?G' | wc -l
210
> find . -maxdepth 1 -mindepth 1 -type d -mtime -76 -exec du -sh {} \; | grep -E '^[0-9]+(\.[0-9]+)?G' | wc -l
206
  1. Found that happened 77 days ago, which means:
> date -d "77 days ago"
Thu Oct  2 01:48:25 PM UTC 2025
  1. What’s interesting here is that every large file under the store --data-dir is related to the Kubernetes Prometheus instance:
> find . -maxdepth 1 -mindepth 1 -type d -mtime -77 -exec du -sh {} \; | grep -E '^[0-9]+(\.[0-9]+)?G' | sed 's/\.\///g' | awk '{print $2}' | while read ulid; do cat /tmp/bucket_short | jq --arg id "$ulid" '. | select (.ulid == $id)'; done | jq '.labels.prometheus' | sort | uniq -c
    210 "k8s"
  1. Finally, I noticed (just now, sigh) that on 2nd October, an MW deploy happened.

image.png (1,006×535 px, 24 KB)

tappof added a comment.Dec 18 2025, 2:24 PM

image.png (2,581×1,423 px, 178 KB)

tappof added a comment.Dec 18 2025, 2:47 PM

image.png (2,987×1,156 px, 170 KB)

tappof added a comment.EditedDec 19 2025, 11:02 AM

I analyzed the block metrics cardinality using promtool tsdb analyze (Promtool 3.8.1) and found that a wild metric appeared after the MediaWiki deployment at the end of September / start of October: mediawiki_action_api_modules_latency_bucket.

|            ULID            |         FROM         |        UNTIL         |     RANGE      |   UNTIL-DOWN    |   #SERIES   |    #SAMPLES     |    #CHUNKS    | COMP-LEVEL | COMP-FAILED |               LABELS                | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|----------------|-----------------|-------------|-----------------|---------------|------------|-------------|-------------------------------------|------------|-----------|

| 01K5G0ZDDDFGHW5TVKX768M6E9 | 2025-09-17T00:00:00Z | 2025-09-19T00:00:00Z | 48h0m0s        | -8h0m0s         | 82,315,298  | 30,254,076,898  | 325,940,610   | 3          | false       | prometheus=k8s,replica=d,site=eqiad | 0s         | compactor |
| 01K5G6GNT3GQZ4XDXX65E6Y1DE | 2025-09-17T00:00:00Z | 2025-09-19T00:00:00Z | 48h0m0s        | -8h0m0s         | 82,178,195  | 30,251,531,409  | 325,808,182   | 3          | false       | prometheus=k8s,replica=c,site=eqiad | 0s         | compactor |
| 01K5ZNEX277AZEGKRRVMG7B807 | 2025-09-23T00:00:00Z | 2025-09-25T00:00:00Z | 48h0m0s        | -8h0m0s         | 70,563,702  | 25,796,075,525  | 277,916,546   | 3          | false       | prometheus=k8s,replica=d,site=codfw | 0s         | compactor |
| 01K7F4JXDH48EK51FQP0XRV049 | 2025-09-25T00:00:00Z | 2025-10-03T00:00:00Z | 191h59m59.999s | 48h0m0.001s     | 101,697,918 | 17,802,669,172  | 234,126,757   | 4          | false       | prometheus=k8s,replica=d,site=eqiad | 5m0s       | compactor |
| 01K7F5CBWE3ZHQCTTF73P3HWX6 | 2025-09-25T00:00:00Z | 2025-10-03T00:00:00Z | 191h59m59.998s | 48h0m0.002s     | 101,696,777 | 17,803,864,711  | 234,130,045   | 4          | false       | prometheus=k8s,replica=c,site=eqiad | 5m0s       | compactor |
| 01K6MCMQY0KJHCQQS6642AFAJP | 2025-10-01T00:00:00Z | 2025-10-03T00:00:00Z | 48h0m0s        | -8h0m0s         | 169,071,310 | 35,848,682,712  | 461,669,564   | 3          | false       | prometheus=k8s,replica=c,site=codfw | 0s         | compactor |
| 01K6MV8BJ97HSTK0KXS3V5KM0G | 2025-10-01T00:00:00Z | 2025-10-03T00:00:00Z | 48h0m0s        | 192h0m0s        | 168,928,710 | 7,258,037,239   | 200,667,949   | 3          | false       | prometheus=k8s,replica=c,site=codfw | 5m0s       | compactor |
MM|            ULID            |         FROM         |        UNTIL         |     RANGE      |   UNTIL-DOWN    |   #SERIES   |    #SAMPLES     |    #CHUNKS    | COMP-LEVEL | COMP-FAILED |               LABELS                | RESOLUTION |  SOURCE   |
  |----------------------------|----------------------|----------------------|----------------|-----------------|-------------|-----------------|---------------|------------|-------------|-------------------------------------|------------|-----------|

07| 01JZPN7GDQ69NRZD0VDPB9X52Z | 2025-07-07T00:00:00Z | 2025-07-09T00:00:00Z | 48h0m0s        | -8h0m0s         | 62,829,583  | 23,643,037,478  | 251,700,481   | 3          | false       | prometheus=k8s,replica=d,site=codfw | 0s         | compactor |
09| 01K5ZNEX277AZEGKRRVMG7B807 | 2025-09-23T00:00:00Z | 2025-09-25T00:00:00Z | 48h0m0s        | -8h0m0s         | 70,563,702  | 25,796,075,525  | 277,916,546   | 3          | false       | prometheus=k8s,replica=d,site=codfw | 0s         | compactor |
10| 01K7K5BNZ408X60YW6SXBVKSZZ | 2025-10-13T00:00:00Z | 2025-10-15T00:00:00Z | 48h0m0s        | -8h0m0s         | 121,096,405 | 37,736,817,388  | 426,689,887   | 3          | false       | prometheus=k8s,replica=d,site=codfw | 0s         | compactor |
root@titan2001:/srv/ts/prometheus-3.8.1.linux-amd64# ./promtool tsdb analyze ../$MM | tee $MM.out
root@titan2001:/srv/ts/prometheus-3.8.1.linux-amd64# cat 07.out | grep "Highest cardinality metric names:" -A 3
Highest cardinality metric names:
4658186 envoy_http_downstream_rq_time_bucket
4308700 envoy_cluster_upstream_rq_time_bucket
4241560 envoy_cluster_upstream_cx_connect_ms_bucket
root@titan2001:/srv/ts/prometheus-3.8.1.linux-amd64# cat 09.out | grep "Highest cardinality metric names:" -A 3
Highest cardinality metric names:
5460843 envoy_http_downstream_rq_time_bucket
5126873 envoy_cluster_upstream_rq_time_bucket
4975930 envoy_cluster_upstream_cx_connect_ms_bucket
root@titan2001:/srv/ts/prometheus-3.8.1.linux-amd64# cat 10.out | grep "Highest cardinality metric names:" -A 3
Highest cardinality metric names:
39904578 mediawiki_action_api_modules_latency_bucket
5383313 envoy_http_downstream_rq_time_bucket
5069219 envoy_cluster_upstream_rq_time_bucket

That metric does not seem to be currently used in Puppet, alerts, or Grafana.
I would like to add a relabel rule to drop its ingestion.

gerritbot added a comment.Dec 19 2025, 2:37 PM

Change #1219878 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] prometheus/k8s-pod: drop mediawiki_action_api_modules_lantecy metric

https://gerrit.wikimedia.org/r/1219878

gerritbot added a project: Patch-For-Review.Dec 19 2025, 2:37 PM
gerritbot added a comment.Dec 19 2025, 2:55 PM

Change #1219878 merged by Tiziano Fogli:

[operations/puppet@production] prometheus/k8s-pod: drop mediawiki_action_api_modules_lantecy metric

https://gerrit.wikimedia.org/r/1219878

tappof added a parent task: T402074: Set up new prometheus label structure for the API monitoring.Dec 19 2025, 3:04 PM
tappof mentioned this in T402074: Set up new prometheus label structure for the API monitoring.Dec 19 2025, 3:07 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 19 2025, 3:30 PM
gerritbot added a comment.Dec 19 2025, 5:20 PM

Change #1219911 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] prometheus/k8s-pods-metrics: drop mediawiki_action_api_modules_latency metric

https://gerrit.wikimedia.org/r/1219911

gerritbot added a project: Patch-For-Review.Dec 19 2025, 5:20 PM
gerritbot added a comment.Dec 19 2025, 5:24 PM

Change #1219911 merged by Cwhite:

[operations/puppet@production] prometheus/k8s-pods-metrics: drop mediawiki_action_api_modules_latency metric

https://gerrit.wikimedia.org/r/1219911

Maintenance_bot removed a project: Patch-For-Review.Dec 19 2025, 5:30 PM
gerritbot added a comment.Dec 19 2025, 6:11 PM

Change #1219923 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] prometheus/k8s-pods-metrics: move drop job to metric_relabel_configs

https://gerrit.wikimedia.org/r/1219923

gerritbot added a project: Patch-For-Review.Dec 19 2025, 6:11 PM
gerritbot added a comment.Dec 19 2025, 6:15 PM

Change #1219923 merged by Cwhite:

[operations/puppet@production] prometheus/k8s-pods-metrics: move drop job to metric_relabel_configs

https://gerrit.wikimedia.org/r/1219923

Maintenance_bot removed a project: Patch-For-Review.Dec 19 2025, 6:31 PM
tappof added a comment.Dec 22 2025, 9:57 AM

I tried to delete unwanted series from a locally stored block (downloaded from swift) using:

#!/bin/bash
# --no-delete-blocks should be used only for testing.
# --rewrite.add-change-log --no-delete-blocks should be used only for testing, otherwise --no-rewrite.add-change-log
systemd-run --scope -p MemoryLimit=80G /usr/bin/thanos tools bucket rewrite --objstore.config-file=/srv/rewrite/objstore.yaml --id="$1" --tmp.dir=/srv/rewrite/temp --no-dry-run --no-delete-blocks --rewrite.add-change-log --rewrite.to-delete-config-file=/srv/rewrite/matchers.yaml --enable-auto-gomemlimit --auto-gomemlimit.ratio=0.9

with:

root@titan2001:/srv/rewrite# cat objstore.yaml
type: FILESYSTEM
config:
  directory: "/srv/rewrite/tsdb"
prefix: "thanos"

root@titan2001:/srv/rewrite# cat matchers.yaml
- matchers: "{__name__=~\"mediawiki_action_api_modules_.*\"}"

This is the filesystem tree used for the test:

root@titan2001:/srv/rewrite# tree -L 3
.
├── matchers.yaml
├── objstore.yaml
├── prometheus
│   ├── 01KAFWCAJQEF1D0JFVXF00GX5H.analyze
│   ├── LICENSE
│   ├── NOTICE
│   ├── prometheus
│   ├── prometheus.yml
│   └── promtool
├── rewrite.sh
├── temp
│   ├── 01KAFWCAJQEF1D0JFVXF00GX5H
│   │   ├── chunks
│   │   ├── index
│   │   └── meta.json
│   ├── 01KD2KC8GCWT6KJTE04YN7GTYV
│   │   └── change.log
│   └── 01KD2KC8GCWT6KJTE04YN7GTYV.tmp-for-creation
│       ├── chunks
│       ├── index
│       ├── index_tmp_p
│       └── index_tmp_po
└── tsdb
    └── thanos
        ├── 01KAFWCAJQEF1D0JFVXF00GX5H
        └── tmp_dbro_sandbox2368245297

promtool tsdb analyze on block 01KAFWCAJQEF1D0JFVXF00GX5H:

root@titan2001:/srv/rewrite# cat prometheus/01KAFWCAJQEF1D0JFVXF00GX5H.analyze | grep "Highest cardinality metric names:" -A 7
Highest cardinality metric names:
70273238 mediawiki_action_api_modules_latency_bucket
5019517 mediawiki_action_api_modules_latency_sum
5019517 mediawiki_action_api_modules_latency_count
5019507 mediawiki_action_api_modules_hit_total
4967825 envoy_http_downstream_rq_time_bucket
4907925 envoy_cluster_upstream_rq_time_bucket
4484420 envoy_cluster_upstream_cx_connect_ms_bucket

The rewrite process output:

root@titan2001:/srv/rewrite# ./rewrite.sh 01KAFWCAJQEF1D0JFVXF00GX5H
Running scope as unit: run-r560072eb6a134fa3a0fc12397e46551d.scope
ts=2025-12-22T08:36:01.115229851Z caller=factory.go:54 level=info msg="loading bucket configuration"
ts=2025-12-22T08:36:01.115766916Z caller=tools_bucket.go:1228 level=info msg="downloading block" source=01KAFWCAJQEF1D0JFVXF00GX5H
ts=2025-12-22T08:38:54.224065234Z caller=tools_bucket.go:1265 level=info msg="changelog will be available" file=/srv/rewrite/temp/01KD2KC8GCWT6KJTE04YN7GTYV/change.log
ts=2025-12-22T08:38:54.22993252Z caller=tools_bucket.go:1280 level=info msg="starting rewrite for block" source=01KAFWCAJQEF1D0JFVXF00GX5H new=01KD2KC8GCWT6KJTE04YN7GTYV toDelete="- matchers: \"{__name__=~\\\"mediawiki_action_api_modules_.*\\\"}\"\n" toRelabel=
ts=2025-12-22T08:41:57.304240878Z caller=compactor.go:42 level=info msg="processed 10.00% of 155226394 series"
ts=2025-12-22T08:43:51.362537119Z caller=compactor.go:42 level=info msg="processed 20.00% of 155226394 series"
ts=2025-12-22T08:45:43.835552674Z caller=compactor.go:42 level=info msg="processed 30.00% of 155226394 series"
ts=2025-12-22T08:49:13.300071845Z caller=compactor.go:42 level=info msg="processed 40.00% of 155226394 series"
ts=2025-12-22T08:52:42.999820172Z caller=compactor.go:42 level=info msg="processed 50.00% of 155226394 series"
ts=2025-12-22T08:55:58.461301759Z caller=compactor.go:42 level=info msg="processed 60.00% of 155226394 series"
ts=2025-12-22T08:59:16.896200801Z caller=compactor.go:42 level=info msg="processed 70.00% of 155226394 series"
ts=2025-12-22T09:02:31.605272042Z caller=compactor.go:42 level=info msg="processed 80.00% of 155226394 series"
ts=2025-12-22T09:05:46.361789742Z caller=compactor.go:42 level=info msg="processed 90.00% of 155226394 series"
ts=2025-12-22T09:08:57.895961885Z caller=compactor.go:42 level=info msg="processed 100.00% of 155226394 series"
ts=2025-12-22T09:08:57.896063033Z caller=tools_bucket.go:1290 level=info msg="wrote new block after modifications; flushing" source=01KAFWCAJQEF1D0JFVXF00GX5H new=01KD2KC8GCWT6KJTE04YN7GTYV
ts=2025-12-22T09:16:56.098924025Z caller=tools_bucket.go:1299 level=info msg="uploading new block" source=01KAFWCAJQEF1D0JFVXF00GX5H new=01KD2KC8GCWT6KJTE04YN7GTYV
ts=2025-12-22T09:19:13.447884021Z caller=tools_bucket.go:1309 level=info msg=uploaded source=01KAFWCAJQEF1D0JFVXF00GX5H new=01KD2KC8GCWT6KJTE04YN7GTYV
ts=2025-12-22T09:19:13.44792303Z caller=tools_bucket.go:1317 level=info msg="rewrite done" IDs=01KAFWCAJQEF1D0JFVXF00GX5H
ts=2025-12-22T09:19:13.44798251Z caller=main.go:174 level=info msg=exiting

Checked the change.log to verify that only mediawiki_action_api_modules_.* series were deleted

root@titan2001:/srv/rewrite/temp/01KD2KC8GCWT6KJTE04YN7GTYV# cat change.log | head -n 3
Deleted {__name__="mediawiki_action_api_modules_hit_total", api_endpoint="abusefiltercheckmatch", api_module="MediaWiki_Api", api_type="ACTION_API", app="statsd-exporter", instance="10.194.130.37:9102", job="k8s-pods", kubernetes_namespace="mw-api-ext", kubernetes_pod_name="statsd-exporter-prometheus-7c8d4f77c9-jwwvn", method="GET", path="action_abusefiltercheckmatch_format_json_maxlag_smaxage_0_maxage_0_assert_assertuser_requestid_servedby_curtimestamp_responselanginfo_origin_crossorigin_uselang_user_variant_errorformat_bc_errorlang_uselang_errorsuselocal_centralauthtoken", pod_template_hash="7c8d4f77c9", release="prometheus", routed_via="prometheus", status="permissiondenied"} [{1763581696291 1763583196291}]
Deleted {__name__="mediawiki_action_api_modules_hit_total", api_endpoint="abusefiltercheckmatch", api_module="MediaWiki_Api", api_type="ACTION_API", app="statsd-exporter", instance="10.194.130.37:9102", job="k8s-pods", kubernetes_namespace="mw-api-ext", kubernetes_pod_name="statsd-exporter-prometheus-7c8d4f77c9-jwwvn", method="GET", path="action_abusefiltercheckmatch_format_jsonfm_maxlag_smaxage_0_maxage_0_assert_assertuser_requestid_servedby_curtimestamp_responselanginfo_origin_crossorigin_uselang_user_variant_errorformat_bc_errorlang_uselang_errorsuselocal_centralauthtoken", pod_template_hash="7c8d4f77c9", release="prometheus", routed_via="prometheus", status="permissiondenied"} [{1763431516291 1763592316291}]
Deleted {__name__="mediawiki_action_api_modules_hit_total", api_endpoint="abusefiltercheckmatch", api_module="MediaWiki_Api", api_type="ACTION_API", app="statsd-exporter", instance="10.194.130.37:9102", job="k8s-pods", kubernetes_namespace="mw-api-ext", kubernetes_pod_name="statsd-exporter-prometheus-7c8d4f77c9-jwwvn", method="POST", path="action_abusefiltercheckmatch_format_json_maxlag_smaxage_0_maxage_0_assert_assertuser_requestid_servedby_curtimestamp_responselanginfo_origin_crossorigin_uselang_user_variant_errorformat_bc_errorlang_uselang_errorsuselocal_centralauthtoken", pod_template_hash="7c8d4f77c9", release="prometheus", routed_via="prometheus", status="200"} [{1763433016291 1763592316291}]


root@titan2001:/srv/rewrite/temp/01KD2KC8GCWT6KJTE04YN7GTYV# cat change.log | wc -l
85331779
root@titan2001:/srv/rewrite/temp/01KD2KC8GCWT6KJTE04YN7GTYV# cat change.log | grep "mediawiki_action_api_modules" | wc -l
85331779

promtool tsdb analyze on block 01KD2KC8GCWT6KJTE04YN7GTYV:

root@titan2001:/srv/rewrite/prometheus# cat 01KD2KC8GCWT6KJTE04YN7GTYV.analyze | grep "Highest cardinality metric names:" -A 7
Highest cardinality metric names:
4967825 envoy_http_downstream_rq_time_bucket
4907925 envoy_cluster_upstream_rq_time_bucket
4484420 envoy_cluster_upstream_cx_connect_ms_bucket
2970575 envoy_http_downstream_cx_length_ms_bucket
2573435 envoy_cluster_upstream_cx_length_ms_bucket
1748831 cache_operations_total
1153080 container_tasks_state

Size after rewriting:

root@titan2001:/srv/rewrite/tsdb/thanos# du -sh *
81G	01KAFWCAJQEF1D0JFVXF00GX5H
55G	01KD2KC8GCWT6KJTE04YN7GTYV

meta.json diff:

root@titan2001:/srv/rewrite/tsdb/thanos# diff -u 01KAFWCAJQEF1D0JFVXF00GX5H/meta.json 01KD2KC8GCWT6KJTE04YN7GTYV/meta.json
--- 01KAFWCAJQEF1D0JFVXF00GX5H/meta.json	2025-12-22 07:51:07.610207787 +0000
+++ 01KD2KC8GCWT6KJTE04YN7GTYV/meta.json	2025-12-22 09:19:13.444947332 +0000
@@ -1,39 +1,16 @@
 {
-	"ulid": "01KAFWCAJQEF1D0JFVXF00GX5H",
+	"ulid": "01KD2KC8GCWT6KJTE04YN7GTYV",
 	"minTime": 1763424000000,
 	"maxTime": 1763596800000,
 	"stats": {
-		"numSamples": 41270721825,
-		"numSeries": 155226394,
-		"numChunks": 488080925
+		"numSamples": 32422921933,
+		"numSeries": 69894615,
+		"numChunks": 331465763
 	},
 	"compaction": {
 		"level": 3,
 		"sources": [
-			"01KAAEFF7M23AA7HT5QQKYB77A",
-			"01KAANB73APDV1ERY86JXER3JG",
-			"01KAAW6XQC9E12NTJ0P5AKPGG6",
-			"01KAB32RDBD19DW9BJG2SBY81R",
-			"01KAB9YC880M3M56MP62DJ00X2",
-			"01KABGT3G2KM07FDJGZ0Z7NZHG",
-			"01KABQNTRBJ0AFXC88CP5Y8YP0",
-			"01KABYHHWXWFSEJD4315BNX8EZ",
-			"01KAC5D97EW633NFSKCS28751F",
-			"01KACC90FQ679TKHX3SXPJWHM9",
-			"01KACK4QR01Z7Y6XER3F1YS60R",
-			"01KACT0EZZRVT6YHDA3J52X828",
-			"01KAD0W684WGRRE53AQR7FZ4Z4",
-			"01KAD7QXFY7XMKQED7H3SZEZDX",
-			"01KADEKMR16F294RY0HDD40A5W",
-			"01KADNFBZTARZ1BDP5KDT4SWB1",
-			"01KADWB3JTAXD3E2CPV7B4WGFB",
-			"01KAE36TFYYBC059S4KEXRRP2C",
-			"01KAEA2HQGC0WH6WVZ1DS2F181",
-			"01KAEGY8ZY56FHB32PD1MFYDVQ",
-			"01KAEQT07XAK7EPN6ZHGJ3QRMJ",
-			"01KAEYNQX0Y7X34XX1A9CJ7RVV",
-			"01KAF5HEQX56EQRPC0Q0RB2MJZ",
-			"01KAFCD5ZXFM0P4RJPG96FV6EF"
+			"01KD2KC8GCWT6KJTE04YN7GTYV"
 		],
 		"parents": [
 			{
@@ -78,7 +55,7 @@
 		"downsample": {
 			"resolution": 0
 		},
-		"source": "compactor",
+		"source": "bucket.rewrite",
 		"segment_files": [
 			"000001",
 			"000002",
@@ -431,120 +408,105 @@
 			},
 			{
 				"rel_path": "chunks/000065",
-				"size_bytes": 536870862
+				"size_bytes": 536870701
 			},
 			{
 				"rel_path": "chunks/000066",
-				"size_bytes": 536870786
+				"size_bytes": 536870766
 			},
 			{
 				"rel_path": "chunks/000067",
-				"size_bytes": 536870884
+				"size_bytes": 536870807
 			},
 			{
 				"rel_path": "chunks/000068",
-				"size_bytes": 536870869
+				"size_bytes": 536870644
 			},
 			{
 				"rel_path": "chunks/000069",
-				"size_bytes": 536870868
+				"size_bytes": 536870829
 			},
 			{
 				"rel_path": "chunks/000070",
-				"size_bytes": 536870873
+				"size_bytes": 536870845
 			},
 			{
 				"rel_path": "chunks/000071",
-				"size_bytes": 536870891
+				"size_bytes": 536870822
 			},
 			{
 				"rel_path": "chunks/000072",
-				"size_bytes": 536870866
+				"size_bytes": 536870785
 			},
 			{
 				"rel_path": "chunks/000073",
-				"size_bytes": 536870905
+				"size_bytes": 536870879
 			},
 			{
 				"rel_path": "chunks/000074",
-				"size_bytes": 536870901
+				"size_bytes": 536870863
 			},
 			{
 				"rel_path": "chunks/000075",
-				"size_bytes": 536870856
+				"size_bytes": 536870273
 			},
 			{
 				"rel_path": "chunks/000076",
-				"size_bytes": 536870854
+				"size_bytes": 536870824
 			},
 			{
 				"rel_path": "chunks/000077",
-				"size_bytes": 536870857
-			},
-			{
-				"rel_path": "chunks/000078",
-				"size_bytes": 536870902
-			},
-			{
-				"rel_path": "chunks/000079",
-				"size_bytes": 536870877
-			},
-			{
-				"rel_path": "chunks/000080",
-				"size_bytes": 536870834
-			},
-			{
-				"rel_path": "chunks/000081",
-				"size_bytes": 536870726
-			},
-			{
-				"rel_path": "chunks/000082",
-				"size_bytes": 536870864
-			},
-			{
-				"rel_path": "chunks/000083",
-				"size_bytes": 536870735
-			},
-			{
-				"rel_path": "chunks/000084",
-				"size_bytes": 536870906
-			},
-			{
-				"rel_path": "chunks/000085",
-				"size_bytes": 536870811
-			},
-			{
-				"rel_path": "chunks/000086",
-				"size_bytes": 536870871
-			},
-			{
-				"rel_path": "chunks/000087",
-				"size_bytes": 536870861
-			},
-			{
-				"rel_path": "chunks/000088",
-				"size_bytes": 536870882
-			},
-			{
-				"rel_path": "chunks/000089",
-				"size_bytes": 536870778
-			},
-			{
-				"rel_path": "chunks/000090",
-				"size_bytes": 536870778
-			},
-			{
-				"rel_path": "chunks/000091",
-				"size_bytes": 479012642
+				"size_bytes": 34975978
 			},
 			{
 				"rel_path": "index",
-				"size_bytes": 37408905390
+				"size_bytes": 17145271458
 			},
 			{
 				"rel_path": "meta.json"
 			}
 		],
+		"rewrites": [
+			{
+				"sources": [
+					"01KAAEFF7M23AA7HT5QQKYB77A",
+					"01KAANB73APDV1ERY86JXER3JG",
+					"01KAAW6XQC9E12NTJ0P5AKPGG6",
+					"01KAB32RDBD19DW9BJG2SBY81R",
+					"01KAB9YC880M3M56MP62DJ00X2",
+					"01KABGT3G2KM07FDJGZ0Z7NZHG",
+					"01KABQNTRBJ0AFXC88CP5Y8YP0",
+					"01KABYHHWXWFSEJD4315BNX8EZ",
+					"01KAC5D97EW633NFSKCS28751F",
+					"01KACC90FQ679TKHX3SXPJWHM9",
+					"01KACK4QR01Z7Y6XER3F1YS60R",
+					"01KACT0EZZRVT6YHDA3J52X828",
+					"01KAD0W684WGRRE53AQR7FZ4Z4",
+					"01KAD7QXFY7XMKQED7H3SZEZDX",
+					"01KADEKMR16F294RY0HDD40A5W",
+					"01KADNFBZTARZ1BDP5KDT4SWB1",
+					"01KADWB3JTAXD3E2CPV7B4WGFB",
+					"01KAE36TFYYBC059S4KEXRRP2C",
+					"01KAEA2HQGC0WH6WVZ1DS2F181",
+					"01KAEGY8ZY56FHB32PD1MFYDVQ",
+					"01KAEQT07XAK7EPN6ZHGJ3QRMJ",
+					"01KAEYNQX0Y7X34XX1A9CJ7RVV",
+					"01KAF5HEQX56EQRPC0Q0RB2MJZ",
+					"01KAFCD5ZXFM0P4RJPG96FV6EF"
+				],
+				"deletions_applied": [
+					{
+						"matchers": [
+							{
+								"Type": 2,
+								"Name": "__name__",
+								"Value": "mediawiki_action_api_modules_.*"
+							}
+						]
+					}
+				]
+			}
+		],
 		"index_stats": {
 			"series_max_size": 848,
 			"chunk_max_size": 1024

No differences were found in the labels section (used by the Store Gateway component):

...
"thanos": {
        "labels": {
                "prometheus": "k8s",
                "replica": "d",
                "site": "codfw"
        },
        ...

Original block index headers size:

root@titan2001:/srv/thanos-store# du -sh 01KAFWCAJQEF1D0JFVXF00GX5H
2.6G	01KAFWCAJQEF1D0JFVXF00GX5H

I'm expecting the index headers for 01KD2KC8GCWT6KJTE04YN7GTYV to be smaller as well.

Stashbot added a comment.Dec 22 2025, 4:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-12-22T16:49:45Z] <tappof> lvextend /dev/vg0/srv on titan1001, titan1002, titan2002. T410152

tappof added a comment.Dec 22 2025, 4:50 PM

Given the current situation, we can wait until the break is over to start the cleanup procedure.

image.png (2,954×1,302 px, 155 KB)

Meanwhile, we added the last disk to the VG on titan1001, titan1002, and titan2002 to stay safe.

We're going to decrease the compactor's concurrency to allow it to start compacting again (this is to avoid backlog explosion).

After the break, we will depool titan2001, stop thanos-store (and thanos-compact), and free up space on /srv/thanos-store to provide enough disk space for the cleanup procedure.

gerritbot added a comment.Dec 22 2025, 4:53 PM

Change #1220385 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] thanos-compact: reduce concurrency to 2

https://gerrit.wikimedia.org/r/1220385

gerritbot added a project: Patch-For-Review.Dec 22 2025, 4:53 PM
gerritbot added a comment.Dec 22 2025, 4:55 PM

Change #1220385 merged by Tiziano Fogli:

[operations/puppet@production] thanos-compact: reduce concurrency to 2

https://gerrit.wikimedia.org/r/1220385

Maintenance_bot removed a project: Patch-For-Review.Dec 22 2025, 5:30 PM
hnowlan moved this task from FY2025/2026-Q2 to FY2025/2026-Q3 on the SRE Observability board.Jan 13 2026, 12:16 PM
hnowlan edited projects, added: SRE Observability (FY2025/2026-Q3); removed: SRE Observability (FY2025/2026-Q2).
tappof added a comment.EditedJan 15 2026, 8:59 AM

This is the list of blocks that need to be rewritten:

# ulid start startts end endts length(h)
# 01K6MBGV6NXVBC7SBT4A251ADD 2025-10-01T00:00:00Z 1759276800 2025-10-03T00:00:00Z 1759449600 48
# 01K6MCMQY0KJHCQQS6642AFAJP 2025-10-01T00:00:00Z 1759276800 2025-10-03T00:00:00Z 1759449600 48
# 01K6MTRR88PGJTJMVBYQ6RKG4N 2025-10-01T00:00:00Z 1759276800 2025-10-03T00:00:00Z 1759449600 48
# 01K6MV8BJ97HSTK0KXS3V5KM0G 2025-10-01T00:00:00Z 1759276800 2025-10-03T00:00:00Z 1759449600 48
# 01K6SCJZNWXJQS8V3R5Y4A8PN9 2025-10-03T00:00:00Z 1759449600 2025-10-05T00:00:00Z 1759622400 48
# 01K6SCK7AQV1M7PNDXYVAAQJ4V 2025-10-03T00:00:00Z 1759449600 2025-10-05T00:00:00Z 1759622400 48
# 01K6ZQ3J7SS9THAV7MXXKK4BZJ 2025-10-03T00:00:00Z 1759449600 2025-10-05T00:00:00Z 1759622400 48
# 01K6ZQ48YTWZ4179E5HMFZW6B6 2025-10-03T00:00:00Z 1759449600 2025-10-05T00:00:00Z 1759622400 48
# 01K6YHEVRHVPAZ0VJ4C7MEJEMF 2025-10-05T00:00:00Z 1759622400 2025-10-07T00:00:00Z 1759795200 48
# 01K6YKK23Q4GXE2JB66WB9BHNM 2025-10-05T00:00:00Z 1759622400 2025-10-07T00:00:00Z 1759795200 48
# 01K704N57245YPJVYQG58C6VKQ 2025-10-05T00:00:00Z 1759622400 2025-10-07T00:00:00Z 1759795200 48
# 01K709YT3YFGK8SWTS88VPWEQR 2025-10-05T00:00:00Z 1759622400 2025-10-07T00:00:00Z 1759795200 48
# 01K76F9J0EZ6KTG39BZH21243E 2025-10-03T00:00:00Z 1759449600 2025-10-07T00:00:00Z 1759795200 96
# 01K7753BE3RZKCBQ0DZYTWGR28 2025-10-03T00:00:00Z 1759449600 2025-10-07T00:00:00Z 1759795200 96
# 01K7C3HK0RM901JYJRNRH4VW94 2025-10-03T00:00:00Z 1759449600 2025-10-07T00:00:00Z 1759795200 96
# 01K7FFWA1PBYJBEYHMERVBW9RT 2025-10-03T00:00:00Z 1759449600 2025-10-07T00:00:00Z 1759795200 96
# 01K73QAAX5PRSHHQQS75K11D3A 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K73RFCF4JKX2KEBKX0J4KQKW 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K73SMHP2YGW2W5017RPW9T7E 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K73ZDZDVVBN7R3A6ZVFJDJN4 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K77KMHYKHT78NMQ9K2Z9PXAJ 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K77MGF06S6Y8X2EVDVRXW9Z3 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K77MGTK3SDJ08YHVKJKMGPFP 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K77Z18DCCAZ5DN2J7JC2BRSE 2025-10-07T00:00:00Z 1759795200 2025-10-09T00:00:00Z 1759968000 48
# 01K78WMHZDPZDP1FN81DWJJV5B 2025-10-09T00:00:00Z 1759968000 2025-10-11T00:00:00Z 1760140800 48
# 01K78YYZ13VR4FYDNGM8XZQBAQ 2025-10-09T00:00:00Z 1759968000 2025-10-11T00:00:00Z 1760140800 48
# 01K7AGT2K4F7Y4WCC11WS5DWSQ 2025-10-09T00:00:00Z 1759968000 2025-10-11T00:00:00Z 1760140800 48
# 01K7AMB8ATGA1VWRQDGKFSWJ8H 2025-10-09T00:00:00Z 1759968000 2025-10-11T00:00:00Z 1760140800 48
# 01K7DYWB2WR7JBSBN7KKR3X8CC 2025-10-11T00:00:00Z 1760140800 2025-10-13T00:00:00Z 1760313600 48
# 01K7E65T85WTQ197EZFCHEK1RM 2025-10-11T00:00:00Z 1760140800 2025-10-13T00:00:00Z 1760313600 48
# 01K7EHKBMTGYGZFGM73A4ZSK6S 2025-10-11T00:00:00Z 1760140800 2025-10-13T00:00:00Z 1760313600 48
# 01K7EQ3BTR5065EG4ARBK1GKMC 2025-10-11T00:00:00Z 1760140800 2025-10-13T00:00:00Z 1760313600 48
# 01K88E4AJ046C768ZKJN8H688Y 2025-10-09T00:00:00Z 1759968000 2025-10-13T00:00:00Z 1760313600 96
# 01K88KQHFE0YQKWJDWFR5DA71V 2025-10-09T00:00:00Z 1759968000 2025-10-13T00:00:00Z 1760313600 96
# 01K8E04J4QH02ZGBZJP7WQ97S4 2025-10-09T00:00:00Z 1759968000 2025-10-13T00:00:00Z 1760313600 96
# 01K8E0YDJ32EW9R7YY600P7WQZ 2025-10-09T00:00:00Z 1759968000 2025-10-13T00:00:00Z 1760313600 96
# 01K7K4G9A64QAHCWSHVFX00JTB 2025-10-13T00:00:00Z 1760313600 2025-10-14T08:00:00Z 1760428800 32
# 01K7K5W8FDKVA47FJ5FJ2YA8DJ 2025-10-13T00:00:00Z 1760313600 2025-10-14T08:00:00Z 1760428800 32
# 01K7K5BNZ408X60YW6SXBVKSZZ 2025-10-13T00:00:00Z 1760313600 2025-10-15T00:00:00Z 1760486400 48
# 01K7KCM1CR107GDZ7KZKMX01DE 2025-10-13T00:00:00Z 1760313600 2025-10-15T00:00:00Z 1760486400 48
# 01K7KPD3VRBNM3WG8Y1X4P1PJ5 2025-10-13T00:00:00Z 1760313600 2025-10-15T00:00:00Z 1760486400 48
# 01K7KQJQ4X6NVZ1A3T2XY1EMHA 2025-10-13T00:00:00Z 1760313600 2025-10-15T00:00:00Z 1760486400 48
# 01K8872021B76MG2XXS4GFQ58A 2025-10-14T16:00:00Z 1760457600 2025-10-16T08:00:00Z 1760601600 40
# 01K892TGQPFZZWH6Q0FG9SCXBW 2025-10-14T16:00:00Z 1760457600 2025-10-16T08:00:00Z 1760601600 40
# 01K8A23W20T8AXPXRX2PTAA6KK 2025-10-14T16:00:00Z 1760457600 2025-10-16T08:00:00Z 1760601600 40
# 01K8B33W5AQSDBCVN6EHVDXAAQ 2025-10-14T16:00:00Z 1760457600 2025-10-16T08:00:00Z 1760601600 40
# 01K7R9P4TNFAHZGQ59SBEHH8AA 2025-10-15T00:00:00Z 1760486400 2025-10-17T00:00:00Z 1760659200 48
# 01K7RG6Q79MF9Q1AD5TG59AYFP 2025-10-15T00:00:00Z 1760486400 2025-10-17T00:00:00Z 1760659200 48
# 01K7XD9JBMAAWY172A607PZX0F 2025-10-17T00:00:00Z 1760659200 2025-10-19T00:00:00Z 1760832000 48
# 01K7XEPQYYCAXK7614MBZKW1TS 2025-10-17T00:00:00Z 1760659200 2025-10-19T00:00:00Z 1760832000 48
# 01K7XYJTNE1ZEJEVD0R1HQY84W 2025-10-17T00:00:00Z 1760659200 2025-10-19T00:00:00Z 1760832000 48
# 01K7XZJKNCVNJGF55VRQYNY66P 2025-10-17T00:00:00Z 1760659200 2025-10-19T00:00:00Z 1760832000 48
# 01K82HX4TF5CYNBJ2F5VRAQ0X6 2025-10-19T00:00:00Z 1760832000 2025-10-20T08:00:00Z 1760947200 32
# 01K82K3T9TXFMCZ7Z6M0NWPN46 2025-10-19T00:00:00Z 1760832000 2025-10-20T08:00:00Z 1760947200 32
# 01K88ZW6898QKM6B5N53VHVGBF 2025-10-17T00:00:00Z 1760659200 2025-10-21T00:00:00Z 1761004800 96
# 01K893VSKMXNTVY3CT56VV8ADJ 2025-10-17T00:00:00Z 1760659200 2025-10-21T00:00:00Z 1761004800 96
# 01K8E6Y8R2R6RSRK57XS5256K0 2025-10-15T00:00:00Z 1760486400 2025-10-21T00:00:00Z 1761004800 144
# 01K8EHQY4CAH37XEEBMG7ZKS5R 2025-10-15T00:00:00Z 1760486400 2025-10-21T00:00:00Z 1761004800 144
# 01K88XDMJ9S0T2DR5K00VG9CFE 2025-10-20T16:00:00Z 1760976000 2025-10-21T08:00:00Z 1761033600 16
# 01K89DCJT4VJ9AMZSWCDNESG83 2025-10-20T16:00:00Z 1760976000 2025-10-21T08:00:00Z 1761033600 16
# 01K87TEHS0XR73S04QJXRGRTCT 2025-10-21T16:00:00Z 1761062400 2025-10-23T00:00:00Z 1761177600 32
# 01K87VZ6WNY0XKKW4XQX44SWVT 2025-10-21T00:00:00Z 1761004800 2025-10-23T00:00:00Z 1761177600 48
# 01K880EE6KS8GNXQMJEH5D5Q8T 2025-10-21T16:00:00Z 1761062400 2025-10-23T00:00:00Z 1761177600 32
# 01K8810N8QMWRCGY0WHM2BTN0G 2025-10-21T00:00:00Z 1761004800 2025-10-23T00:00:00Z 1761177600 48
# 01K89M9V91VBZYN2NX98T2PNDM 2025-10-21T00:00:00Z 1761004800 2025-10-23T00:00:00Z 1761177600 48
# 01K89S2TETQZ37B8QYTNDC5ZZ4 2025-10-21T00:00:00Z 1761004800 2025-10-23T00:00:00Z 1761177600 48
# 01K88SNGT89W0DCYMF1CWPPBYM 2025-10-23T00:00:00Z 1761177600 2025-10-23T08:00:00Z 1761206400 8
# 01K88V10A2C1VZY2NEZZMH0VFJ 2025-10-23T00:00:00Z 1761177600 2025-10-23T08:00:00Z 1761206400 8
# 01K8CV71P5N8WTYY9RDBT28N0C 2025-10-23T16:00:00Z 1761235200 2025-10-25T00:00:00Z 1761350400 32
# 01K8CWJSXX253VAMHZJBTH2PP4 2025-10-23T16:00:00Z 1761235200 2025-10-25T00:00:00Z 1761350400 32
# 01K8HZFDY0MPGG34RJSEZGBB1Y 2025-10-25T00:00:00Z 1761350400 2025-10-27T00:00:00Z 1761523200 48
# 01K8J1BPA8YFDFV77ZN5MPREM8 2025-10-25T00:00:00Z 1761350400 2025-10-27T00:00:00Z 1761523200 48
# 01K8JKKX323RTQEZNKB1QF5SNH 2025-10-25T00:00:00Z 1761350400 2025-10-27T00:00:00Z 1761523200 48
# 01K8JPSNXDXS52C5BCY5FKWP9V 2025-10-25T00:00:00Z 1761350400 2025-10-27T00:00:00Z 1761523200 48
# 01K9CARDQA9ER04W66BQ844Y5B 2025-10-23T00:00:00Z 1761177600 2025-10-27T00:00:00Z 1761523200 96
# 01K9CG14YGKD5MJNR1BKSAH202 2025-10-23T00:00:00Z 1761177600 2025-10-27T00:00:00Z 1761523200 96
# 01K9J90PPJA8KXTD6GTRAZKHZF 2025-10-23T00:00:00Z 1761177600 2025-10-27T00:00:00Z 1761523200 96
# 01K9J9G0JRM88D1M5V60AS0GZX 2025-10-23T00:00:00Z 1761177600 2025-10-27T00:00:00Z 1761523200 96
# 01K8Q54H0QE79GH108Y2MZ2PPX 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8Q6C8Y5ZDP0GBDNW1EYC5XQ 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8Q6FZPYG9GET461QE5HMPTK 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8QEGJ64FJMF63E3ZFG5Y09E 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8QWFYHC6V4YA57MB2FJK56P 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8QWZV1HKE3ED6N228X7ZP6V 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8QY1TSBV63NVBKW8B48V419 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8R53J26TQ47S3CFJKMQ6800 2025-10-27T00:00:00Z 1761523200 2025-10-29T00:00:00Z 1761696000 48
# 01K8WBF8KPBPH5JKAPG7BNY714 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K8WCEKZW0G3D2KRJWVEETJ8F 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K8WCXZK1W0VNBHDZGZZHK76H 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K8WJVGYENZZT64P8W8HJEM7S 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K8X0MMJEKVDJDJDFECBQWXAQ 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K8X1RE8GR0ABPA8878KDVWZE 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K8X39XW8KXY8G3VJ5EZHPY6N 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K8XATZP888Z1SHZWWE1P6KPE 2025-10-29T00:00:00Z 1761696000 2025-10-31T00:00:00Z 1761868800 48
# 01K91F0YVQYCQ7RKM2FG1FWNQB 2025-10-31T00:00:00Z 1761868800 2025-11-02T00:00:00Z 1762041600 48
# 01K91GFN608CBW18PMVPHA75YE 2025-10-31T00:00:00Z 1761868800 2025-11-02T00:00:00Z 1762041600 48
# 01K91X5390HMNQB9ZNGCFHDG7X 2025-10-31T00:00:00Z 1761868800 2025-11-02T00:00:00Z 1762041600 48
# 01K91ZS0DVAJZ08B2ARKJVYP09 2025-10-31T00:00:00Z 1761868800 2025-11-02T00:00:00Z 1762041600 48
# 01K96N0G3NCS1ECC9ESJE2QY41 2025-11-02T00:00:00Z 1762041600 2025-11-04T00:00:00Z 1762214400 48
# 01K96PXKWKM2VQ5JWQNRN5R2SM 2025-11-02T00:00:00Z 1762041600 2025-11-04T00:00:00Z 1762214400 48
# 01K976V9SNBFFH1T7TJE7SN0BT 2025-11-02T00:00:00Z 1762041600 2025-11-04T00:00:00Z 1762214400 48
# 01K977KJC3E73QDN15BHM52FWF 2025-11-02T00:00:00Z 1762041600 2025-11-04T00:00:00Z 1762214400 48
# 01K9CVCBVZ5H8Q1DHVHE2PA206 2025-10-31T00:00:00Z 1761868800 2025-11-04T00:00:00Z 1762214400 96
# 01K9D5TVPSE8YZ4W1A5SXQ7TN8 2025-10-31T00:00:00Z 1761868800 2025-11-04T00:00:00Z 1762214400 96
# 01K9JFKZ5Y70SNWSHCVQXHZHHF 2025-10-31T00:00:00Z 1761868800 2025-11-04T00:00:00Z 1762214400 96
# 01K9JRAJWB94HR6CJGZQ35XSTV 2025-10-31T00:00:00Z 1761868800 2025-11-04T00:00:00Z 1762214400 96
# 01K9BSWQTGX0JVR9NX3K85YJFJ 2025-11-04T00:00:00Z 1762214400 2025-11-04T16:00:00Z 1762272000 16
# 01K9BW6BC55S93MYA1NAX47SGP 2025-11-04T00:00:00Z 1762214400 2025-11-04T16:00:00Z 1762272000 16
# 01K9BVGE9HQSTTF4XJJNMJ6XT4 2025-11-04T00:00:00Z 1762214400 2025-11-06T00:00:00Z 1762387200 48
# 01K9BZB91KV0ZAW0DK3AN0VHZM 2025-11-04T00:00:00Z 1762214400 2025-11-06T00:00:00Z 1762387200 48
# 01K9C4Q9Q4T7Q24JXA2XXW8W99 2025-11-05T00:00:00Z 1762300800 2025-11-06T00:00:00Z 1762387200 24
# 01K9CBYTZ0XHKAMQAGS932XNRT 2025-11-05T00:00:00Z 1762300800 2025-11-06T00:00:00Z 1762387200 24
# 01K9DHKE61Y5AE6V1Z1BCPNSTJ 2025-11-04T00:00:00Z 1762214400 2025-11-06T00:00:00Z 1762387200 48
# 01K9DPRXRK001FEEFGKXNPV3KA 2025-11-04T00:00:00Z 1762214400 2025-11-06T00:00:00Z 1762387200 48
# 01K9GX5YA3H5R02PHN7H42G4DF 2025-11-06T00:00:00Z 1762387200 2025-11-08T00:00:00Z 1762560000 48
# 01K9GYNDAJSXJQT4A79E8EBTP6 2025-11-06T00:00:00Z 1762387200 2025-11-08T00:00:00Z 1762560000 48
# 01K9HK4CHAT3GYR9PE89QHBC55 2025-11-06T00:00:00Z 1762387200 2025-11-08T00:00:00Z 1762560000 48
# 01K9HPEBRZ6DYC16R0B3J9YV25 2025-11-06T00:00:00Z 1762387200 2025-11-08T00:00:00Z 1762560000 48
# 01K9P22RWQQS1EG7ETH7TWFFZ0 2025-11-08T00:00:00Z 1762560000 2025-11-10T00:00:00Z 1762732800 48
# 01K9P3DCR5HB9VQDNCRQYYSG69 2025-11-08T00:00:00Z 1762560000 2025-11-10T00:00:00Z 1762732800 48
# 01K9PPNKW5PGEWJA5TDAMFSPQG 2025-11-08T00:00:00Z 1762560000 2025-11-10T00:00:00Z 1762732800 48
# 01K9PQQZ61NZZR5RFSRAM8A1H0 2025-11-08T00:00:00Z 1762560000 2025-11-10T00:00:00Z 1762732800 48
# 01KAJ5VYTRHA7K0HKE6CCSTJ69 2025-11-06T00:00:00Z 1762387200 2025-11-10T00:00:00Z 1762732800 96
# 01KAJYERQ5T7VS0XX7GEEAD94D 2025-11-06T00:00:00Z 1762387200 2025-11-10T00:00:00Z 1762732800 96
# 01KAQKRTHG7743PA4XAFZ8DYBK 2025-11-06T00:00:00Z 1762387200 2025-11-10T00:00:00Z 1762732800 96
# 01KAQKTADP36AQ6SAC348HANZ7 2025-11-06T00:00:00Z 1762387200 2025-11-10T00:00:00Z 1762732800 96
# 01K9V7XHKMH97VZ03NYVXYBTBH 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01K9V8SHDVR06KY9T7HHXM1D3V 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01K9V96XHZCRNW78VSSJQ160JA 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01K9VFXPZW36HRDE7V9CE9PNN6 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01K9VVDSYAPE2BKCMRCNQVRHV6 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01K9VVXTVR1PTF1DXTBQ6FZBTW 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01K9VWKJCDWYN69PF1G08PKYAD 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01K9W626EB339WQYPQWD88V8CM 2025-11-10T00:00:00Z 1762732800 2025-11-12T00:00:00Z 1762905600 48
# 01KA0CJBWXPTTMSE38NRAG6XPV 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA0DZW3P71N9VQPGEQ9006G9 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA0E2RW79MWD3R24YK308PKE 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA0Q0XDP7NMKTG9M7MMCAM7C 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA184HVK9A9QD56FSN6RGX0V 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA1A4XVD699TPV1MBD9CVD9N 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA1B0BTE8D08F0FPQMM62VWF 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA1N1A9GW3NQ47V2R281WQ7S 2025-11-12T00:00:00Z 1762905600 2025-11-14T00:00:00Z 1763078400 48
# 01KA5FRD99D63E1FPSG5NVEBJ7 2025-11-14T00:00:00Z 1763078400 2025-11-16T00:00:00Z 1763251200 48
# 01KA5GS3B9CHSTA25P0BVG3S0X 2025-11-14T00:00:00Z 1763078400 2025-11-16T00:00:00Z 1763251200 48
# 01KA5H0XG0RXTSESH0G6G4YS2M 2025-11-14T00:00:00Z 1763078400 2025-11-16T00:00:00Z 1763251200 48
# 01KA5PX8DE5WG3QQ0ESKMJYZW4 2025-11-14T00:00:00Z 1763078400 2025-11-16T00:00:00Z 1763251200 48
# 01KA62DH4XNC68FEDM747J5B1Y 2025-11-14T00:00:00Z 1763078400 2025-11-16T00:00:00Z 1763251200 48
# 01KA6AS6271C8KG2M1ZA738FR0 2025-11-14T00:00:00Z 1763078400 2025-11-16T00:00:00Z 1763251200 48
# 01KAANCF25PY091N308KFT77VP 2025-11-16T00:00:00Z 1763251200 2025-11-18T00:00:00Z 1763424000 48
# 01KAAPA0JFDC11N9NMWDK92N4X 2025-11-16T00:00:00Z 1763251200 2025-11-18T00:00:00Z 1763424000 48
# 01KAAPP1K3JNPQ4YTPRH00AX47 2025-11-16T00:00:00Z 1763251200 2025-11-18T00:00:00Z 1763424000 48
# 01KAAYCVVMW5P6HDH5D7JGS93V 2025-11-16T00:00:00Z 1763251200 2025-11-18T00:00:00Z 1763424000 48
# 01KABCZJ0MNN1MXRFZJ24ZNFZY 2025-11-16T00:00:00Z 1763251200 2025-11-18T00:00:00Z 1763424000 48
# 01KABE2MC9JN49NQG81XJ4YJXF 2025-11-16T00:00:00Z 1763251200 2025-11-18T00:00:00Z 1763424000 48
# 01KAQWEKESYTBPRPQDKD2RYQ2X 2025-11-14T00:00:00Z 1763078400 2025-11-18T00:00:00Z 1763424000 96
# 01KAQWFR86842R0KY8SN3XBS5E 2025-11-14T00:00:00Z 1763078400 2025-11-18T00:00:00Z 1763424000 96
# 01KAFV1ANGVQT3BK3HF96X49T1 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAFWCAJQEF1D0JFVXF00GX5H 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAFWYQ6T1XMQZPW82DPT5MYF 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAG2W5F9GRMQ37NG0TFNYDK8 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAKDEM6AP69B927D2A44QYJJ 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAKDHWR3R1XNHR05MV3NRF8C 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAKDZFDSFERZV2BY019BXWFZ 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAKQQFWE0YPN8537MNDD09ST 2025-11-18T00:00:00Z 1763424000 2025-11-20T00:00:00Z 1763596800 48
# 01KAPATMJ81WF4XR41KCHTW6J5 2025-11-20T00:00:00Z 1763596800 2025-11-20T16:00:00Z 1763654400 16
# 01KAPE5Q1KS7H6DG38C7Z36JX8 2025-11-20T00:00:00Z 1763596800 2025-11-20T16:00:00Z 1763654400 16
# 01KAPNK90QHV106YFRYEAZRET2 2025-11-21T00:00:00Z 1763683200 2025-11-22T00:00:00Z 1763769600 24
# 01KAPNKKSSWFB2QNZX0P5KMQ34 2025-11-21T00:00:00Z 1763683200 2025-11-22T00:00:00Z 1763769600 24
# 01KAT01MDE4A2B65KDAXRKDD7V 2025-11-22T08:00:00Z 1763798400 2025-11-24T00:00:00Z 1763942400 40
# 01KAT8AAGXGMJHW2P8HMFYSJPY 2025-11-22T08:00:00Z 1763798400 2025-11-24T00:00:00Z 1763942400 40
# 01KATGGGE064ENGTE6QZMQM3MG 2025-11-22T08:00:00Z 1763798400 2025-11-24T00:00:00Z 1763942400 40
# 01KATP9FJFR68M0VP9FEJ953PX 2025-11-22T08:00:00Z 1763798400 2025-11-24T00:00:00Z 1763942400 40
# 01KBMG51E4RGAXV1KQ5K5G4PEC 2025-11-20T00:00:00Z 1763596800 2025-11-24T00:00:00Z 1763942400 96
# 01KBN9BG24Z6TJDB9M66AE347W 2025-11-20T00:00:00Z 1763596800 2025-11-24T00:00:00Z 1763942400 96
# 01KBSQZ0NAAD4ZXWDSH63VED58 2025-11-20T00:00:00Z 1763596800 2025-11-24T00:00:00Z 1763942400 96
# 01KBSR11CSWYQG3B81MAEH9CJJ 2025-11-20T00:00:00Z 1763596800 2025-11-24T00:00:00Z 1763942400 96
# 01KAZ9FXJD9KG5DMPFGCHCKJRM 2025-11-24T00:00:00Z 1763942400 2025-11-25T08:00:00Z 1764057600 32
# 01KAZJ8AD61S0QCPGN2SVK4AK8 2025-11-24T00:00:00Z 1763942400 2025-11-25T08:00:00Z 1764057600 32
# 01KAZAD7SQFMB32G0ZGX2YY927 2025-11-24T00:00:00Z 1763942400 2025-11-26T00:00:00Z 1764115200 48
# 01KAZJ7Y9XYD3PY417R81Q9QBM 2025-11-24T00:00:00Z 1763942400 2025-11-26T00:00:00Z 1764115200 48
# 01KB01HHAX2710WXDVGK53RW61 2025-11-24T00:00:00Z 1763942400 2025-11-26T00:00:00Z 1764115200 48
# 01KB044DXSMND6GKPBQ0FTJNAV 2025-11-24T00:00:00Z 1763942400 2025-11-26T00:00:00Z 1764115200 48
# 01KBMC3G58ZKGVK8BYJ3D2GY6Q 2025-11-25T16:00:00Z 1764086400 2025-11-26T08:00:00Z 1764144000 16
# 01KBN5EDBAH1CEW0NW8RFP4CW4 2025-11-25T16:00:00Z 1764086400 2025-11-26T08:00:00Z 1764144000 16
# 01KB4DBPYK89FKX9PQ3809W5TM 2025-11-26T16:00:00Z 1764172800 2025-11-28T00:00:00Z 1764288000 32
# 01KB4EQ82AR4AZHN54BBQ35FXH 2025-11-26T00:00:00Z 1764115200 2025-11-28T00:00:00Z 1764288000 48
# 01KB4FRGY5DRRT06TFVRSBFCWK 2025-11-26T16:00:00Z 1764172800 2025-11-28T00:00:00Z 1764288000 32
# 01KB4NVN4SMREK8C5J0G0R65YF 2025-11-26T00:00:00Z 1764115200 2025-11-28T00:00:00Z 1764288000 48
# 01KB4ZZ0GRDE9KCFKASYWVAH47 2025-11-26T00:00:00Z 1764115200 2025-11-28T00:00:00Z 1764288000 48
# 01KB50ZTP6V0MJK9TFWVYFCP7S 2025-11-26T00:00:00Z 1764115200 2025-11-28T00:00:00Z 1764288000 48
# 01KB9K1WW87JJ0JY8VTKNP82CX 2025-11-28T00:00:00Z 1764288000 2025-11-30T00:00:00Z 1764460800 48
# 01KB9MVWVHQ3BZ26N31E3T021K 2025-11-28T00:00:00Z 1764288000 2025-11-30T00:00:00Z 1764460800 48
# 01KBA9JK1CJHY75EHT2DKZ22DH 2025-11-28T00:00:00Z 1764288000 2025-11-30T00:00:00Z 1764460800 48
# 01KBAAD12GQXSE48RPCE0CY3XY 2025-11-28T00:00:00Z 1764288000 2025-11-30T00:00:00Z 1764460800 48
# 01KBERVMBY51CE6BSTPMSCB5DY 2025-11-30T00:00:00Z 1764460800 2025-12-02T00:00:00Z 1764633600 48
# 01KBF078VX9K691AHTKBXQDAX5 2025-11-30T00:00:00Z 1764460800 2025-12-02T00:00:00Z 1764633600 48
# 01KBFQV3VNVPM1A9NDFBQBBB2C 2025-11-30T00:00:00Z 1764460800 2025-12-02T00:00:00Z 1764633600 48
# 01KBFT8DXQ0C1AT4SG8VBKH92Z 2025-11-30T00:00:00Z 1764460800 2025-12-02T00:00:00Z 1764633600 48
# 01KBMXFZC9J48BZKVQ1E8PE2MS 2025-11-28T00:00:00Z 1764288000 2025-12-02T00:00:00Z 1764633600 96
# 01KBNM34M5878GF7026DJ9E6Y6 2025-11-28T00:00:00Z 1764288000 2025-12-02T00:00:00Z 1764633600 96
# 01KBT7GEGR283SQX4Z30ZSE99S 2025-11-28T00:00:00Z 1764288000 2025-12-02T00:00:00Z 1764633600 96
# 01KBT7NTTM34BT6Z6A9NNE8CVR 2025-11-28T00:00:00Z 1764288000 2025-12-02T00:00:00Z 1764633600 96
# 01KBKVQK5HWWZAZJ6G068DPQHH 2025-12-02T00:00:00Z 1764633600 2025-12-03T00:00:00Z 1764720000 24
# 01KBKZW8GZTJS44AGR7JYY5WZR 2025-12-02T00:00:00Z 1764633600 2025-12-03T00:00:00Z 1764720000 24
# 01KBKX944VZXNNC5KBZYK2B1A5 2025-12-02T00:00:00Z 1764633600 2025-12-04T00:00:00Z 1764806400 48
# 01KBM49D71PYTBCQVXX9344VEA 2025-12-03T08:00:00Z 1764748800 2025-12-04T00:00:00Z 1764806400 16
# 01KBMX5NMED6C3JKJWNXRA577Q 2025-12-03T08:00:00Z 1764748800 2025-12-04T00:00:00Z 1764806400 16
# 01KBMXXWDC3YD2EWS5B6JFD87H 2025-12-02T00:00:00Z 1764633600 2025-12-04T00:00:00Z 1764806400 48
# 01KBNW9E3DQYG4Y189Z6KS2PKV 2025-12-02T00:00:00Z 1764633600 2025-12-04T00:00:00Z 1764806400 48
# 01KBPXN7DSNN21RJVCYR4ATATS 2025-12-02T00:00:00Z 1764633600 2025-12-04T00:00:00Z 1764806400 48
# 01KDCH6GFJS9W0JEJ5MTM5VB4P 2025-12-04T00:00:00Z 1764806400 2025-12-18T00:00:00Z 1766016000 336
# 01KDCTV95JKV1RG3AKXNMF4J6Z 2025-12-04T00:00:00Z 1764806400 2025-12-18T00:00:00Z 1766016000 336
# 01KDFFD341TD2SVKMMHQD40ZWH 2025-12-04T00:00:00Z 1764806400 2025-12-18T00:00:00Z 1766016000 336
# 01KDFXRVABR9ZYGX7B4KVNPZZ2 2025-12-04T00:00:00Z 1764806400 2025-12-18T00:00:00Z 1766016000 336

Checklist:

  • depool titan2001
  • disable puppet
  • stop thanos-compact
  • stop thanos-store
  • clean up /srv/thanos-store
  • update the list of blocks (root@titan2001:/srv/rewrite# bash rewrite.sh dumpulid)
  • manually clean up the blocks inside a tmux session
  • Test -> This query should display a continuous graph: sum by (site) ({app="mediawiki", deployment="mw-videoscaler", envoy_http_conn_manager_prefix="admin", kubernetes_namespace="mw-videoscaler", le="+Inf", prometheus="k8s"}) (hint: Execute the query by tunneling directly to the Thanos Query to bypass the Query Frontend cache)
Stashbot added a comment.Jan 16 2026, 8:50 AM

Mentioned in SAL (#wikimedia-operations) [2026-01-16T08:50:01Z] <tappof> depool titan2001, cleaning up block 01K88XDMJ9S0T2DR5K00VG9CFE (T410152)

tappof added a comment.EditedJan 16 2026, 8:50 AM
root@titan2001:/srv/rewrite# du -sh ../thanos-store/01K88XDMJ9S0T2DR5K00VG9CFE
1.7G	../thanos-store/01K88XDMJ9S0T2DR5K00VG9CFE
{__name__="mediawiki_action_api_modules_hit_total", api_endpoint="query", api_module="MediaWiki_Api", api_type="ACTION_API", app="statsd-exporter", instance="10.67.135.132:9102", job="k8s-pods", kubernetes_namespace="mw-api-ext", kubernetes_pod_name="statsd-exporter-prometheus-7c8d4f77c9-dwwwg", method="GET", path="action_query_format_php_maxlag_smaxage_0_maxage_0_assert_assertuser_requestid_1027860695_servedby_1_curtimestamp_responselanginfo_origin_crossorigin_uselang_user_variant_errorformat_bc_errorlang_uselang_errorsuselocal_centralauthtoken", pod_template_hash="7c8d4f77c9", release="prometheus", routed_via="prometheus", status="200"}

{F71537865}

root@titan2001:/srv/rewrite# systemd-run --scope -p MemoryLimit=80G /usr/bin/thanos tools bucket rewrite --objstore.config-file=/etc/thanos-store/objstore.yaml --id=01K88XDMJ9S0T2DR5K00VG9CFE --tmp.dir=/srv/rewrite/temp --no-dry-run --delete-blocks --no-rewrite.add-change-log --rewrite.to-delete-config-file=/srv/rewrite/matchers.yaml --enable-auto-gomemlimit --auto-gomemlimit.ratio=0.9 2>&1 | tee -a out/01K88XDMJ9S0T2DR5K00VG9CFE.out

Running scope as unit: run-ra10b454692c8412392a2bb1d84f121c8.scope
ts=2026-01-16T09:00:23.642406395Z caller=factory.go:54 level=info msg="loading bucket configuration"
ts=2026-01-16T09:00:23.643272175Z caller=tools_bucket.go:1228 level=info msg="downloading block" source=01K88XDMJ9S0T2DR5K00VG9CFE
ts=2026-01-16T09:02:52.009145805Z caller=tools_bucket.go:1280 level=info msg="starting rewrite for block" source=01K88XDMJ9S0T2DR5K00VG9CFE new=01KF30P3JVN50HEJVF4HAASX5H toDelete="- matchers: \"{__name__=~\\\"mediawiki_action_api_modules_.*\\\"}\"\n" toRelabel=
ts=2026-01-16T09:04:06.833485781Z caller=compactor.go:42 level=info msg="processed 10.00% of 79450615 series"
ts=2026-01-16T09:05:15.182962255Z caller=compactor.go:42 level=info msg="processed 20.00% of 79450615 series"
ts=2026-01-16T09:06:30.977285441Z caller=compactor.go:42 level=info msg="processed 30.00% of 79450615 series"
ts=2026-01-16T09:07:43.049075613Z caller=compactor.go:42 level=info msg="processed 40.00% of 79450615 series"
ts=2026-01-16T09:08:54.84806927Z caller=compactor.go:42 level=info msg="processed 50.00% of 79450615 series"
ts=2026-01-16T09:10:09.001767469Z caller=compactor.go:42 level=info msg="processed 60.00% of 79450615 series"
ts=2026-01-16T09:11:23.00639362Z caller=compactor.go:42 level=info msg="processed 70.00% of 79450615 series"
ts=2026-01-16T09:12:35.369566684Z caller=compactor.go:42 level=info msg="processed 80.00% of 79450615 series"
ts=2026-01-16T09:13:48.049149509Z caller=compactor.go:42 level=info msg="processed 90.00% of 79450615 series"
ts=2026-01-16T09:14:59.607136949Z caller=compactor.go:42 level=info msg="processed 100.00% of 79450615 series"
ts=2026-01-16T09:14:59.607203868Z caller=tools_bucket.go:1290 level=info msg="wrote new block after modifications; flushing" source=01K88XDMJ9S0T2DR5K00VG9CFE new=01KF30P3JVN50HEJVF4HAASX5H
ts=2026-01-16T09:17:18.859798514Z caller=tools_bucket.go:1299 level=info msg="uploading new block" source=01K88XDMJ9S0T2DR5K00VG9CFE new=01KF30P3JVN50HEJVF4HAASX5H
ts=2026-01-16T09:22:47.265052538Z caller=tools_bucket.go:1309 level=info msg=uploaded source=01K88XDMJ9S0T2DR5K00VG9CFE new=01KF30P3JVN50HEJVF4HAASX5H
ts=2026-01-16T09:22:47.673488729Z caller=block.go:203 level=info msg="block has been marked for deletion" block=01K88XDMJ9S0T2DR5K00VG9CFE
ts=2026-01-16T09:22:47.673528998Z caller=tools_bucket.go:1317 level=info msg="rewrite done" IDs=01K88XDMJ9S0T2DR5K00VG9CFE
ts=2026-01-16T09:22:47.673563225Z caller=main.go:174 level=info msg=exiting
root@titan2001:/srv/rewrite# thanos tools bucket cleanup --objstore.config-file /etc/thanos-compact/objstore.yaml --delete-delay 0h
...
ts=2026-01-16T11:02:23.455787344Z caller=blocks_cleaner.go:54 level=info msg="deleted block marked for deletion" block=01K88XDMJ9S0T2DR5K00VG9CFE
...
root@titan2001:/srv/rewrite# du -sh ../thanos-store/01KF30P3JVN50HEJVF4HAASX5H
868M	../thanos-store/01KF30P3JVN50HEJVF4HAASX5H

Querying the data without deduplication shows that the action API series is missing in the rewritten block.

{F71538937}
{F71572892}

tappof added a comment.EditedJan 21 2026, 3:55 PM

While processing larger blocks (48h), some errors popped up:

ts=2026-01-21T14:44:24.048672522Z caller=factory.go:54 level=info msg="loading bucket configuration"
ts=2026-01-21T14:44:32.901144624Z caller=tools_bucket.go:1228 level=info msg="downloading block" source=01K6ZQ3J7SS9THAV7MXXKK4BZJ
ts=2026-01-21T14:55:21.371253101Z caller=tools_bucket.go:1280 level=info msg="starting rewrite for block" source=01K6ZQ3J7SS9THAV7MXXKK4BZJ new=01KFGGV48EG33NDPVK2Z1B00Q6 toDelete="- matchers: \"{__name__=~\\\"mediawiki_action_api_modules_.*\\\"}\"\n" toRelabel=
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x24ead6c]

goroutine 119 [running]:
github.com/thanos-io/thanos/pkg/compactv2.(*lazyPopulatableChunk).Bytes(0xc0011da540?)
        /build/thanos-0.38.0/pkg/compactv2/chunk_series_set.go:131 +0x2c
github.com/prometheus/prometheus/tsdb/chunks.(*Writer).WriteChunks(0xc00126c550, {0xc0011da840, 0x1, 0x1})
        /tmp/thanos-build/go/pkg/mod/github.com/prometheus/prometheus@v0.301.0/tsdb/chunks/chunks.go:465 +0x131
github.com/thanos-io/thanos/pkg/block.(*statsGatheringSeriesWriter).WriteChunks(0xc0009bc750, {0xc0011da840, 0x1, 0x5?})
        /build/thanos-0.38.0/pkg/block/writer.go:173 +0x2f
github.com/thanos-io/thanos/pkg/compactv2.(*Compactor).write(0xc00178fda0, {0x3c8c6f0, 0xc00118a500}, {0x3c83df0, 0xc00126c5f0}, {0x3c8d9c8, 0xc00126c640}, {0x7f5cd19c0d10, 0xc0009bc750}, {0x3c67760, ...})
        /build/thanos-0.38.0/pkg/compactv2/chunk_series_set.go:219 +0x248
github.com/thanos-io/thanos/pkg/compactv2.(*Compactor).WriteSeries(0xc00178fda0, {0x3c8c6f0, 0xc00118a500}, {0xc00178ff30, 0x1, 0xc000963340?}, {0x3c91fc0, 0xc0009bc750}, {0x3c67760, 0xc0009f6080}, ...)
        /build/thanos-0.38.0/pkg/compactv2/compactor.go:148 +0xe2d
main.registerBucketRewrite.func1.1()
        /build/thanos-0.38.0/cmd/thanos/tools_bucket.go:1281 +0x1412
github.com/oklog/run.(*Group).Run.func1({0xc0005bd930?, 0xc000963530?})
        /tmp/thanos-build/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
        /tmp/thanos-build/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x5a

Due to https://github.com/thanos-io/thanos/issues/4298, proceeding with blacklisting already downsampled blocks.

tappof added a subtask: T415317: Implement an alert to detect changes in the number of ingested series.Jan 22 2026, 10:32 PM
tappof added a comment.Jan 29 2026, 9:47 PM

The rewriting process finished today.

We have seen some good results, especially in terms of memory usage, which is most noticeable on titan1002.

image.png (2,284×637 px, 122 KB)

However, the index headers are smaller but still quite large:

01KFWMK94AMBNGK7J8DTH5S3J9  2025-09-25T00:00:00Z  2025-10-03T00:00:00Z  191h59m59.999s  100,185,249  88,717,525,826   832,007,491    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  309M
01KFWT9Y8Q1YZWG6C7TS2JSW33  2025-09-25T00:00:00Z  2025-10-01T00:00:00Z  144h0m0s        131,401,833  94,214,810,409   916,199,235    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  179M
01KFX0V5S4V2H55PR3N1TN17WX  2025-09-25T00:00:00Z  2025-10-03T00:00:00Z  191h59m59.998s  100,183,803  88,720,837,205   831,984,013    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  309M
01KFX6VE9QF7XQEVVKX4JWM3H0  2025-09-25T00:00:00Z  2025-10-01T00:00:00Z  143h59m59.999s  131,573,982  94,216,763,734   916,380,445    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  179M
01KFFPFB9562T35FWHEZ6G10NG  2025-10-01T00:00:00Z  2025-10-03T00:00:00Z  47h59m59.999s   114,155,988  30,926,509,986   364,295,488    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  907M
01KFFTFFRF597640GJBWNYW3P6  2025-10-01T00:00:00Z  2025-10-03T00:00:00Z  48h0m0s         114,341,289  30,925,747,292   364,504,667    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  909M
01KFGAH0F50DMK9949ZCP4WTSX  2025-10-03T00:00:00Z  2025-10-05T00:00:00Z  48h0m0s         13,316,717   21,340,101,837   185,045,863    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.1G
01KFGDMKCQ34XE50EW3XC9BPGR  2025-10-03T00:00:00Z  2025-10-05T00:00:00Z  48h0m0s         13,317,363   21,338,906,221   185,050,278    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.1G
01KFXDMX29C3ADQ2XVVN6N8T74  2025-10-03T00:00:00Z  2025-10-07T00:00:00Z  96h0m0s         59,894,901   63,540,631,235   585,719,713    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.8G
01KFXJSA5DHGMQ2X7T5ZB6HR1H  2025-10-03T00:00:00Z  2025-10-07T00:00:00Z  96h0m0s         59,797,606   63,541,120,696   585,618,320    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.8G
01KFGNZQF55N9NV5HX2W20NCRT  2025-10-05T00:00:00Z  2025-10-07T00:00:00Z  48h0m0s         41,562,449   22,115,137,153   218,834,001    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.1G
01KFGSEBVR2ZN8PDE505PKGEE7  2025-10-05T00:00:00Z  2025-10-07T00:00:00Z  48h0m0s         41,560,352   22,113,761,889   218,835,010    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.1G
01KFGX0Y5E4MKDKTEBTH9MNZRW  2025-10-07T00:00:00Z  2025-10-09T00:00:00Z  48h0m0s         56,036,912   21,606,233,718   229,173,346    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.2G
01KFH0YEC8RZNQN35DHH4CGVDB  2025-10-07T00:00:00Z  2025-10-09T00:00:00Z  48h0m0s         66,425,152   30,649,216,525   313,933,684    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.2G
01KFH4CMHH1E4BJCMJZE5BED5Y  2025-10-07T00:00:00Z  2025-10-09T00:00:00Z  48h0m0s         56,037,200   21,608,109,272   229,147,619    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.2G
01KFH8C390H6AW42GCNDZSE4M6  2025-10-07T00:00:00Z  2025-10-09T00:00:00Z  48h0m0s         66,592,894   30,648,459,537   314,085,639    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.2G
01KFHBRFDPPK0JMZPQFKTT1Y40  2025-10-09T00:00:00Z  2025-10-11T00:00:00Z  48h0m0s         35,420,484   21,304,073,789   205,927,238    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.4G
01KFHFG8W4M6X9S324YPSEDART  2025-10-09T00:00:00Z  2025-10-11T00:00:00Z  48h0m0s         35,426,174   21,300,990,416   205,918,856    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.4G
01KFXR35FY89FEGRW06H5XVMWZ  2025-10-09T00:00:00Z  2025-10-13T00:00:00Z  96h0m0s         47,258,804   62,116,561,140   560,804,188    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.7G
01KFXX867S8EBA54901AXA8EV6  2025-10-09T00:00:00Z  2025-10-13T00:00:00Z  96h0m0s         48,028,925   62,117,545,257   561,576,504    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.7G
01KFHKBNEJRR7F6F6XEXHDF7FK  2025-10-11T00:00:00Z  2025-10-13T00:00:00Z  48h0m0s         7,972,260    22,381,163,850   187,570,742    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.5G
01KFHQ5VK2MP972WE4E14EQXRD  2025-10-11T00:00:00Z  2025-10-13T00:00:00Z  47h59m59.998s   7,968,938    22,381,779,133   187,576,534    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.5G
01KFJ2GZKVKAPFYA7ZRB47Y2Z0  2025-10-13T00:00:00Z  2025-10-14T08:00:00Z  32h0m0s         31,891,406   14,391,631,493   144,208,769    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.8G
01KFJ55ZFA00PPW29D2CS2BQC0  2025-10-13T00:00:00Z  2025-10-15T00:00:00Z  48h0m0s         72,640,843   29,637,631,195   312,074,662    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  835M
01KFJ87WY1J3YE02DBJ7JK9MN7  2025-10-13T00:00:00Z  2025-10-14T08:00:00Z  32h0m0s         31,889,534   14,390,441,544   144,202,801    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.8G
01KFJAVJXPK5GD7YZSE0ACW4HP  2025-10-13T00:00:00Z  2025-10-15T00:00:00Z  48h0m0s         72,763,582   29,639,263,418   312,211,053    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  835M
01KFBG0J45TTD5THXXZTJ01ATB  2025-10-14T08:00:00Z  2025-10-14T16:00:00Z  8h0m0s          20,181,261   3,463,566,452    41,768,113     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  498M
01KFBH435GKP9HPHPMKJA2J7B3  2025-10-14T08:00:00Z  2025-10-14T16:00:00Z  8h0m0s          20,182,176   3,463,406,657    41,808,569     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  500M
01KFJDY39VHGETA7BB640GKEKZ  2025-10-14T16:00:00Z  2025-10-16T08:00:00Z  40h0m0s         45,681,688   17,783,142,680   187,700,058    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.1G
01KFJH56T2W8SQQPHQ538Z5WGV  2025-10-14T16:00:00Z  2025-10-16T08:00:00Z  40h0m0s         45,681,087   17,783,744,235   187,701,580    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.1G
01KFJM9SGT5KJGJJZEGR2GSTT0  2025-10-15T00:00:00Z  2025-10-17T00:00:00Z  48h0m0s         59,348,682   28,912,199,496   294,221,996    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  837M
01KFJQ67044GWQFCE6KPZGQRY1  2025-10-15T00:00:00Z  2025-10-17T00:00:00Z  48h0m0s         59,427,101   28,914,073,317   294,301,409    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  837M
01KFBQQHBC9BKRBRAXFVQ62T2H  2025-10-16T08:00:00Z  2025-10-16T16:00:00Z  8h0m0s          13,535,354   3,610,343,121    36,249,541     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  516M
01KFBRGSF488VWZ2JEYAPBSBDM  2025-10-16T08:00:00Z  2025-10-16T16:00:00Z  8h0m0s          13,536,854   3,610,554,649    36,251,349     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  518M
01KFBZFR3Q3FAVER6F1Q4TK354  2025-10-16T16:00:00Z  2025-10-17T00:00:00Z  8h0m0s          18,763,710   3,593,377,609    41,161,760     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  457M
01KFC09F7SP9114KX8HAM3E3S1  2025-10-16T16:00:00Z  2025-10-17T00:00:00Z  8h0m0s          18,763,319   3,593,421,463    41,161,523     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  461M
01KFJT3ZB2T83ZCNAV0QNRB605  2025-10-17T00:00:00Z  2025-10-19T00:00:00Z  48h0m0s         8,190,797    22,490,896,619   189,132,800    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.5G
01KFJX50XGBGBJF1Y2NSEVYE4F  2025-10-17T00:00:00Z  2025-10-19T00:00:00Z  48h0m0s         8,196,001    22,490,487,307   189,138,742    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.5G
01KFY2A2HXFZQHJ2YNTH4YMC9X  2025-10-17T00:00:00Z  2025-10-21T00:00:00Z  96h0m0s         30,290,228   60,876,949,969   535,668,248    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.6G
01KFY6X30J0RFAGB7WQDMQA36R  2025-10-17T00:00:00Z  2025-10-21T00:00:00Z  96h0m0s         30,317,581   60,879,142,627   535,670,854    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.6G
01KFK0WRM4ACK8WE0XW5NCJ1A6  2025-10-19T00:00:00Z  2025-10-20T08:00:00Z  32h0m0s         8,052,989    15,002,567,704   125,819,853    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.7G
01KFK307FBNKMMDYM4EC7H801T  2025-10-19T00:00:00Z  2025-10-20T08:00:00Z  32h0m0s         8,055,068    15,002,734,426   125,821,738    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.7G
01KFC13MAB4K9SZZ89N3AS2D03  2025-10-20T08:00:00Z  2025-10-20T16:00:00Z  8h0m0s          13,846,007   3,702,673,387    36,900,351     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  513M
01KFC1XCDEKNG83Q9GRW2EGRE2  2025-10-20T08:00:00Z  2025-10-20T16:00:00Z  8h0m0s          13,845,310   3,703,060,914    36,902,487     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  514M
01KF30P3JVN50HEJVF4HAASX5H  2025-10-20T16:00:00Z  2025-10-21T08:00:00Z  16h0m0s         19,395,921   7,013,067,891    71,125,626     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  868M
01KF3XMTJBV0GGHAYGWGEEX0XP  2025-10-20T16:00:00Z  2025-10-21T08:00:00Z  16h0m0s         19,396,295   7,013,563,384    71,126,829     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  872M
01KFK54ZVR7Y9Y12P563KJX2AC  2025-10-21T00:00:00Z  2025-10-23T00:00:00Z  48h0m0s         68,422,327   28,928,436,151   302,771,471    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  967M
01KFK87VF5JQCMCZ82ZJ6WVKVJ  2025-10-21T00:00:00Z  2025-10-23T00:00:00Z  48h0m0s         68,549,971   28,927,541,030   302,902,730    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  968M
01KFC2P8CBHRAS3FAZZYQKT49S  2025-10-21T08:00:00Z  2025-10-21T16:00:00Z  8h0m0s          18,962,327   3,455,331,717    40,688,781     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  520M
01KFC3JG1Y6ZX6KXCDZWPYDBT5  2025-10-21T08:00:00Z  2025-10-21T16:00:00Z  8h0m0s          18,958,848   3,455,215,256    40,685,367     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  522M
01KFKB8XNY26S9E8F0D5DFNYNC  2025-10-21T16:00:00Z  2025-10-23T00:00:00Z  32h0m0s         39,497,703   13,842,520,803   147,623,966    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.7G
01KFKDY39Z67D7AVQAHZFVZZHY  2025-10-21T16:00:00Z  2025-10-23T00:00:00Z  32h0m0s         39,500,954   13,842,638,358   147,629,309    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.8G
01KFBDF7QM5J0VJ6WXZT12H5H9  2025-10-23T00:00:00Z  2025-10-23T08:00:00Z  8h0m0s          11,103,473   3,514,371,556    33,330,257     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  542M
01KFC4E172PCEKWEYVVNN12AG7  2025-10-23T00:00:00Z  2025-10-23T08:00:00Z  8h0m0s          11,103,246   3,514,530,175    33,330,652     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  546M
01KFYBN88GA4627JCP1FCWEYBV  2025-10-23T00:00:00Z  2025-10-27T00:00:00Z  96h0m0s         47,913,182   60,736,552,682   552,645,281    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.7G
01KFYH443TC8V02FZHWNV28KGJ  2025-10-23T00:00:00Z  2025-10-27T00:00:00Z  96h0m0s         47,953,398   60,748,706,344   552,687,684    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.7G
01KFC56YQFWV49MTVP5KN248CN  2025-10-23T08:00:00Z  2025-10-23T16:00:00Z  8h0m0s          15,011,439   3,489,986,657    36,987,153     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  523M
01KFC5ZQNY75W1K57WQ5YY773D  2025-10-23T08:00:00Z  2025-10-23T16:00:00Z  8h0m0s          15,012,946   3,490,022,268    36,989,721     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  526M
01KFKGH8H01JBH86CJK7H63Y47  2025-10-23T16:00:00Z  2025-10-25T00:00:00Z  32h0m0s         21,638,501   14,486,533,964   135,817,381    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.8G
01KFKJVK3VGZVXZRBB2WBE7FVF  2025-10-23T16:00:00Z  2025-10-25T00:00:00Z  32h0m0s         21,638,904   14,486,671,655   135,820,668    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.8G
01KFKNA2EHW4CW1P7DBS4KV1MY  2025-10-25T00:00:00Z  2025-10-27T00:00:00Z  48h0m0s         10,599,475   22,467,031,989   191,514,040    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.3G
01KFKRAQR64HQ45NAJ9KB062Y3  2025-10-25T00:00:00Z  2025-10-27T00:00:00Z  48h0m0s         10,599,113   22,467,169,764   191,516,861    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.3G
01KFKY3DW2E8ATB1G1M50Q3JV1  2025-10-27T00:00:00Z  2025-10-29T00:00:00Z  48h0m0s         58,447,122   21,864,067,576   233,458,355    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.9G
01KFM1FFX0F7KBC0RVHFBXA7ZV  2025-10-27T00:00:00Z  2025-10-29T00:00:00Z  48h0m0s         70,819,988   30,141,119,411   314,693,539    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  911M
01KFM54JWXYPQ0AK9DC405HXHX  2025-10-27T00:00:00Z  2025-10-29T00:00:00Z  48h0m0s         58,444,397   21,864,825,475   233,490,021    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.9G
01KFM8WWFPSC4Z46YT54QQ8K7A  2025-10-27T00:00:00Z  2025-10-29T00:00:00Z  48h0m0s         70,883,298   30,142,108,805   314,759,087    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  911M
01KFMCKRGXVWTB1X45ZRYCTF8S  2025-10-29T00:00:00Z  2025-10-31T00:00:00Z  48h0m0s         60,931,912   22,868,841,743   244,655,384    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.1G
01KFMGFC60ZC84DKFVBN5BY9PM  2025-10-29T00:00:00Z  2025-10-31T00:00:00Z  48h0m0s         75,596,939   31,539,315,101   331,612,617    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  948M
01KFMM273YJRJN22KKYKMSZDRF  2025-10-29T00:00:00Z  2025-10-31T00:00:00Z  48h0m0s         60,928,214   22,869,582,387   244,653,896    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.1G
01KFMQZ0XM2H7QJ5F0KFTZXGK6  2025-10-29T00:00:00Z  2025-10-31T00:00:00Z  48h0m0s         75,644,184   31,539,223,835   331,732,444    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  949M
01KFMVHZWJX3ZFM0BMJNTR5GR5  2025-10-31T00:00:00Z  2025-11-02T00:00:00Z  48h0m0s         11,067,333   24,025,568,656   204,329,290    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.1G
01KFMYP0NZR2812KW98FNPFVR2  2025-10-31T00:00:00Z  2025-11-02T00:00:00Z  48h0m0s         11,066,948   24,026,180,940   204,340,744    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.1G
01KFYQ7QMHWFHRDRHQEGBZEJ89  2025-10-31T00:00:00Z  2025-11-04T00:00:00Z  96h0m0s         54,677,031   67,532,887,703   615,133,500    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.8G
01KFYXQ7WM0YV6EKJPY46AWCWS  2025-10-31T00:00:00Z  2025-11-04T00:00:00Z  96h0m0s         55,231,916   67,535,516,010   615,709,738    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.8G
01KFNTYEPB5BA3C9SEG4H2T6GS  2025-11-02T00:00:00Z  2025-11-04T00:00:00Z  48h0m0s         39,518,602   24,068,689,016   232,703,484    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.2G
01KFNYNTCSMKC12DQZK76BCRRB  2025-11-02T00:00:00Z  2025-11-04T00:00:00Z  48h0m0s         39,521,269   24,068,948,539   232,690,529    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.2G
01KFC6W1ZCRDWFK3WKQ137HW1M  2025-11-04T00:00:00Z  2025-11-04T16:00:00Z  16h0m0s         16,921,695   7,819,916,043    74,737,821     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  898M
01KFCZQVXDJK0HXXE85M26KZ2N  2025-11-04T00:00:00Z  2025-11-04T16:00:00Z  16h0m0s         16,921,745   7,819,920,180    74,738,335     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  899M
01KFP2B9AWK71DFXEYS5VBD67G  2025-11-04T00:00:00Z  2025-11-06T00:00:00Z  48h0m0s         62,024,434   33,173,370,616   331,868,121    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.1G
01KFP5P3PATH6R6K58M8S9W09B  2025-11-04T00:00:00Z  2025-11-06T00:00:00Z  48h0m0s         62,066,098   33,174,312,529   331,915,226    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.1G
01KFD130KF6GWWRBNDQ36T03TG  2025-11-04T16:00:00Z  2025-11-05T00:00:00Z  8h0m0s          24,019,086   3,790,962,545    48,094,385     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  455M
01KFD23DHEG253GTDV40ZGBV2H  2025-11-04T16:00:00Z  2025-11-05T00:00:00Z  8h0m0s          24,009,111   3,790,907,848    48,084,190     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  457M
01KFD385ZY3GQ4K2KSD7ZNMDAV  2025-11-05T00:00:00Z  2025-11-06T00:00:00Z  24h0m0s         25,414,849   11,363,050,349   112,644,025    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.2G
01KFD54YNXA5B631HCXJJ17QX4  2025-11-05T00:00:00Z  2025-11-06T00:00:00Z  24h0m0s         25,412,720   11,362,895,070   112,643,738    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.2G
01KFP9AMPBVC3VKP8K13496X51  2025-11-06T00:00:00Z  2025-11-08T00:00:00Z  48h0m0s         43,297,414   22,982,981,855   227,807,621    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.3G
01KFPD21MQ1DX795HA1ZSQBGQ4  2025-11-06T00:00:00Z  2025-11-08T00:00:00Z  48h0m0s         43,296,743   22,986,858,327   227,810,132    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.3G
01KFZ3H52NHYX6RTANKNJ9XXYS  2025-11-06T00:00:00Z  2025-11-10T00:00:00Z  96h0m0s         57,723,180   68,663,033,643   627,302,446    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  2.3G
01KFZ9SWJPAMQHYFP1SJAFDHZF  2025-11-06T00:00:00Z  2025-11-10T00:00:00Z  96h0m0s         57,763,433   68,664,673,301   627,358,415    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  2.3G
01KFPGSSD5W2Q1CGD9R2Y9PCVJ  2025-11-08T00:00:00Z  2025-11-10T00:00:00Z  48h0m0s         8,694,457    24,045,558,656   201,817,493    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.3G
01KFPM2527MX7KHQ3E8E3HZZZZ  2025-11-08T00:00:00Z  2025-11-10T00:00:00Z  48h0m0s         8,695,098    24,046,210,836   201,821,199    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.3G
01KFPQFX98FS7XBN1TQPS32XYX  2025-11-10T00:00:00Z  2025-11-12T00:00:00Z  48h0m0s         60,325,797   23,299,594,477   246,370,310    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.3G
01KFPVJSGWP2RD8MBC99C49KNG  2025-11-10T00:00:00Z  2025-11-12T00:00:00Z  48h0m0s         74,804,156   34,168,760,106   350,602,077    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.2G
01KFQGWVCTMAJ4C1XY06SQQVP8  2025-11-10T00:00:00Z  2025-11-12T00:00:00Z  48h0m0s         60,325,903   23,329,340,950   246,373,494    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.3G
01KFQMTRQM7Q4GF5KPQ9XWJ3J7  2025-11-10T00:00:00Z  2025-11-12T00:00:00Z  48h0m0s         74,918,978   34,169,784,096   350,724,524    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.2G
01KFQRD1H9F6YJGGCXHAD5CRBB  2025-11-12T00:00:00Z  2025-11-14T00:00:00Z  48h0m0s         61,116,162   23,128,594,407   246,660,623    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.2G
01KFQW55A6V35VWBXHVDXHKDFQ  2025-11-12T00:00:00Z  2025-11-14T00:00:00Z  48h0m0s         78,865,512   33,563,708,385   351,085,451    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.4G
01KFQZXWWVX2YH0YYXT4TWT2XW  2025-11-12T00:00:00Z  2025-11-14T00:00:00Z  48h0m0s         61,114,622   23,128,908,216   246,685,771    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.2G
01KFR3RCAAWQM7X6G4RQS179HN  2025-11-12T00:00:00Z  2025-11-14T00:00:00Z  48h0m0s         79,038,436   33,565,196,969   351,274,628    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.4G
01KFR7F7GP7WRE6R31BXR422VY  2025-11-14T00:00:00Z  2025-11-16T00:00:00Z  48h0m0s         14,474,291   33,372,997,404   285,192,267    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.3G
01KFRA2JRHAPMM1QB03NJYS7Z2  2025-11-14T00:00:00Z  2025-11-16T00:00:00Z  48h0m0s         8,453,428    23,518,801,715   197,152,426    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.0G
01KFRCPS54YMES0D80YNJ4D3FH  2025-11-14T00:00:00Z  2025-11-16T00:00:00Z  48h0m0s         14,502,608   33,370,069,118   285,140,732    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.3G
01KFRFAGAM6JV1JHGA7S08JCGS  2025-11-14T00:00:00Z  2025-11-16T00:00:00Z  47h59m59.996s   8,454,794    23,518,615,594   197,152,601    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.0G
01KFRMT9Z741T0HKS417ACBSR4  2025-11-16T00:00:00Z  2025-11-18T00:00:00Z  48h0m0s         28,493,501   23,625,300,572   217,990,020    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.1G
01KFRQWSMSSBAD2KR1B6PKKHB4  2025-11-16T00:00:00Z  2025-11-18T00:00:00Z  48h0m0s         38,261,990   33,833,639,146   312,695,281    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.3G
01KFRTXT91KHFC95Q4EVPTGV22  2025-11-16T00:00:00Z  2025-11-18T00:00:00Z  48h0m0s         28,492,631   23,625,516,482   217,991,515    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.1G
01KFRY17K6KBTF90Q41DRE2Y8R  2025-11-16T00:00:00Z  2025-11-18T00:00:00Z  48h0m0s         38,311,364   33,835,044,296   312,755,880    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.3G
01KFS14Y1TK3XQDHX3QWKWH94S  2025-11-18T00:00:00Z  2025-11-20T00:00:00Z  48h0m0s         52,334,735   22,716,049,525   235,048,684    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.1G
01KFS4PST64B2X7GS25SXPCSS0  2025-11-18T00:00:00Z  2025-11-20T00:00:00Z  48h0m0s         69,894,615   32,422,921,933   331,465,763    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.3G
01KFS8CR5RZTKVJQGHX01R5YGN  2025-11-18T00:00:00Z  2025-11-20T00:00:00Z  48h0m0s         52,329,093   22,714,931,572   235,046,146    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.1G
01KFSCASHRJ6ZD5XMA3970XPTT  2025-11-18T00:00:00Z  2025-11-20T00:00:00Z  48h0m0s         69,973,316   32,423,897,205   331,551,323    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.3G
01KFD6X74Q9MEQDBWD33XMHHWN  2025-11-20T00:00:00Z  2025-11-20T16:00:00Z  16h0m0s         19,933,459   7,503,401,734    75,409,283     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  818M
01KFD86WHWE9CY8DQ8HFBXYVVW  2025-11-20T00:00:00Z  2025-11-20T16:00:00Z  16h0m0s         19,932,351   7,502,171,968    75,408,045     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  819M
01KFZFC47F13RAGN4E23RFFZCQ  2025-11-20T00:00:00Z  2025-11-24T00:00:00Z  96h0m0s         54,343,468   66,568,148,643   605,497,153    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.9G
01KFZMMANTS48CMSAAVKYJKPGE  2025-11-20T00:00:00Z  2025-11-24T00:00:00Z  96h0m0s         54,821,823   66,570,161,306   605,969,000    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.9G
01KFD9ESCEV4K99BXHSJ44K0YJ  2025-11-20T16:00:00Z  2025-11-21T00:00:00Z  8h0m0s          23,908,630   3,678,833,635    46,263,318     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  460M
01KFDABHVZWP6SHQT31CTAHAQ2  2025-11-20T16:00:00Z  2025-11-21T00:00:00Z  8h0m0s          23,906,212   3,678,889,943    46,253,390     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  462M
01KFDBDPPAYPY1S7596BMKFSX9  2025-11-21T00:00:00Z  2025-11-22T00:00:00Z  24h0m0s         12,011,490   11,642,708,513   101,779,300    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.3G
01KFDHMHHY19CA3Q8CQX7R2R3Z  2025-11-21T00:00:00Z  2025-11-22T00:00:00Z  24h0m0s         12,007,292   11,642,070,021   101,777,615    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.3G
01KFDK7S7RX07DV84D9CBVBAC6  2025-11-22T00:00:00Z  2025-11-22T08:00:00Z  8h0m0s          8,328,336    3,951,073,560    33,214,202     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  522M
01KFDKYQB51DTWCMKM9R0V7SBQ  2025-11-22T00:00:00Z  2025-11-22T08:00:00Z  8h0m0s          8,333,628    3,951,170,135    33,219,355     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  523M
01KFSFSSQJ9GFBWMXK3KCVRTKJ  2025-11-22T08:00:00Z  2025-11-24T00:00:00Z  40h0m0s         8,612,523    19,950,296,136   167,691,630    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.2G
01KFSJFP45F5MHT7HY9Q4CKS1H  2025-11-22T08:00:00Z  2025-11-24T00:00:00Z  40h0m0s         8,613,661    19,951,275,998   167,698,054    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.2G
01KFT4NGZJY6YFQZJVGHQ56Z01  2025-11-24T00:00:00Z  2025-11-25T08:00:00Z  32h0m0s         37,343,429   15,497,458,896   158,722,621    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.8G
01KFT7FJZ33ANED9K5K3AA9THX  2025-11-24T00:00:00Z  2025-11-26T00:00:00Z  48h0m0s         90,341,598   33,282,380,503   358,891,205    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  951M
01KFTB2TS2DP836KZKF16S12NS  2025-11-24T00:00:00Z  2025-11-26T00:00:00Z  48h0m0s         90,406,042   33,283,925,790   358,989,405    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  952M
01KFTEKPSGEAAHTXT5FQAZ1BRS  2025-11-24T00:00:00Z  2025-11-25T08:00:00Z  32h0m0s         37,340,758   15,496,599,629   158,711,407    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.8G
01KFDMPSCC1YK6GW4Y2HRHJVA6  2025-11-25T08:00:00Z  2025-11-25T16:00:00Z  8h0m0s          32,170,063   3,768,962,029    55,887,214     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  510M
01KFDNVDRKSA86B9K4NKZZBR6E  2025-11-25T08:00:00Z  2025-11-25T16:00:00Z  8h0m0s          32,170,819   3,769,260,156    55,883,591     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  513M
01KFDQ1XBEDVP5QAA09JNNQSF0  2025-11-25T16:00:00Z  2025-11-26T08:00:00Z  16h0m0s         24,962,260   7,552,028,804    79,841,761     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  925M
01KFDRK14WJRPDK0ANJT01XVMX  2025-11-25T16:00:00Z  2025-11-26T08:00:00Z  16h0m0s         24,964,162   7,552,312,232    79,839,403     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  919M
01KFTH8W60PCHVY7KKQ539GTEQ  2025-11-26T00:00:00Z  2025-11-28T00:00:00Z  48h0m0s         72,586,573   32,395,119,669   335,286,277    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  978M
01KFTMP2FNNMYZT08C0SYVP0DD  2025-11-26T00:00:00Z  2025-11-28T00:00:00Z  48h0m0s         72,653,981   32,396,337,719   335,361,372    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  978M
01KFDT1EZD3B1QFDQZ5S3VVAEK  2025-11-26T08:00:00Z  2025-11-26T16:00:00Z  8h0m0s          21,666,338   3,718,305,848    44,925,086     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  498M
01KFDTZKH7YDM4TMP6M31NPH75  2025-11-26T08:00:00Z  2025-11-26T16:00:00Z  8h0m0s          21,659,076   3,718,486,053    44,918,160     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  501M
01KFTQYW5K66N24YQW2QYBDAGZ  2025-11-26T16:00:00Z  2025-11-28T00:00:00Z  32h0m0s         44,698,888   15,277,417,191   164,863,444    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.5G
01KFTTQKA8KNQFWGGMGX68XDQZ  2025-11-26T16:00:00Z  2025-11-28T00:00:00Z  32h0m0s         44,699,485   15,278,176,747   164,866,221    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.6G
01KFTXVYKM1RZVEF8S8Q4DT9BN  2025-11-28T00:00:00Z  2025-11-30T00:00:00Z  48h0m0s         13,220,770   23,718,035,082   203,909,624    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.3G
01KFV16XPTVSDXK24M7X7J27J4  2025-11-28T00:00:00Z  2025-11-30T00:00:00Z  48h0m0s         13,220,601   23,718,974,356   203,911,196    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.3G
01KFZST2F9BMZPR07E0HGT9X2Z  2025-11-28T00:00:00Z  2025-12-02T00:00:00Z  96h0m0s         47,699,158   68,083,279,526   612,692,278    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  1.7G
01KFZYT11QHATNH96D5CFXJ0WT  2025-11-28T00:00:00Z  2025-12-02T00:00:00Z  96h0m0s         47,899,936   68,082,574,169   612,900,645    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  1.7G
01KFV9FDXQN7PGAJZEKF74Q21C  2025-11-30T00:00:00Z  2025-12-02T00:00:00Z  48h0m0s         31,616,798   23,625,509,717   222,084,716    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  2.3G
01KFVD7RCRHRVHXK6R19ZFDCHV  2025-11-30T00:00:00Z  2025-12-02T00:00:00Z  47h59m59.999s   31,613,079   23,624,546,075   222,074,618    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  2.3G
01KFDW4VAXKQZAQ54E06DJCC52  2025-12-02T00:00:00Z  2025-12-03T00:00:00Z  24h0m0s         42,581,974   11,384,154,586   130,593,314    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  1.2G
01KFDYAYDRER28YJ6R85GJ612M  2025-12-02T00:00:00Z  2025-12-03T00:00:00Z  24h0m0s         42,584,573   11,385,639,988   130,596,456    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  1.2G
01KFVGXSWTJDM3GWWRGJMXSBJZ  2025-12-02T00:00:00Z  2025-12-04T00:00:00Z  48h0m0s         93,302,323   33,385,132,828   364,723,588    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  789M
01KFVMTR18MN669GFDRZBR03PT  2025-12-02T00:00:00Z  2025-12-04T00:00:00Z  48h0m0s         93,317,284   33,385,826,954   364,745,195    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  789M
01KFE0AFG6WYQ6HJTW2D11TDQ7  2025-12-03T00:00:00Z  2025-12-03T08:00:00Z  8h0m0s          17,563,031   3,808,405,878    41,681,595     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  514M
01KFE16E0PVQFDTA8AQKC6M8A6  2025-12-03T00:00:00Z  2025-12-03T08:00:00Z  8h0m0s          17,561,843   3,808,128,467    41,679,431     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  518M
01KFE24DQE9A5TTMWDYZ5BCPE9  2025-12-03T08:00:00Z  2025-12-04T00:00:00Z  16h0m0s         34,918,903   7,604,539,884    90,527,449     prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  727M
01KFE3KQSCQN53WQC4Y5G8P0RK  2025-12-03T08:00:00Z  2025-12-04T00:00:00Z  16h0m0s         34,907,620   7,604,023,353    90,513,669     prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  718M
01KFVRJCD9CDBB8NF9JEDGG8PZ  2025-12-04T00:00:00Z  2025-12-06T00:00:00Z  48h0m0s         34,096,832   23,848,886,591   225,318,536    prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  445M
01KFVTWC3609YWNCW47QEV374N  2025-12-04T00:00:00Z  2025-12-06T00:00:00Z  48h0m0s         34,147,021   23,849,290,928   225,371,942    prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  447M
01KG03KDDNJFC2XTARX8AT5T7V  2025-12-04T00:00:00Z  2025-12-08T00:00:00Z  96h0m0s         47,049,536   70,621,349,313   630,742,158    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  173M
01KG07BHKR75QNETQ8DBXFB42V  2025-12-04T00:00:00Z  2025-12-08T00:00:00Z  96h0m0s         46,978,205   70,622,044,092   630,612,719    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  172M
01KG1H46VCAHWBYTGYVSGAEX6B  2025-12-06T00:00:00Z  2025-12-18T00:00:00Z  287h59m59.997s  142,831,341  147,496,787,952  1,367,753,616  prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  194M
01KG1SKJAN5N6H7F86FJXKN2V1  2025-12-06T00:00:00Z  2025-12-18T00:00:00Z  287h59m59.999s  142,834,015  147,496,352,419  1,367,744,685  prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  194M
01KFVX7BBYDBFZ9SV8X7TMMHZC  2025-12-08T00:00:00Z  2025-12-10T00:00:00Z  48h0m0s         56,997,745   35,303,471,208   342,635,513    prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  84M
01KFW016M7NYWFES5129BEH7SG  2025-12-08T00:00:00Z  2025-12-10T00:00:00Z  48h0m0s         56,866,343   35,304,259,769   342,514,620    prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  83M
01KG0BMH17TEHS7WVN6PS5PXFA  2025-12-10T00:00:00Z  2025-12-18T00:00:00Z  192h0m0s        129,985,551  144,586,043,251  1,338,392,948  prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  205M
01KG0KHK868A1B9Q7Y1JEDYBQ3  2025-12-10T00:00:00Z  2025-12-18T00:00:00Z  192h0m0s        130,415,204  144,588,634,601  1,338,835,464  prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  206M
01KG21ZFG6JR6DW2TGT2FSJHFS  2025-12-18T00:00:00Z  2026-01-01T00:00:00Z  336h0m0s        39,817,829   175,074,601,293  1,494,669,098  prometheus=k8s,replica=c,site=eqiad  0s          bucket.rewrite  60M
01KG29MPSF11SP7GFNKZ6M3A7J  2025-12-18T00:00:00Z  2026-01-01T00:00:00Z  336h0m0s        39,837,937   175,073,419,993  1,494,821,512  prometheus=k8s,replica=d,site=eqiad  0s          bucket.rewrite  60M
01KG2J9W9HKATZ33MSAWF0PSAW  2025-12-18T00:00:00Z  2026-01-01T00:00:00Z  336h0m0s        66,788,314   252,511,899,091  2,185,075,350  prometheus=k8s,replica=c,site=codfw  0s          bucket.rewrite  164M
01KG2YNG06B8A4N14KTZ7RBW8F  2025-12-18T00:00:00Z  2026-01-01T00:00:00Z  335h59m59.999s  66,674,964   252,510,409,257  2,184,975,822  prometheus=k8s,replica=d,site=codfw  0s          bucket.rewrite  164M

Analyzing some blocks with promtool seems to show that many pod restarts occurred, causing churn.

Running scope as unit: run-r663a50d26bca417ca2a3f232eda5db7b.scope                                                                                                                                                                                  │Running scope as unit: run-re24decd34ee44ffeb5804413a421ed24.scope
Block ID: 01KEGT6XMXRW4Q3FGZWEWWVBXV                                                                                                                                                                                                                │Block ID: 01KFR3RCAAWQM7X6G4RQS179HN
Duration: 48h0m0s                                                                                                                                                                                                                                   │Duration: 48h0m0s
Total Series: 62039696                                                                                                                                                                                                                              │Total Series: 79038436
Label names: 457                                                                                                                                                                                                                                    │Label names: 454
Postings (unique label pairs): 690755                                                                                                                                                                                                               │Postings (unique label pairs): 849953
Postings entries (total label pairs): 851391002                                                                                                                                                                                                     │Postings entries (total label pairs): 1060910857

Label pairs most involved in churning:                                                                                                                                                                                                              │Label pairs most involved in churning:
35567258 app=mediawiki                                                                                                                                                                                                                              │46172132 app=mediawiki
35450772 routed_via=main                                                                                                                                                                                                                            │42753880 job=k8s-pods-tls
34333033 release=main                                                                                                                                                                                                                               │36388313 routed_via=main
32793138 job=k8s-pods-tls                                                                                                                                                                                                                           │28814727 release=main
13583806 kubernetes_namespace=mw-web                                                                                                                                                                                                                │19565213 kubernetes_namespace=mw-web

Label names most involved in churning:                                                                                                                                                                                                              │Label names most involved in churning:
49291033 __name__                                                                                                                                                                                                                                   │67119353 __name__
49290817 job                                                                                                                                                                                                                                        │67118919 instance
49290817 instance                                                                                                                                                                                                                                   │67118919 job
39109712 kubernetes_namespace                                                                                                                                                                                                                       │52849997 kubernetes_namespace
39109598 kubernetes_pod_name                                                                                                                                                                                                                        │52849836 kubernetes_pod_name
39088588 pod_template_hash                                                                                                                                                                                                                          │52817847 pod_template_hash
36337638 app                                                                                                                                                                                                                                        │49358079 app
36330781 release                                                                                                                                                                                                                                    │49343034 release
36328610 routed_via                                                                                                                                                                                                                                 │49341373 routed_via
35567188 deployment                                                                                                                                                                                                                                 │46172167 deployment
18794113 envoy_cluster_name                                                                                                                                                                                                                         │24533024 envoy_cluster_name
17138476 le                                                                                                                                                                                                                                         │22760337 le
13902130 envoy_http_conn_manager_prefix                                                                                                                                                                                                             │18075877 envoy_http_conn_manager_prefix
12783539 namespace                                                                                                                                                                                                                                  │17510781 namespace
12564464 pod                                                                                                                                                                                                                                        │17305146 pod

Label names with highest cumulative label value length:                                                                                                                                                                                             │Label names with highest cumulative label value length:
14316379 container_id                                                                                                                                                                                                                               │18458440 container_id
11793998 name                                                                                                                                                                                                                                       │16045441 name
1368997 pod                                                                                                                                                                                                                                         │1807954 version
1341878 version                                                                                                                                                                                                                                     │1684491 pod
1289448 uid                                                                                                                                                                                                                                         │1583352 uid
1101097 device                                                                                                                                                                                                                                      │1213709 kubernetes_pod_name
1046124 instance                                                                                                                                                                                                                                    │1110072 instance
891394 kubernetes_pod_name                                                                                                                                                                                                                          │868822 __name__
862763 __name__                                                                                                                                                                                                                                     │865352 device

Highest cardinality labels:                                                                                                                                                                                                                         │Highest cardinality labels:
187506 name                                                                                                                                                                                                                                         │253835 name
185927 container_id                                                                                                                                                                                                                                 │239720 container_id
56065 instance                                                                                                                                                                                                                                      │59492 instance
35820 pod                                                                                                                                                                                                                                           │43999 pod
35818 uid                                                                                                                                                                                                                                           │43982 uid
27335 interface                                                                                                                                                                                                                                     │35551 interface
24569 kubernetes_pod_name                                                                                                                                                                                                                           │32921 kubernetes_pod_name
22543 version                                                                                                                                                                                                                                       │30092 version
15951 pod_ip                                                                                                                                                                                                                                        │16054 pod_ip
14358 resourcename                                                                                                                                                                                                                                  │12382 resourcename
12069 owner_name                                                                                                                                                                                                                                    │12014 owner_name
11485 created_by_name                                                                                                                                                                                                                               │11431 created_by_name
10941 job_name                                                                                                                                                                                                                                      │10821 job_name
10431 device                                                                                                                                                                                                                                        │9772 __name__
9831 __name__                                                                                                                                                                                                                                       │8331 device

I don’t think it’s easy to graph without querying too much data, but I tried with a restricted set of series and it does seem to highlight the perturbation.
Temporarily, everything seems to start with the datacenter switchover (24-25-26/09/2025).

image.png (2,907×909 px, 169 KB)

image.png (2,265×1,148 px, 210 KB)

Stashbot added a comment.Jan 30 2026, 2:11 PM

Mentioned in SAL (#wikimedia-operations) [2026-01-30T14:11:23Z] <tappof> puppet enabled and services repooled on titan2001 T410152

gerritbot added a comment.Feb 2 2026, 3:57 PM

Change #1235829 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] thanos/compact: increase meta-fetch goroutines to fix compactor inactivity

https://gerrit.wikimedia.org/r/1235829

gerritbot added a project: Patch-For-Review.Feb 2 2026, 3:57 PM
gerritbot added a comment.Feb 3 2026, 7:41 AM

Change #1235829 merged by Tiziano Fogli:

[operations/puppet@production] thanos/compact: increase meta-fetch goroutines to fix compactor inactivity

https://gerrit.wikimedia.org/r/1235829

Maintenance_bot removed a project: Patch-For-Review.Feb 3 2026, 8:32 AM
gerritbot added a comment.Feb 5 2026, 10:00 AM

Change #1237156 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] thanos/compact: revert concurrency to 2

https://gerrit.wikimedia.org/r/1237156

gerritbot added a project: Patch-For-Review.Feb 5 2026, 10:00 AM

Change #1237156 merged by Tiziano Fogli:

[operations/puppet@production] thanos/compact: revert concurrency to 2

https://gerrit.wikimedia.org/r/1237156

Maintenance_bot removed a project: Patch-For-Review.Feb 5 2026, 10:31 AM
tappof added a subtask: T416745: ThanosCompactHalted: add series - symbol table size exceeds.Feb 9 2026, 2:03 PM
herron added a subtask: T417308: DiskSpace: Disk space titan2001:9100:/srv 2.71% free.Feb 12 2026, 6:55 PM
herron created subtask T417313: titan2001: expand ssd storage.Feb 12 2026, 6:58 PM
Clement_Goubert added subscribers: Mooeypoo, Clement_Goubert.EditedFeb 16 2026, 1:28 PM

The mediawiki_action_api_modules_latency_bucket metric is emitted directly from mediawiki through the mw-on-k8s statsd-prometheus-exporter in ApiBase.php

It was introduced by this commit

Tagging in @Mooeypoo for awareness this is currently being dropped, and maybe future discussion on how to reduce cardinality.

tappof added a subtask: T417742: ThanosCompactHalted: pre compaction overlap check - overlaps found while gathering blocks..Feb 18 2026, 9:52 AM
Jhancock.wm closed subtask T417313: titan2001: expand ssd storage as Declined.Feb 19 2026, 4:31 PM
Blake mentioned this in T418383: Investigate mw-on-k8s statsd-exporter RAM usage pattern.Mar 11 2026, 2:56 PM
Blake subscribed.
gerritbot added a comment.Mar 23 2026, 5:21 PM

Change #1259168 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] thanos/compact: increase meta-fetch goroutines to fix compactor inactivity

https://gerrit.wikimedia.org/r/1259168

gerritbot added a project: Patch-For-Review.Mar 23 2026, 5:21 PM
gerritbot added a comment.Mar 23 2026, 5:32 PM

Change #1259168 merged by Tiziano Fogli:

[operations/puppet@production] thanos/compact: increase meta-fetch goroutines to fix compactor inactivity

https://gerrit.wikimedia.org/r/1259168

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2026, 6:32 PM
hnowlan moved this task from FY2025/2026-Q3 to Backlog on the SRE Observability board.Wed, Sep 23, 2:58 PM
hnowlan edited projects, added: SRE Observability; removed: SRE Observability (FY2025/2026-Q3).
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits