Context from other related work:

T417694: Perform a one-time clean up of retained data sets in event_sanitize

T431391: HDFS under 20% of free space

T381707: Low available space on Hadoop / HDFS

TL;DR

• The 0.5 PB/month rate is the Jul-to-Aug rate. It doesn’t appear to be the trend. The 6-month rate is 0.152 PB/month. The 23-month rate is 0.108 PB/month. Webrequest retention changes move the total up and down, but they add only 0.014 PB/month to the 6-month rate. Runway is 4 months at 0.5 PB/month. It is about 13 months at 0.152 PB/month.
• The growth is not spread out. Datasets that do not exist in Feb 2026 hold about 1,022 TB. This is 20% of the cluster. /wmf/data/wmf_mediawiki and /wmf/data/wmf_traffic add about 150 TB each month. They still grow fast.
• Webrequest and its derived datasets hold 2,290 TB of 5,197 TB. This is 44% of the cluster. A third copy of raw webrequest is new this year at /wmf/data/wmf_traffic/webrequest.
• User and team Hive databases add 468.8 TB. This is 50% of the net growth. T431391 asked engineers to clean these up. That did not happen.
• .Trash holds about 157 TB for no apparent value. The drop jobs do not use -skipTrash.
• We should rank paths by growth, not by size. T381707, T431391 and T417694 all ranked by size. This hides small datasets that grow fast.
• The first four actions are: set a retention policy for the four new datasets, fix the drop jobs, delete five stale paths, and build a growth dashboard.

Investigation findings

Method: we made a first overview with hdfs dfs -count. Then we used wmf.hdfs_usage, which holds weekly fsimage snapshots from 2022-10-16. All numbers below are on-disk bytes, which include replication. The queries filter type = 'DIRECTORY' and one single path_depth value. This prevents the double counting that T418780 corrected. The fsimage numbers agree with the -count overview.

1. The 0.5 PB/month rate is one month, not the trend

Cluster use does not grow at a constant rate. It moves up and down. The changes match the webrequest retention changes. /wmf/data/wmf/webrequest went from 1,212.9 TB to 711.1 TB in Dec 2025. It went from 1,615.7 TB to 848.7 TB in Jul 2026. Both are 180d to 90d reverts.

The 0.5 PB/month rate is correct for Jul to Aug. Webrequest is flat in that month, so the increase is new data.

Compare the second and third rows. They use the same window. Webrequest adds only 0.014 PB/month of the 0.152 PB/month. The retention changes move the level of the total up and down. They do not change the slope. The new datasets in section 2 cause the growth.

At 0.152 PB/month, 2 PB gives about 13 months. At 0.5 PB/month, it gives 4 months. The correct answer depends on whether the new datasets hold their present rate.

2. About 1 PB is new since February 2026

20 of the top 40 growth paths do not exist in Feb 2026. Together they hold about 1,022 TB. This is about 20% of the cluster.

Two of these still grow fast. /wmf/data/wmf_mediawiki goes 63.0, 110.7, 188.3 TB in Jun, Jul and Aug. /wmf/data/wmf_traffic goes 93.7, 177.7, 268.3 TB in the same months. Together they add about 150 TB each month. They explain most of the August increase.

/wmf/data/raw/google_search_console holds 67 TB. This is T420996. T431391 predicted about 85 TB for it.

3. Webrequest and its derived datasets use 44% of the cluster

The total is 2,290 TB of 5,197 TB. A third copy of raw webrequest is new this year at /wmf/data/wmf_traffic/webrequest. It holds 122.8 TB. The 550 TB of new derivations all arrived in 6 months. We must decide if all of these datasets need their present retention.

4. User and team Hive databases give 50% of the net growth

The total is +468.8 TB. The net cluster growth is +941 TB. T431391 asked engineers to clean up these directories. That did not happen.

5. Quick wins

• .Trash holds about 157 TB. /user/analytics-search is 99.96% trash. /user/analytics and /user/hdfs are both 99.7% trash. @EBernhardson reported the cause in T381707. The drop jobs move data to trash. Should we use -skipTrash?
• Two paths hold many files and no bytes. /wmf/data/wmf/webrequest_officewiki holds 1,350,215 files and 0.0 TB. /wmf/data/wmf/interlanguage holds 818,175 files and 0.0 TB. Both put load on the namenode for apparently no value?
• Three datasets do not change for 12 months. /wmf/data/wmf/mediarequests_to_backfill holds 5.2 TB. /wmf/data/wmf/pagecounts-all-sites holds 9.5 TB. /wmf/data/wmf/webrequest_dataloss_2021-12_2021_01 holds 0.7 TB.
• /wmf/data/event_sanitized grows again. T417694 reduced it from 34.7M to 20.8M files. It now holds 21.78M files. /wmf/data/event_sanitized/netflow added 627K files since February. T424904 tracks the netflow retention window.

6. Two results to note

/wmf/data/wmf/mediawiki went from 321.9 TB to 50.3 TB in Jan 2026. /wmf/data/wmf_content went from 424.1 TB to 114.8 TB between Feb and Apr 2026. Together these reclaimed about 580 TB.

7. Proposed next steps

1. Ask the owners of /wmf/data/wmf_mediawiki, /wmf/data/wmf_traffic, /wmf/data/research/webrequest_timeseries and /wmf/data/discovery/embeddings to set a retention policy.
2. Consider changing the drop jobs to use -skipTrash.
3. Consider deletion of the three frozen datasets and the two zero-byte file farms.
4. Build a dashboard on wmf.hdfs_usage. Rank paths by growth, not by size. T381707, T431391 and T417694 all ranked by size. This hides small datasets that grow fast.
5. Set a capacity review before we add new dataset? Seems like HDFS is at capacity.

Thanks for tackling this! I personally think it's not so bad for us to do some cleaning semi-regularly. Quick thoughts:

• Reduce the number of snapshots we retain for snapshot datasets? MWH has 7 snapshots (2026-02 to 2026-08 at the moment) and I'm not sure who if anyone uses old snapshots. You obviously want some runway (maybe 2-3 snapshots?) so one-off code isn't constantly breaking and there's something to compare to if oddities arise, but it seems like that would be a cheap way to regain a decent bit of storage? Though maybe I overestimate how much that would affect. Not sure if there are equivalent Iceberg settings that could be played around with too.
• On a related note, there are 2016-12_private and 2017-07_private partitions for a bunch of wmf_raw tables that I've long wondered about. I'm sure they take up very little space but seems like they could be deleted?
• If we have so many webrequest derivatives now, why store 90 days of it? Why not drop it to 30 days?

wmf_mediawiki.mediawiki_history_incremental_v1 holds 0.5 TB of live data but occupies about 211 TB on HDFS. The Iceberg snapshot window is 90 days. The table rewrites past rows every day, so the window keeps every file that those rewrites produced. The write rate is 0.848 TB/day logical, or 2.54 TB/day on-disk. This gives a write amplification of about 117 to 1.

No expiry has run yet. The table still holds its first snapshot from 2026-06-20. It reaches day 90 on 2026-09-18 and then plateaus near 229 TB on-disk.

As a fast fix, I will set the snapshot window to 45 days in the Airflow DAG config.

So we reclaim about 97 TB on-disk now. We also avoid a further 18 TB of growth before day 90. The total saving against the 229 TB plateau is about 115 TB on-disk, which is 2.2% of the cluster.

The table reaches its new steady state at once, because it is already older than 45 days. It then holds near 114 TB and stops growing.

Time travel supports our verification work at T425734: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel, so we keep the longer window for now. A move to merge-on-read would be ideal, but we deferred that work for time reasons. We should pick it up in T432025: Incremental MediaWiki History - v2.

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2642

table_maintenance: Reduce mediawiki_history_incremental_v1 retention from 90 to 45 days

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2642

table_maintenance: Reduce mediawiki_history_incremental_v1 retention from 90 to 45 days

@KCVelaga_WMF, wmf_traffic.api_requests_refined holds about 144.6 TB on-disk. It is the second largest new dataset in the cluster. We have HDFS storage pressure, so I looked at its Iceberg metadata.

Two settings in config/datasets.yaml line 1073 drive the size. The table is already at its steady state, so a change to either one gives space back at once.

The first change only shortens the time-travel reach. Do you use that at all? It deletes no data. We could even set it to 7 days and save even more?

For the second, the yaml comment says 180 days is temporary:

"need the output for downstream jobs for aggregate metrics; keeping 180 until then, we can switch back to 90 later".

Is this stale and we can change?

@xcollazo ah right, this slipped through. We can reduce the data delete from 180 to 90 days (standard retention for raw data), as all the aggregate backfills are complete, and set to the expire snapshots to 7 days - I don't see a need to keep it beyond that period. I will create an MR shortly.

kcvelaga updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2653

Update data retention for api_requests_refined

kcvelaga merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2653

Update data retention for api_requests_refined

Probably not very helpful, but just in case: I was looking through my files in HDFS and I noticed that there were what seemed like old temporary job files in the .staging, .sparkStaging, and .skein directories. Judging from the age of the subdirectories, most of them were from years ago.

I deleted them from my own user directory without saving the information, but I did copy down what was in the /user/analytics-product directory: P96542.

They were not taking up much space at all (300 MB for analytics-product), but maybe multiplied by all the users on the cluster it might be meaningful?