For search-platform i reviewed the directories, we essentially have the following data. Data sizes are per hdfs dfs -du and should include replication:

• discovery.cirrus_index is 15TB. This keeps 4 historical imports, it could be reduced to 3 or maybe even 2.
• discovery.cirrus_index_without_content is also 15TB, apparently drop_mediawiki_snapshots.py (from discolytics) is not properly dropping old data. We should be able to fix this and save 14TB.
• 25T of wikidata dumps.
• 25T in /user/analytics-search/.Trash. I suspect the way we cleanup historical data, maybe via drop_mediawiki_snapshots.py or refinery-drop-older-than is moving data into the trash rather than actually deleting. 30 days of deleted data apparently adds up to 25TB.
• 7TB in everything else.

Not sure what to do with these, looking into the snapshot's not being dropped and the excessive Trash seems potentially worthwhile.

The dumps 2.0 intermediate table is currently hoarding these many bytes:

xcollazo@stat1011:~$ hdfs dfs -count /wmf/data/wmf_dumps/wikitext_raw_rc2/*
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
      194131      2303606    151151869470330 /wmf/data/wmf_dumps/wikitext_raw_rc2/data
           1        10604        44165993108 /wmf/data/wmf_dumps/wikitext_raw_rc2/metadata

So ~137TBs of data.

Considering we need the space, let's delete all snapshots older than 7 days, even though we typically want to have the last 90 days here.

ssh an-launcher1002.eqiad.wmnet
sudo -u analytics bash

kerberos-run-command analytics spark3-sql \
--driver-cores 8 \
--master yarn \
--conf spark.dynamicAllocation.maxExecutors=128 \
--conf spark.executor.memoryOverhead=3072  \
--executor-memory 8G \
--executor-cores 2 \
--driver-memory 32G


spark-sql (default)> SELECT NOW() - INTERVAL 7 DAYS;
CAST(now() - INTERVAL '7 days' AS TIMESTAMP)
2024-12-02 20:35:30.395
Time taken: 0.974 seconds, Fetched 1 row(s)



spark-sql (default)> CALL spark_catalog.system.expire_snapshots(
                   >   table => 'wmf_dumps.wikitext_raw_rc2',
                   >   older_than => TIMESTAMP '2024-12-02 20:35:30.395',
                   >   max_concurrent_deletes => 50,
                   >   stream_results => true
                   > );
24/12/09 20:38:05 WARN NioEventLoop: Selector.select() returned prematurely 512 times in a row; rebuilding Selector io.netty.channel.nio.SelectedSelectionKeySetSelector@106e6234.
deleted_data_files_count	deleted_manifest_files_count	deleted_manifest_lists_count
2046893	8860	1319
Time taken: 737.888 seconds, Fetched 1 row(s)

And we now have:

xcollazo@stat1011:~$ hdfs dfs -count /wmf/data/wmf_dumps/wikitext_raw_rc2/*
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
      194131       256713     29068054308835 /wmf/data/wmf_dumps/wikitext_raw_rc2/data
           1          425         1673207239 /wmf/data/wmf_dumps/wikitext_raw_rc2/metadata

That's ~26TB, thus we saved a nice ~111TBs. Happy holidays! :D

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/941 has been merged and should also reduce storage consumption (see T379024)

The current rate of capacity consumption seems to be around 10% / month since October 1. If this stays stable, we'll be below 10% of capacity before we are fully back from the end of year holiday. This seems to close to the limit to be comfortable.

Note: While the unique devices backfill and the Dumps 2 renaming is in progress before the holiday break, expect a higher capacity consumption rate.
Both efforts will most likely by 12/18, latest 12/20 release the extra consumed data.

As part of the DPE SRE / DE sync up meeting the following mid-term solutions could be considered:

• Dumps 2
  • Short term option: retain only 30+ days for initial adaptation phase (although 120 is speced / communicated)
• Data storage efficiencies
  • Change snappy to gzip or other compression for less frequently accessed data (re-computation efforts unclear yet)
  • Reduce replication factor on older data
• Leverage Ceph capacity with Iceberg tables
  • Requires performance testing
  • Rack proximity might be a problem … data locality / network topology
  • Ceph space available: 400-500TB (non replicated)
• Park cold storage data on Ceph
  • To be investigated … no dedicated cold storage space
  • Note: Newer versions Hadoop support hot / cold concept (within HDFS)
• Repurpose retirement nodes into Hadoop nodes
  • 1.5TB x 80 = 7%
  • Next Hadoop workers will have double the space (rolling refresh cycle)
• Make a tactical investment in hard drive
  • 100 Hadoop workers, swap out 1000 8TB drive over course of several months

We're still using up the available space much too quickly to survive the holiday. Currently at 12% and we're consuming space at about 1.5% per day.

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/980

Temporarily delete snapshots at 7 day mark for wikitext_raw table.

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/980

Temporarily delete snapshots at 7 day mark for wikitext_raw table.

We have now added about 4% to the total capacity by adding the old an-presto100[1-5] servers to the cluster.

See {T382372} for the discussion with DC-Ops about a mass hard drive upgrade for 70 active Hadoop worker nodes.
We are considering the relative costs of upgrading 840 drives to either 8TB or 16TB.

Reclaiming additionally retained webrequest data got us back to 28% free disk space.

Closing this task as we've been able to reduce usage sufficiently (reducing the retention of web requests to the normal 90 days). We still need a longer term strategy around HDFS storage.