Investigation findings

We investigated two angles: the top offenders by file count and size (from the ticket description), and all schemas previously removed from the sanitization allowlist (identified via git log). This is not an exhaustive scan of the full namespace — there may be additional datasets worth reviewing. Findings are grouped by suggested action.

1. Orphaned datasets — suggest deleting

These schemas were removed from the sanitization allowlist in prior tasks but their HDFS data was never deleted. No further stakeholder input appears to be needed, as the decision to deprecate each one is already documented in the referenced tasks.

From T360579 (MobileWikiApp schemas, allowlist cleaned up Jun 2024):

36 schemas under /wmf/data/event_sanitized/mobilewikiapp* — all confirmed present on HDFS.

From other prior cleanup tasks:

Subtotal orphans: ~9.7M files, ~4,524 GB (~4.4 TB)

2. Delete data + remove from allowlist

These schemas are still listed in the allowlist but appear to no longer be written to. We suggest deleting the data and removing the allowlist entries.

Subtotal: ~1.7M files, ~2,200 GB

3. Needs stakeholder discussion — retention policy

These schemas are actively written to but accumulating data well beyond what most use cases likely require.

netflow — ~4.8M files, 9,921 GB total, actively written:

File count is growing ~2x every 2 years. When this was set up (T231339), the decision was to sanitize and retain indefinitely — but no retention limit was established. Even a 3-year rolling window could free ~996K files and ~3.5 TB (2020–2022). It may be worth revisiting this with the original owner (@mforns).

searchsatisfaction — ~2.3M files, 6,426 GB total, actively written, data goes back to 2021. Retention was explicitly requested in T274607 for analytics use. A long-term retention policy may be worth establishing — suggested contacts are @MNeisler (requester) and @mforns (owner).

4. Anomalies worth investigating separately

• referencepreviewspopups: year=2026 partition has 6,986 files but only 0.01 GB of data — near-empty files being written. Possibly a pipeline issue or a quiet deprecation.
• mediawiki_revision_score: 2024–2026 has ~59K files but only 0.09 GB total (near-empty). Also has an anomalous spike in 2019 (486K files vs ~25–74K for all other years). Something may be broken in the ingestion pipeline.

5. Hive tables

All event_sanitized datasets have associated Hive tables in the event_sanitized database. For each dataset we delete, we should also plan for:

• DROP TABLE event_sanitized.<schema> for fully removed schemas
• DROP PARTITION for schemas where we only trim older years (e.g. netflow if we agree on a rolling window)

This should be coordinated to avoid leaving stale table metadata after the HDFS data is gone.

Summary

The suggested immediate deletions represent ~33% of the current total file count (~11.4M of ~34.1M files), which would meaningfully reduce namenode pressure.

I'd be totally in favor of setting a long-time retention period for event_sanitized.
The overall plan looks great to me!

@SNowick_WMF and @phuedx:

You both seem to be the owners of the original work items that decommissioned some event_sanitizied datasets.

Some data was left behind. I would like to move forward with removing all that data. Thus want to double check with you both that we are good and that you do not need this data?:

In T417694#11799604, @xcollazo wrote:

...

1. Orphaned datasets — suggest deleting

These schemas were removed from the sanitization allowlist in prior tasks but their HDFS data was never deleted. No further stakeholder input appears to be needed, as the decision to deprecate each one is already documented in the referenced tasks.

From T360579 (MobileWikiApp schemas, allowlist cleaned up Jun 2024):

36 schemas under /wmf/data/event_sanitized/mobilewikiapp* — all confirmed present on HDFS.

Path (glob)	Files	Size
mobilewikiapp* (36 schemas)	~7.7M	~4,400 GB

From other prior cleanup tasks:

Dataset(s)	Files	Size	Original task
mediaviewer, multimediaviewer* (5 paths)	~505K	~55 GB	T310890
externalguidance	121,917	31.6 GB	T303508
changeslistfilters, changeslistfiltergrouping	242,240	26.7 GB	T317525
echointeraction, echomail	459,632	6.7 GB	T344167
twocolconflictconflict, twocolconflictexit	353,614	2.4 GB	T397611
editoractivation	214,360	1.1 GB	T330766
mediawiki_wikistories_consumption_event, mediawiki_wikistories_contribution_event	341,067	0.5 GB	T408178
flowreplies	101,354	0.3 GB	T309025
guidedtourinternallinkactivation	7,254	~0 GB	T288416

Subtotal orphans: ~9.7M files, ~4,524 GB (~4.4 TB)

...

Confirming we aren't using the data in any of the /wmf/data/event_sanitized/mobilewikiapp* schemas, signing off on data deletion.

As @xcollazo specified: no new data is being written to event_sanitized from the MobileWikiApp* schemas since Jun 2024. This is just pre-existing data that we suspect no one is using.

From T360579 (MobileWikiApp schemas, allowlist cleaned up Jun 2024):

36 schemas under /wmf/data/event_sanitized/mobilewikiapp* — all confirmed present on HDFS.

Path (glob) Files Size
mobilewikiapp* (36 schemas) ~7.7M ~4,400 GB

In T417694#11809583, @SNowick_WMF wrote:

Confirming we aren't using the data in any of the /wmf/data/event_sanitized/mobilewikiapp* schemas, signing off on data deletion.

Thanks @SNowick_WMF.

Its Friday, so I will refrain from deleting data today. Sometime next week we will go ahead and apply DROP TABLEs (P90374) and hdfs dfs -rm -r (P90375) for all of these mobilewikiapp* assets.

Re. the tables in the "From other prior cleanup tasks" table:

General

IIRC I tidied those instruments up about 4 years ago, including removing sanitization rules, i.e. those tables have been written to in years. If someone were using the data in those tables, then it would be woefully out of date.

When you delete these tables, could you please also delete their counterparts in the event database. There are a couple of tasks about removing these tables that never seem to be prioritized:

• T311229: Drop MediaViewer and MultimediaViewer* tables
• T304714: Drop sanitized ExternalGuidance data
• T317942: Drop event.changeslistfiltergrouping table

Specific

@xcollazo

searchsatisfaction — ~2.3M files, 6,426 GB total, actively written, data goes back to 2021. Retention was explicitly requested in T274607 for analytics use. A long-term retention policy may be worth establishing

I no longer work regularly with this dataset. From the product analytics side, I believe it would be fine to delete some of the older data; however, I'd recommend checking with the Search Platform team to confirm. Maybe @EBernhardson or @dcausse?

In T417694#11815172, @MNeisler wrote:

@xcollazo

searchsatisfaction — ~2.3M files, 6,426 GB total, actively written, data goes back to 2021. Retention was explicitly requested in T274607 for analytics use. A long-term retention policy may be worth establishing

I no longer work regularly with this dataset. From the product analytics side, I believe it would be fine to delete some of the older data; however, I'd recommend checking with the Search Platform team to confirm. Maybe @EBernhardson or @dcausse?

Thanks for the ping, personally I do not use the searchsatisfaction table from the event_sanitized database and I'd fine dropping/cleaning it up unless @EBernhardson needs it.

I don't believe I've used the event_santized tables either. We do use some of the data beyond 90 days, but that's in a separate rollup table. It should be safe, afaik, to drop searchsatisfaction from the event_sanitized database.

In T417694#11809614, @xcollazo wrote:

In T417694#11809583, @SNowick_WMF wrote:

Confirming we aren't using the data in any of the /wmf/data/event_sanitized/mobilewikiapp* schemas, signing off on data deletion.

Thanks @SNowick_WMF.

Its Friday, so I will refrain from deleting data today. Sometime next week we will go ahead and apply DROP TABLEs (P90374) and hdfs dfs -rm -r (P90375) for all of these mobilewikiapp* assets.

Attempted to run the DROP TABLEs from P90374, but all 36 tables timed out — each has 56K+ partitions and the metastore cannot handle dropping them all in a single transaction.

Workaround: Drop partitions in batches before the final DROP TABLE. Since the tables are partitioned by year/month/day/hour, we are dropping one month at a time per table (36 statements per month-block), then the final DROP TABLE presumably will run instantly with an empty table.

• Year 2017 succeeded dropping by full year (~984 partitions/table, partial year starting Nov 2017).
• Year 2018 failed by full year (~8,760 partitions/table), succeeded month-by-month (~730 partitions/table per statement, ~6s each).
• Years 2019-2024 are currently running month-by-month, ETA a few hours.

Will follow up once partition drops are complete and final DROP TABLEs have been executed.

All these new DROPs documented at P91234.

In T417694#11841263, @xcollazo wrote:

In T417694#11809614, @xcollazo wrote:

In T417694#11809583, @SNowick_WMF wrote:

Confirming we aren't using the data in any of the /wmf/data/event_sanitized/mobilewikiapp* schemas, signing off on data deletion.

Thanks @SNowick_WMF.

Its Friday, so I will refrain from deleting data today. Sometime next week we will go ahead and apply DROP TABLEs (P90374) and hdfs dfs -rm -r (P90375) for all of these mobilewikiapp* assets.

Attempted to run the DROP TABLEs from P90374, but all 36 tables timed out — each has 56K+ partitions and the metastore cannot handle dropping them all in a single transaction.

Workaround: Drop partitions in batches before the final DROP TABLE. Since the tables are partitioned by year/month/day/hour, we are dropping one month at a time per table (36 statements per month-block), then the final DROP TABLE presumably will run instantly with an empty table.

• Year 2017 succeeded dropping by full year (~984 partitions/table, partial year starting Nov 2017).
• Year 2018 failed by full year (~8,760 partitions/table), succeeded month-by-month (~730 partitions/table per statement, ~6s each).
• Years 2019-2024 are currently running month-by-month, ETA a few hours.

Will follow up once partition drops are complete and final DROP TABLEs have been executed.

All these new DROPs documented at P91234.

All DROPs from P91234 successful. All hdfs dfs -rm -r from P90375 successful. 🎉

Now removing all other "Orphaned datasets (removed from allowlist, HDFS data never deleted)" with the following scripts:

P91293: orphaned_hdfs_rm.sh
P91294: orphaned_drop_partitions.hql
P91295: orphaned_drop_tables.hql

In T417694#11844788, @xcollazo wrote:

Now removing all other "Orphaned datasets (removed from allowlist, HDFS data never deleted)" with the following scripts:

P91293: orphaned_hdfs_rm.sh
P91294: orphaned_drop_partitions.hql
P91295: orphaned_drop_tables.hql

All done. 🎉

In T417694#11813575, @phuedx wrote:

...
When you delete these tables, could you please also delete their counterparts in the event database. There are a couple of tasks about removing these tables that never seem to be prioritized:

• T311229: Drop MediaViewer and MultimediaViewer* tables
• T304714: Drop sanitized ExternalGuidance data
• T317942: Drop event.changeslistfiltergrouping table

...

Done.

Moving on to next set:

@mforns, @MNeisler, ok if we delete these?

Change #1275982 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery@master] Remove DesktopWebUIActionsTracking, MobileWebUIActionsTracking, ReadingDepth from sanitization allowlist

https://gerrit.wikimedia.org/r/1275982

@awight, @Ottomata, ok if we delete these?

Change #1276699 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[operations/mediawiki-config@master] Remove stream 'mediawiki.dump.revision_content_history.reconcile.rc0'

https://gerrit.wikimedia.org/r/1276699

Change #1276699 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove stream 'mediawiki.dump.revision_content_history.reconcile.rc0'

https://gerrit.wikimedia.org/r/1276699

Mentioned in SAL (#wikimedia-operations) [2026-04-23T19:06:32Z] <otto@deploy1003> Started scap sync-world: Backport for [[gerrit:1276699|Remove stream 'mediawiki.dump.revision_content_history.reconcile.rc0' (T417694)]], [[gerrit:1276397|EventStreamConfig - add rc0 streams for html and feature count change (T423920)]]

Mentioned in SAL (#wikimedia-operations) [2026-04-23T19:14:07Z] <otto@deploy1003> xcollazo, otto: Backport for [[gerrit:1276699|Remove stream 'mediawiki.dump.revision_content_history.reconcile.rc0' (T417694)]], [[gerrit:1276397|EventStreamConfig - add rc0 streams for html and feature count change (T423920)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

@mforns, @MNeisler, ok if we delete these?

OK by me!

mediawiki_revision_score

Yes I think we can delete!

Mentioned in SAL (#wikimedia-operations) [2026-04-23T19:28:37Z] <otto@deploy1003> Finished scap sync-world: Backport for [[gerrit:1276699|Remove stream 'mediawiki.dump.revision_content_history.reconcile.rc0' (T417694)]], [[gerrit:1276397|EventStreamConfig - add rc0 streams for html and feature count change (T423920)]] (duration: 22m 05s)

@mforns, @MNeisler, ok if we delete these?

Yes, mobilewebuiactionstracking and desktopwebuiactionstracking can be deleted.

In T417694#11846081, @xcollazo wrote:

Moving on to next set:

Schema	Files	Size	Last Write	Introduced by	Date	Commit	Task	Notes
readingdepth	~388K	~1,729 GB	year=2022	@mforns	2018-06-26	61caff71	T193176	Superseded by mediawiki_reading_depth (EP, active as of Jan 2026). Remove ReadingDepth from allowlist.
mobilewebuiactionstracking	~827K	~300 GB	year=2024 (Oct)	@MNeisler	2019-10-09	67b471cf	T234563	Stopped simultaneously with desktop sibling — likely intentional coordinated deprecation. No EP successor in allowlist.
desktopwebuiactionstracking	~450K	~172 GB	year=2024 (Oct)	@MNeisler	2020-10-04	5ebe8b41	T263143	Same as above.

@mforns, @MNeisler, ok if we delete these?

DROPs of partitions and tables on P91812.

hdfs dfs -rm -rs are quite short:

#\!/bin/bash
# HDFS deletes for allowlist-removed event_sanitized datasets.
# Safe to run in parallel with partition drops (external tables).

hdfs dfs -rm -r /wmf/data/event_sanitized/readingdepth
hdfs dfs -rm -r /wmf/data/event_sanitized/mobilewebuiactionstracking
hdfs dfs -rm -r /wmf/data/event_sanitized/desktopwebuiactionstracking

All ran successfully.

Change #1275982 merged by Xcollazo:

[analytics/refinery@master] Remove DesktopWebUIActionsTracking, MobileWebUIActionsTracking, ReadingDepth from sanitization allowlist

https://gerrit.wikimedia.org/r/1275982

In T417694#11852772, @Ottomata wrote:

mediawiki_revision_score

Yes I think we can delete!

DROPs for mediawiki_revision_score at P91813.

And a single hdfs dfs -rm:

hdfs dfs -rm -r /wmf/data/event_sanitized/mediawiki_revision_score

All ran successfully.

Change #1278746 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery@master] Remove mediawiki_revision_score from sanitization main allowlist

https://gerrit.wikimedia.org/r/1278746

Please don't delete referencepreviewspopups in this round! We still need the data, although we might be able to clean older history once we figure out exactly what is still relevant.

The dead stream is concerning, I can't remember if we might have done that on purpose or not, but my team will either restore or reimplement soon.

Change #1278746 merged by Xcollazo:

[analytics/refinery@master] Remove mediawiki_revision_score from sanitization main allowlist

https://gerrit.wikimedia.org/r/1278746

In T417694#11815172, @MNeisler wrote:

@xcollazo

searchsatisfaction — ~2.3M files, 6,426 GB total, actively written, data goes back to 2021. Retention was explicitly requested in T274607 for analytics use. A long-term retention policy may be worth establishing

I no longer work regularly with this dataset. From the product analytics side, I believe it would be fine to delete some of the older data; however, I'd recommend checking with the Search Platform team to confirm. Maybe @EBernhardson or @dcausse?

In T417694#11816374, @dcausse wrote:

...
Thanks for the ping, personally I do not use the searchsatisfaction table from the event_sanitized database and I'd fine dropping/cleaning it up unless @EBernhardson needs it.

In T417694#11838808, @EBernhardson wrote:

I don't believe I've used the event_santized tables either. We do use some of the data beyond 90 days, but that's in a separate rollup table. It should be safe, afaik, to drop searchsatisfaction from the event_sanitized database.

Just double checking here:

I think what @dcausse and @EBernhardson are saying is that they are ok with dropping ALL data and removing searchsatisfaction from event_sanitized.

@MNeisler, since you no longer work regularly with this either, would you be ok if I do delete it instead of instituting a rolling window? Or are you saying that even though sporadic, you do still depend on it, and you would rather have the rolling window? If rolling window, what would work for you? 1, 2, 3 years?

@MNeisler, since you no longer work regularly with this either, would you be ok if I do delete it instead of instituting a rolling window?

It's fine by me if you delete it. Thank you for checking @xcollazo

In T417694#11873025, @MNeisler wrote:

@MNeisler, since you no longer work regularly with this either, would you be ok if I do delete it instead of instituting a rolling window?

It's fine by me if you delete it. Thank you for checking @xcollazo

Got it. For completeness, do note that even though event_sanitized.searchsatisfaction will no longer be accessible, you will continue to have the regular 90d window at the non-sanitized event.searchsatisfaction.

Change #1279483 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery@master] Remove SearchSatisfaction from sanitization analytics allowlist

https://gerrit.wikimedia.org/r/1279483

searchsatisfaction — ~2.3M files, 6,426 GB total, actively written, data goes back to 2021. Retention was explicitly requested in T274607 for analytics use.

DROPs at P91991.

And a single hdfs dfs -rm:

hdfs dfs -rm -r /wmf/data/event_sanitized/searchsatisfaction

All ran successfully.

Change #1279483 merged by Xcollazo:

[analytics/refinery@master] Remove SearchSatisfaction from sanitization analytics allowlist

https://gerrit.wikimedia.org/r/1279483

Moving the netflow retention discussion to a dedicated task: T424904: Set up a rolling retention window for event_sanitized.netflow.

All cleanup work completed. In total, this task removed approximately 15M files and ~13.5 TB of data from event_sanitized:

• ~10M files / ~4.5 TB from orphaned datasets (schemas previously removed from the allowlist but never deleted from HDFS)
• ~5M files / ~9 TB from datasets still in the allowlist that were no longer being written to

Retention policy for netflow (the remaining top offender) is being tracked separately in T424904.