In T417694#11809583, @SNowick_WMF wrote:

Confirming we aren't using the data in any of the /wmf/data/event_sanitized/mobilewikiapp* schemas, signing off on data deletion.

I think we implemented this via T373693. @KCVelaga_WMF can you confirm?

Reopening and tagging Data-Engineering since we do still need to make the pipeline changes that @JAllemandou suggests:

@SNowick_WMF and @phuedx:

We investigated two angles: the top offenders by file count and size (from the ticket description), and all schemas previously removed from the sanitization allowlist (identified via git log). This is not an exhaustive scan of the full namespace — there may be additional datasets worth reviewing. Findings are grouped by suggested action.

In T401296#11798862, @CodeReviewBot wrote:

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2080

Refactor WikibaseDump to use OOP pattern and add output_dir support

Boldly closing this as out of date. Other tickets like T419055 have better context.

CC @BTullis

I concur that this looks like a bug.

Ok confirming this particular issue seems to be solved now. Note how overall inconsistencies are way down:

I confirm that the dump is being produced and exists on phab1004.

I am now of the opinion that we should just sunset these metrics:

Noting that this has only happened once so far.

1. T420787 — MWCH Data Quality Dashboard: Summary

Note further that there seems to be other streams with ValidationErrors over last 30 days, but I did not dig deeper than the following query:

More serious EventBus error rate: T421257: EventBus: Unable to deliver all events: 503: Service Unavailable.

One (redacted) example from logstash:

Minor bug on EventGate found: T421237: `mediawiki.page_change.v1`: two schema validation errors causing events to be silently dropped by EventGate.

In T420787#11735822, @Ottomata wrote:

If there is a way to relate this to missing events in the event sources, I would be much obliged! Xabriel came up with a nice query for this but I haven't had time to explore it.

Folks often ask me "but how lossy are the events?". A visual chart that answers this would be very helpful!

In T266374#11556983, @JAllemandou wrote:

Flagging for @Ahoelzl : This could be something we wish to consider.

(We are now waiting on the next monthly reconcile to happen to see where we are at.)

In T418804#11743801, @tchin wrote:

I got the same error running an airflow devenv while developing a Spark 3.3.2 DAG.

Ivy Default Cache set to: /tmp/runuser/ivy_spark3/cache
The jars for the packages stored in: /tmp/runuser/ivy_spark3/home/jars
org.apache.iceberg#iceberg-spark-runtime-3.3_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-6700c2db-77a4-421a-864c-fcab30a038e7;1.0
	confs: [default]
	found org.apache.iceberg#iceberg-spark-runtime-3.3_2.12;1.6.1 in mirrored
Exception in thread "main" java.io.FileNotFoundException: /tmp/runuser/ivy_spark3/cache/resolved-org.apache.spark-spark-submit-parent-6700c2db-77a4-421a-864c-fcab30a038e7-1.0.xml (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)

The option of telling the few consumers of the rsync to change the hostname/IP they're using to mirror us (to separate it from HTTP via the CDN) would be far simpler and avoid the need for such a proxy.

1. Make local development compatible with git worktrees

In T401296#11703564, @CodeReviewBot wrote:

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2080

Refactor WikibaseDump to use OOP pattern and add output_dir support

In T419055#11729676, @xcollazo wrote:

spark_process_reconciliation_events ingest for 2026-03-16 failed multiple times. I suspect this happened due to the same cluster issues reported on T419291#11723404: T420168 and T415002.

Rerunning as is now via https://yarn.wikimedia.org/proxy/application_1773845446826_10194

File export successful. All files now exposed publicly at https://dumps.wikimedia.org/other/mediawiki_content_history/enwiki/2026-03-01/xml/bzip2/.

spark_process_reconciliation_events ingest for 2026-03-16 failed multiple times. I suspect this happened due to the same cluster issues reported on T419291#11723404: T420168 and T415002.

In T411116#11726800, @JAllemandou wrote:

I think it's easier to make it happen this way (reducing the mapper weight in sqoop script) than changing puppet. If ok for everyone, let's make it happen (with a comment in the code :) )

For completeness, a link to the dashboard that was fixed: https://superset.wikimedia.org/superset/dashboard/409

Opened T420582: Migrate Airflow Search instance code away from deprecated VariableProperties to tackle that last bit of VariableProperties dependency.

All usage of VariableProperties from within Airflow DAGs have now been migrated to DagProperties.

In T419291#11723404, @xcollazo wrote:

Failed again. There was cluster instability due to T420168 and T415002 so retrying as is.

https://yarn.wikimedia.org/proxy/application_1773845446826_0041

In T411116#11723883, @JAllemandou wrote:

What if we spilt by another column like lu_local_id?

I don't think that using lu_local_id is a good idea because ut's not a table index, making splitting the data a lot less efficient.

My way of dealing with that would be to change the puppet code to using 32 mappers.

In T411116#11723763, @Snwachukwu wrote:

I can also see that that we are doing a split-by on a String column when sqooping the centralauth_localuser table so it does makes sense. What if we spilt by another column like lu_local_id?
https://github.com/wikimedia/analytics-refinery/blob/35e7f416fe4bc9e3aeb194474a4fa803d8983823/python/refinery/sqoop.py#L1355

In T411116#11723611, @JAllemandou wrote:

I've read our code and the bug report again, there is something I don't understand: the bug is supposed to happen when splitting a table on String type field, but we split on a Long type field:
https://github.com/wikimedia/analytics-refinery/blob/master/python/refinery/sqoop.py#L1320
I'd really like for us to investigate more. let's sync @APizzata-WMF .

Failed again. There was cluster instability due to T420168 and T415002 so retrying as is.

Rerunning via https://yarn.wikimedia.org/proxy/application_1764064841637_2255221/jobs/

Ran the following as a one off:

$ hostname -f
an-launcher1003.eqiad.wmnet
$ whoami
analytics

In T419291#11720295, @CodeReviewBot wrote:

xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2096

mw_content_xml_export: fix enwiki timeout and clear output folder on retry

This continues to be an issue:

Context: this dashboard was put together as part of T381707: Low available space on Hadoop / HDFS.

Being bold and reverting changes to mw-page-content-change-enrich to avoid inadvertently repeating T408918#11715866.