mlitn opened https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/29

Alter scripts/*.py to write their outputs to parquets

mlitn opened https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/30

Draft: Consume parquets instead of static files

mlitn opened https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/39

Draft: Accept denylist as parquet

mlitn opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/583

Draft: Periodically invoke a bunch of section-topics scripts

mlitn opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/584

Draft: Periodically invoke a bunch of section-topics scripts

All merge requests reviewed, Airflow test runs now ongoing.

mlitn closed https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/30

Draft: Consume parquets instead of static files

mlitn closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/584

Draft: Update section-topics pipeline arguments

https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/29 reviewed, looks great to me.

Also done with https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/39 and https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/583.
@matthiasmullie the floor is back to you.

mfossati merged https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/29

Alter scripts/*.py to write their outputs to parquets

mfossati merged https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/39

Accept denylist as parquet

mfossati merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/583

New image suggestions data quality DAGs

Deployed. New DAGs will start next Monday 2024-09-23.
This ticket definitely needs monitoring.

I've tweaked this week's section topics DAG to let it generate the 2024-09-09 snapshot, so that image suggestions can run again after a (resolved) missing upstream dependency:

• manually generated section titles denylist with the latest available SEAL - python section_topics/scripts/gather_section_titles_denylist.py -a /user/analytics-platform-eng/structured-data/seal/alignments/2024-08-19 -o 2024-08-19
• bad parsing & HTML tables as per previous runs

The new section titles denylist takes as input SEAL alignments and is much more aggressive, thus significantly decreasing total section topics:

curr = spark.read.parquet('/user/analytics-platform-eng/structured-data/section_topics/2024-09-09')
prev = spark.read.parquet('/user/analytics-platform-eng/structured-data/section_topics/2024-08-19')
prev.count(), curr.count()

(258520059, 146648150)

For instance, ptwiki has an order of magnitude more section titles compared the old denylist, i.e., 1403 VS 172.

SLIS have also significantly decreased due to the denylist:

Update

• check bad parsing: the next run should have been on 2024-10-01, but for some reason the DAG didn't start, so I've manually triggered it. Now running and waiting for upstream dependencies (2024-09 wikitext)
• detect HTML tables: started on 2024-10-03, now waiting for upstream dependencies (20241001 HTML dumps)
• section titles denylist: started on 2024-09-30, successful run! 🎉
• all data pipelines successful!!! 🚀

I'll keep monitoring the new DAGs.

Update

• check bad parsing: 2024-09 wikitext is still missing, so the DAG run is likely to time out failed hotfixed, now running
• detect HTML tables: blocked, see this Slack thread

Update

• check bad parsing: successful run that generated the 2024-09 snapshot
• detect HTML tables: paused. Manually fed a frozen snapshot to avoid breaking image suggestions,

This ticket is currently blocked: discussion needed to prioritize T305688: Make HTML Dumps available in hadoop.