This has happened sporadically multiple times on job for spark_create_intermediate_table, which runs this code backfill_create_intermediate_table.py.
I believe a quick fix similar to what we did in T340863#9397991 will fix this issue. TL;DR: Make the temporary table also be ALTER TABLE tmp.tmp_table WRITE ORDERED BY wiki_db, revision_timestamp.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Milimetric | T330296 Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark | |||
| Resolved | xcollazo | T357143 wmf_dumps.wikitext_raw_rc2 backfill failing with FetchFailedException |
xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/merge_requests/25
Use WRITE ORDERED BY in backfill_create_intermediate_table.py.
xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/merge_requests/25
Use WRITE ORDERED BY in backfill_create_intermediate_table.py.
Released via https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/packages/582.
Currently slated to be tested on the next backfill run.
Spark job: https://yarn.wikimedia.org/cluster/app/application_1707226456123_87407. Finished successfully in ~17 hours. Some evindence of retries, but overall the best time so far.
Will wait until the full backfill is done to do a full comparison.
The spark_create_intermediate_table job did not show evidence of FetchFailedExceptions. There were retries, but they were unrelated. It also finished ~21% faster as shown below.
The remaining spark tasks were comparable to last successful run. In general, the whole backfill process finished 9% faster. We are good here.
| Task | Before (2024-01-01) | After (2024-02-01) | Diff factor | Notes |
|---|---|---|---|---|
| wait_for_data_in_mw_wikitext_history | 0:00:04 | 0:00:02 | 0.5 | |
| wait_for_data_in_raw_mediawiki_revision | 0:00:03 | 0:00:02 | 0.6666666667 | |
| spark_create_intermediate_table | 21:32:33 | 16:58:52 | 0.7882609312 | |
| spark_backfill_merge_into_2001 | 0:03:08 | 0:03:19 | 1.058510638 | |
| spark_backfill_merge_into_2002 | 0:03:40 | 0:04:27 | 1.213636364 | |
| spark_backfill_merge_into_2003 | 0:03:39 | 0:04:44 | 1.296803653 | |
| spark_backfill_merge_into_2004 | 0:07:08 | 0:05:58 | 0.8364485981 | |
| spark_backfill_merge_into_2005 | 0:17:45 | 0:16:45 | 0.9436619718 | |
| spark_backfill_merge_into_2006 | 0:59:55 | 0:46:36 | 0.7777468707 | |
| spark_backfill_merge_into_2007 | 1:38:11 | 1:09:18 | 0.705822441 | |
| spark_backfill_merge_into_2008 | 1:38:40 | 1:26:29 | 0.8765202703 | |
| spark_backfill_merge_into_2009 | 1:30:53 | 1:33:50 | 1.032459197 | |
| spark_backfill_merge_into_2010 | 1:41:07 | 1:24:53 | 0.8394593704 | |
| spark_backfill_merge_into_2011 | 1:36:24 | 1:24:17 | 0.8743084371 | |
| spark_backfill_merge_into_2012 | 1:38:54 | 1:35:26 | 0.9649477587 | |
| spark_backfill_merge_into_2013 | 1:46:36 | 1:40:52 | 0.9462163852 | |
| spark_backfill_merge_into_2014 | 1:41:41 | 1:40:59 | 0.9931158826 | |
| spark_backfill_merge_into_2015 | 1:54:56 | 1:37:21 | 0.847012761 | |
| spark_backfill_merge_into_2016 | 2:07:12 | 2:06:14 | 0.9924004193 | |
| spark_backfill_merge_into_2017 | 2:10:11 | 2:13:32 | 1.025732941 | |
| spark_backfill_merge_into_2018 | 2:36:20 | 2:54:49 | 1.118230277 | |
| spark_backfill_merge_into_2019 | 3:29:06 | 2:58:00 | 0.8512673362 | |
| spark_backfill_merge_into_2020 | 2:22:12 | 3:11:50 | 1.349038912 | |
| spark_backfill_merge_into_2021 | 2:52:15 | 2:41:29 | 0.9374939526 | |
| spark_backfill_merge_into_2022 | 4:08:28 | 4:25:13 | 1.067413469 | |
| spark_backfill_merge_into_2023 | 4:28:56 | 3:49:54 | 0.854858701 | |
| spark_backfill_merge_into_2024 | 0:07:10 | 0:36:58 | 5.158139535 | 2024 data keeps growing so this one is not informative. |
| drop_intermediate_table | 0:00:25 | 0:00:27 | 1.08 | |
| remove_intermediate_files | 0:00:31 | 0:00:30 | 0.9677419355 | |
| Total | 62:38:03 | 56:53:06 | 0.9082103751 | 9% total time improvement |