Summary
Run a one-time Spark backfill that fetches the raw Parsoid HTML of every Did You Know (DYK) archive page across the 25 in-scope Wikipedias and lands it in an Iceberg table on HDFS.
This task is fetch-only. It does not parse DYK facts, resolve linked pages, or enrich topics — those are separate downstream tasks. Its single job is to make the historical archive HTML available for the parser to run against.
Background
- T421730: [M] Community Picks: Did you know List view — DYK List View
- T418898: [M] Community Picks: Did You Know Module — DYK Module
- T426347: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 — Wiki page HTML backfill (related infrastructure)
- Design Document
- T432658: [Data Persistence Design Review] Did You Know Archive — Data Persistence Design Review
DYK facts rotate off each wiki's Main Page into archive pages. Per the design, ongoing ingestion comes from the HTML content-change event stream, but there are no historical entries for archive pages in the event table. We therefore seed the archive by calling the MediaWiki core REST API once per archive page.
Scope
- Discover archive page titles for the 25 in-scope wikis (Action API).
- Fetch each page via GET /w/rest.php/v1/page/{title}/with_html.
- Write one row per archive page to ebysans.dyk_archive_pages_html_content (Iceberg, partitioned by wiki_id), including revision metadata and a dated flag indicating whether the wiki exposes per-entry date headings.
In-scope wikis (25)
enwiki, dewiki, frwiki, svwiki, fiwiki, bgwiki, nlwiki, hewiki, ruwiki, srwiki, idwiki, ukwiki, viwiki, zhwiki, arwiki, ptwiki, kowiki, dawiki, nowiki, huwiki, cswiki, elwiki, trwiki, thwiki, fawiki.
Excluded from the list according to the audit:
jawiki (cannot find),
itwiki (cannot find),
eswiki (no active DYK),
plwiki (their links don't match their article, worry that link title will point to a different article),
rowiki (archive not maintained),
cawiki (cannot find),
hiwiki (cannot find),
Implementation
Standalone spark-submit job (structured to later drop into the
research/dags/content_html_dag.py DAG pattern):
- Discover: per-wiki strategy — allpages by prefix (most wikis), categorymembers (arwiki), or single page (nlwiki, srwiki, cswiki, elwiki, fawiki). Namespace resolved dynamically.
- Fetch: /with_html per page — descriptive User-Agent with metadata wrapper, 404 recorded as a row-level error rather than failing the job.
- Write: Iceberg dynamic partition overwrite by wiki_id — re-running a wiki replaces its partition, so the job is idempotent.
Each wiki has an ArchiveStructure that determines how pages are discovered:
| Structure | Description | Wikis |
|---|---|---|
| MONTHLY_SUBPAGES | allpages API with monthly prefix | enwiki, dewiki, zhwiki, ruwiki, ukwiki |
| YEARLY_SUBPAGES | allpages API with yearly prefix | frwiki, svwiki, hewiki, ptwiki, viwiki, kowiki, dawiki, huwiki |
| DATE_SUBPAGES | allpages API with date-named prefix | idwiki, thwiki |
| CATEGORY_MEMBERS | categorymembers API (archive pages in a category, not subpages) | arwiki |
| NUMBERED_ARCHIVES | fixed list (Arkisto10–14) | fiwiki |
| MULTI_PAGE | fixed list of era pages | trwiki |
| SINGLE_PAGE | single fixed page title | nowiki, elwiki, nlwiki, cswiki, bgwiki, srwiki, fawiki |
Notable non-obvious configs:
- arwiki: uses the CATEGORY_MEMBERS structure — archive pages are members of تصنيف:قائمة_مشروع_هل_تعلم and discovered via the categorymembers API, not allpages.
- ruwiki: namespace 4 is Проект: (not Википедия:); archive root is Проект:Знаете_ли_вы/Архив_рубрики.
- ukwiki: namespace 4 is Вікіпедія:; the page name within that namespace contains an internal colon: Проєкт:Чи_ви_знаєте/Архів_рубрики.
- elwiki: archive is in the Portal namespace (Πύλη:, ns=100), not Wikipedia: (ns=4). Single page.
- bgwiki: audit URL had a #Архив anchor — this is navigation only; the full page Уикипедия:Последни_попълнения is fetched.
Rate limiting: 5 partitions × 2 req/s per executor = 10 req/s global (WMF API convention).
Output table — ebysans.dyk_archive_pages_html_content
Iceberg table, PARTITIONED BY (wiki_id). 11 columns:
| field | type | comment |
|---|---|---|
| wiki_id | string | wiki database code, e.g. enwiki (partition key) |
| host | string | wiki host used for fetching, e.g. en.wikipedia.org |
| archive_path | string | title/path of the archive page fetched (not a linked-article title) |
| content_body | string | Parsoid HTML of the archive page |
| rev_id | bigint | latest revision id of the archive page at fetch time |
| rev_dt | timestamp | latest timestamp of that revision |
| dated | boolean | whether the wiki archive exposes per-entry date headings (false for nowiki, elwiki) |
| error | string | error message; null on success |
| http_status | int | HTTP status of the fetch |
| source_url | string | REST API URL fetched |
| fetched_at | timestamp | when this row was fetched (UTC) |
Partitioned by `wiki_id`. ## Acceptance criteria 1. `wmf_content.dyk_archive_pages_html_content` exists and is queryable in the Iceberg catalog. 2. All 25 wikis have at least one row in the table. 3. Overall fetch error rate < 5%. 5. Job is idempotent: re-running overwrites existing partitions without duplicating rows.