Page MenuHomePhabricator
Create Task
Maniphest T414066

Download enterprise structured content snapshots in hdfs
Closed, ResolvedPublic
Actions

Description

The enterprise structured content snapshots will be the source of the semantic search MVP.
We want to refresh the indices weekly and for this we need to download such dumps at this same rate.

We are going to join these dumps with the main search index dumps done on Sundays.

Related: T403298
Related: import_enterprise_dumps.py

Preliminary work: https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/commit/c5925b4e6825fc9c2bf400e08d8c44fd55e3ab26

AC:

  • structured content snapshots are available in hdfs
  • updated weekly on Sundays

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
airflow-search: add enterprise extra_secretsoperations/deployment-chartsmaster+9 -0
Customize query in gerrit

Related Objects

Event Timeline

dcausse created this task.Jan 8 2026, 10:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 8 2026, 10:38 AM
pfischer updated the task description. (Show Details)Jan 8 2026, 10:52 AM
dcausse updated the task description. (Show Details)Jan 8 2026, 11:12 AM
pfischer updated the task description. (Show Details)Jan 8 2026, 11:34 AM
MGerlach subscribed.Jan 8 2026, 12:19 PM
dcausse edited projects, added Discovery-Search (2026.01.05 - 2026.01.30); removed Discovery-Search.Jan 8 2026, 3:30 PM
dcausse claimed this task.Jan 8 2026, 4:01 PM
dcausse moved this task from Incoming to In Progress on the Discovery-Search (2026.01.05 - 2026.01.30) board.
CodeReviewBot added a project: Patch-For-Review.Jan 8 2026, 6:43 PM

dcausse opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1911

search: download wikimedia enterprise structured content snapshots

dcausse moved this task from In Progress to Needs Review on the Discovery-Search (2026.01.05 - 2026.01.30) board.Jan 8 2026, 6:44 PM
gerritbot added a comment.Jan 9 2026, 9:37 AM

Change #1224894 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] airflow-search: add enterprise extra_secrets

https://gerrit.wikimedia.org/r/1224894

awight subscribed.Jan 12 2026, 8:28 AM

The full HTML snapshots are "Updated twice-monthly (on the 2nd and 21st)" (1) so I'm curious to know whether Structured Contents follows that same cadence.

dcausse added a comment.EditedJan 12 2026, 9:29 AM
In T414066#11511114, @awight wrote:

The full HTML snapshots are "Updated twice-monthly (on the 2nd and 21st)" (1) so I'm curious to know whether Structured Contents follows that same cadence.

The structured content snapshots are updated daily (and the html snapshots as well assuming you have an "non-free" account), from the same page:

If you require snapshots updated daily and/or access to the new Structured Contents snapshots, please contact our sales team for access.
Downloadable bundle of structured contents of all current revisions in a specified project and namespace. Updated daily at 12:00 UTC.

The WMF search platform has an account that we will be using for this and download them weekly (on Sundays) because this cadence is enough for us.
Ultimately (if the project is successful) we won't rely on them but do the passages extraction from within MW itself once we get a better understanding of how to "chunk" the articles.

CodeReviewBot added a comment.Wed, Jan 14, 9:45 AM

dcausse merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1911

search: download wikimedia enterprise structured content snapshots

gerritbot added a comment.Wed, Jan 14, 3:13 PM

Change #1224894 merged by Bking:

[operations/deployment-charts@master] airflow-search: add enterprise extra_secrets

https://gerrit.wikimedia.org/r/1224894

Maintenance_bot removed a project: Patch-For-Review.Wed, Jan 14, 3:46 PM
CodeReviewBot added a project: Patch-For-Review.Wed, Jan 14, 9:00 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/72

import_enterprise_dumps: use proper auth endpoint

CodeReviewBot added a comment.Wed, Jan 14, 9:17 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/72

import_enterprise_dumps: use proper auth endpoint

Maintenance_bot removed a project: Patch-For-Review.Wed, Jan 14, 9:33 PM
CodeReviewBot added a project: Patch-For-Review.Wed, Jan 14, 10:12 PM

dcausse opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1919

search: bump discolytics version

CodeReviewBot added a comment.Wed, Jan 14, 10:21 PM

dcausse merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1919

search: bump discolytics version

Maintenance_bot removed a project: Patch-For-Review.Wed, Jan 14, 10:31 PM
dcausse closed this task as Resolved.Thu, Jan 15, 8:20 AM
dcausse moved this task from Needs Review to Done on the Discovery-Search (2026.01.05 - 2026.01.30) board.

Dumps are available as text files (raw ndjson) under /wmf/data/discovery/wikimedia_enterprise/structured_content_snapshots/snapshot=$YYYYMMDD/project=${WIKI}_namespace_0 and will be updated weekly on Sundays.

pfischer moved this task from Done to Reported on the Discovery-Search (2026.01.05 - 2026.01.30) board.Fri, Jan 16, 12:59 PM
awight mentioned this in T414803: Write an Airflow sensor to detect Enterprise Snapshots.Fri, Jan 16, 1:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits