In the context of migrating the metric dbt jobs to use wmf_mediawiki.mediawiki_history_incremental_v1 in a weekly fashion, the following MW tables have to be sqooped daile:
Checklist of completion:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| add daily sqoop repair job | repos/data-engineering/airflow-dags!2418 | a-pizzata | T432208-repair-table-daily-sqoops | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T204950 Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users | |||
| Open | None | T258511 Data Lake incremental Data Updates | |||
| Open | Ahoelzl | T418032 Weekly delivery cadence of core contributor metrics | |||
| In Progress | xcollazo | T424350 Incremental MediaWiki History Phase I | |||
| Open | APizzata-WMF | T429770 migrate dbt_jobs pipelines to use MWH incremental | |||
| Open | APizzata-WMF | T432208 Daily sqoops for logging and cu_log tables |
The logging sqoop succeeded with the configuration:
/usr/bin/python3 /srv/deployment/analytics/refinery/bin/sqoop-mediawiki-tables \
--job-name sqoop-mediawiki-loigging-$(/bin/date --date="yesterday" +'%Y-%m-%d') \
--clouddb \
--output-dir /path/to/raw/mediawiki/tables \
--wiki-file /mnt/hdfs/wmf/data/wmf/mediawiki/database/grouped_wikis.csv \
--tables logging \
--user s53272 \
--password-file /path/to/mysql-analytics-labsdb-client-pw.txt\
--from-timestamp $(/bin/date --date="yesterday" +'%Y%m%d000000') \
--to-timestamp $(/bin/date +'%Y%m%d000000') \
--partition-name day \
--partition-value $(/bin/date --date="yesterday" +'%Y-%m-%d') \
--mappers 64 \
--processors 10 \
--yarn-queue default \
--output-format avrodata \
--log-file sqoop-test/log/refinery/sqoop-mediawiki.logIt took roughly 1 hour and this is the du:
hdfs dfs -du -h -s /path/to/raw/mediawiki/tables/logging/day=2026-07-14 12.4 M /path/to/raw/mediawiki/tables/logging/day=2026-07-14
Regarding the target path, I was thinking of using wmf/data/raw/mediawiki[_private]/tables/daily/name_of_the_table in order to physically divide the daily sqoops from the monthly sqoops.
The cu_log sqoop succeeded with the configuration:
/usr/bin/python3 /srv/deployment/analytics/refinery/bin/sqoop-mediawiki-tables \
--job-name sqoop-mediawiki-daily-private-$(/bin/date --date="yesterday" +'%Y-%m-%d') \
--output-dir /path/to/raw/mediawiki_private/tables \
--wiki-file /mnt/hdfs/wmf/data/wmf/mediawiki/database/grouped_wikis.csv \
--tables cu_log \
--user research \
--password-file /path/to/mysql-analytics-research-client-pw.txt \
--from-timestamp $(/bin/date --date="yesterday" +'%Y%m%d000000') \
--to-timestamp $(/bin/date +'%Y%m%d000000') \
--partition-name day \
--partition-value $(/bin/date --date="yesterday" +'%Y-%m-%d') \
--mappers 4 \
--processors 10 \
--output-format avrodata \
--yarn-queue default \
--log-file sqoop-test/log/refinery/sqoop-mediawiki-private-daily.logIt took roughly 2 hours and this is the du:
hdfs dfs -du -h -s path/to/raw/mediawiki_private/tables/cu_log/day=2026-07-14 1.6 M /path/to/raw/mediawiki/tables/logging/day=2026-07-14
Change #1311398 had a related patch set uploaded (by A-pizzata; author: A-pizzata):
[analytics/refinery@master] Add creation statement for daily sqooped tables cu_log and logging.
Change #1311397 had a related patch set uploaded (by A-pizzata; author: A-pizzata):
[operations/puppet@production] Daily sqoops for logging and cu_log tables
Change #1311397 merged by Brouberol:
[operations/puppet@production] Daily sqoops for logging and cu_log tables
Change #1311398 merged by A-pizzata:
[analytics/refinery@master] Add creation statement for daily sqooped tables cu_log and logging.
Change #1313228 had a related patch set uploaded (by A-pizzata; author: A-pizzata):
[operations/puppet@production] Fix daily sqoop.
hi @APizzata-WMF , since we have platform in the heartbeat metrics, i was wondering if mediawiki change_tag and change_tag_def was also in scope ?
change_tag and change_tag_def was also in scope
These are ingested daily in the stream table: event.mediawiki_revision_tags_change.
These are ingested daily in the stream table: event.mediawiki_revision_tags_change.
Adding on this, the stream has revision focused change_tags but it will be improved in T423935.
yes, i was going to add that these aren't revision related tags, we rely on change_tag and change_tag_def for getting the platform for moderator actions.
T423935 is phase II of incremental_mwh. so that means the weekly active UWERs metric will not have platform dimension as of now. I think thats ok. copying @Milimetric and @nshahquinn-wmf so we're all on the same page.
Change #1314729 had a related patch set uploaded (by A-pizzata; author: A-pizzata):
[operations/puppet@production] Add deletion jobs for mediawiki_logging_daily and mediawiki_private_cu_log_daily.
Change #1314787 had a related patch set uploaded (by A-pizzata; author: A-pizzata):
[operations/puppet@production] Add deletion jobs for daily logging and cu_log tables.
Change #1314729 abandoned by A-pizzata:
[operations/puppet@production] Add deletion jobs for mediawiki_logging_daily and mediawiki_private_cu_log_daily.
Reason:
duplicated
a-pizzata opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2418
add daily sqoop repair job
a-pizzata merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2418
add daily sqoop repair job
Change #1314787 merged by Brouberol:
[operations/puppet@production] Add deletion jobs for daily logging and cu_log tables.