Summary

The daily systemd timer refinery-sqoop-mediawiki-production-daily (which imports the discussiontools_subscription table) fails every run for ~115 small/closed wikis. The job exits non-zero and prints a large "Jobs to re-run" list, but no data is actually lost — every wiki that has the
table imports correctly.

Slack
Alert Email

Root cause

discussiontools_subscription is defined in refinery/python/refinery/sqoop.py (queries['discussiontools_subscription'], ~line 1173) without a sqoopable_dbnames restriction. The is_sqoopable guard (sqoop.py ~line 58) therefore treats every wiki in grouped_wikis.csv as importable, so sqoop is invoked even for wikis where the DiscussionTools schema was never applied and the table does not exist.

On those wikis sqoop dies with:

ERROR 1146 (42S02): Table '<wiki>.discussiontools_subscription' doesn't exist

This is masked in the logs because sqoop_wiki runs sqoop with check_call(sqoop_arguments, stdout=DEVNULL, stderr=DEVNULL) (sqoop.py ~line 175), so the operator only sees the Python wrapper's CalledProcessError ... returned non-zero exit status 1 and a re-run list — not the underlying MySQL error.

The table is absent on those wikis by design: wmgUseDiscussionTools in operations/mediawiki-config wmf-config/InitialiseSettings.php is default => true but explicitly set to false for exactly these wikis under T420052: Drop extensions from closed wikis where the database tables are unused ("Disable on wikis that were closed before DiscussionTools permalinks were deployed ... This reduces the number of database tables on s3"). With the extension unloaded, the table was dropped — so sqoop attempting it is guaranteed to fail.

Evidence (from an-launcher1003)

$ analytics-mysql lrcwiki -e "SELECT COUNT(*) FROM discussiontools_subscription"
ERROR 1146 (42S02): Table 'lrcwiki.discussiontools_subscription' doesn't exist

$ analytics-mysql nawiki -e "SELECT COUNT(*) FROM discussiontools_subscription"
ERROR 1146 (42S02): Table 'nawiki.discussiontools_subscription' doesn't exist

$ analytics-mysql enwiki -e "SELECT COUNT(*) FROM discussiontools_subscription"
+----------+
| COUNT(*) |
| 1569988  |
+----------+

All failing wikis connect to dbstore1007.eqiad.wmnet:3313 (the s3 section, i.e. the long tail of small/closed wikis).

Suggested fix — use sqoopable_dbnames (preferred)

The table already supports scoping via the sqoopable_dbnames key in queries['discussiontools_subscription'] (sqoop.py ~line 1173). When a wiki is not in that collection, is_sqoopable (sqoop.py ~line 58) is False and sqoop_wiki takes the clean SKIPPING path (sqoop.py ~line 82) — no sqoop is ever launched. This is the same mechanism wbc_entity_usage, centralauth_*, and globalimagelinks use.

The only question is how to source the set of wikis that actually have the table. Note a naive all.dblist − closed.dblist subtraction is not reliable — as it includes wikis not necessarily in closed.dblist — so it must track the real wmgUseDiscussionTools config. Two options:

- Dynamic (most robust, self-maintaining): build the set from where the table actually exists — introspect information_schema.tables for discussiontools_subscription across the relevant sections — and assign that set to sqoopable_dbnames. Never drifts from the MediaWiki config.
- Declarative: have a discussiontools.dblist generated in operations/mediawiki-config from wmgUseDiscussionTools, then scope with get_dbnames_from_mw_config(['discussiontools.dblist']) — a one-liner exactly like wbc_entity_usage → wikidataclient.dblist. (No such dblist exists today; only mobile-anon-talk.dblist is present.)

Per T420052: Drop extensions from closed wikis where the database tables are unused the dropped tables are only in closed wikis. So we can use closed.dblist. Some tables may have their own dblist we can use: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1254217

Secondary hardening (independent of the above)

Regardless of the scoping fix, sqoop_wiki should not hide the real error. It runs sqoop with stdout=DEVNULL, stderr=DEVNULL (sqoop.py ~line 175), which is why this took manual DB queries to diagnose. Capturing stderr and, on a missing-table signature (1146 / 42S02 / doesn't exist), logging it clearly (or treating it as a skip) would make future incidents self-explanatory and protect any other table with uneven per-wiki deployment.

Impact / urgency

No data loss and no downstream breakage — only affected wikis are ones with no discussiontools_subscription data to import. The cost is

1. a daily non-zero exit and noisy re-run list that obscures real failures, and
2. wasted retries (3 tries × ~115 wikis, ~05:00–07:15 each day). Medium priority: worth fixing to keep the daily job green and alerting meaningful.

References

• Timer/wrapper: operations/puppet modules/profile/manifests/analytics/refinery/job/sqoop_mediawiki.pp + modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-production-daily.sh.erb
• Table def: analytics/refinery python/refinery/sqoop.py (queries['discussiontools_subscription'])
• Related: T432208 (daily logging/cu_log sqoops), T425385 (monthly wiki-list gate)