### Summary
The daily systemd timer `refinery-sqoop-mediawiki-production-daily` (which imports the `discussiontools_subscription` table) fails every run for ~115 small/closed wikis. The job exits non-zero and prints a large "Jobs to re-run" list, but **no data is actually lost** — every wiki that has the
table imports correctly.
[[ https://wikimedia.slack.com/archives/C05RHK7PS6Q/p1785256933086449 | Slack ]]
[[ https://groups.google.com/a/wikimedia.org/g/data-engineering-alerts/c/b8-tfl9XpjU/m/wBgbD0zxCAAJ | Alert Email ]]
### Root cause
`discussiontools_subscription` is defined in `refinery/python/refinery/sqoop.py` (`queries['discussiontools_subscription']`, ~line 1173) **without a `sqoopable_dbnames` restriction**. The `is_sqoopable` guard (sqoop.py ~line 58) therefore treats every wiki in `grouped_wikis.csv` as importable, so sqoop is invoked even for wikis where the DiscussionTools schema was never applied and the table does not exist.
On those wikis sqoop dies with:
```
ERROR 1146 (42S02): Table '<wiki>.discussiontools_subscription' doesn't exist
```
This is masked in the logs because `sqoop_wiki` runs sqoop with `check_call(sqoop_arguments, stdout=DEVNULL, stderr=DEVNULL)` (sqoop.py ~line 175), so the operator only sees the Python wrapper's `CalledProcessError ... returned non-zero exit status 1` and a re-run list — not the underlying MySQL error.
The table is absent on those wikis **by design**: `wmgUseDiscussionTools` in operations/mediawiki-config `wmf-config/InitialiseSettings.php` is `default => true` but explicitly set to `false` for exactly these wikis under {T420052} ("Disable on wikis that were closed before DiscussionTools permalinks were deployed ... This reduces the number of database tables on s3"). With the extension unloaded, the table was dropped — so sqoop attempting it is guaranteed to fail.
### Evidence (from an-launcher1003)
```
$ analytics-mysql lrcwiki -e "SELECT COUNT(*) FROM discussiontools_subscription"
ERROR 1146 (42S02): Table 'lrcwiki.discussiontools_subscription' doesn't exist
$ analytics-mysql nawiki -e "SELECT COUNT(*) FROM discussiontools_subscription"
ERROR 1146 (42S02): Table 'nawiki.discussiontools_subscription' doesn't exist
$ analytics-mysql enwiki -e "SELECT COUNT(*) FROM discussiontools_subscription"
+----------+
| COUNT(*) |
| 1569988 |
+----------+
```
All failing wikis connect to `dbstore1007.eqiad.wmnet:3313` (the **s3** section, i.e. the long tail of small/closed wikis).
== Suggested fix — use sqoopable_dbnames (preferred) ==
The table already supports scoping via the `sqoopable_dbnames` key in `queries['discussiontools_subscription']` (sqoop.py ~line 1173). When a wiki is not in that collection, `is_sqoopable` (sqoop.py ~line 58) is False and `sqoop_wiki` takes the clean `SKIPPING` path (sqoop.py ~line 82) — no sqoop is ever launched. This is the same mechanism `wbc_entity_usage`, `centralauth_*`, and `globalimagelinks` use.
The only question is how to source the set of wikis that actually have the table. Note a naive `all.dblist − closed.dblist` subtraction is **not** reliable — as it includes wikis not necessarily in `closed.dblist` — so it must track the real `wmgUseDiscussionTools` config. Two options:
- **Dynamic (most robust, self-maintaining):** build the set from where the table actually exists — introspect `information_schema.tables` for `discussiontools_subscription` across the relevant sections — and assign that set to `sqoopable_dbnames`. Never drifts from the MediaWiki config.
- **Declarative:** have a `discussiontools.dblist` generated in operations/mediawiki-config from `wmgUseDiscussionTools`, then scope with `get_dbnames_from_mw_config(['discussiontools.dblist'])` — a one-liner exactly like `wbc_entity_usage` → `wikidataclient.dblist`. (No such dblist exists today; only `mobile-anon-talk.dblist` is present.)
== Secondary hardening (independent of the above) ==
Regardless of the scoping fix, `sqoop_wiki` should not hide the real error. It runs sqoop with `stdout=DEVNULL, stderr=DEVNULL` (sqoop.py ~line 175), which is why this took manual DB queries to diagnose. Capturing stderr and, on a missing-table signature (`1146` / `42S02` / `doesn't exist`), logging it clearly (or treating it as a skip) would make future incidents self-explanatory and protect any other table with uneven per-wiki deployment.
### Impact / urgency
No data loss and no downstream breakage — only affected wikis are ones with no `discussiontools_subscription` data to import. The cost is
1. a daily non-zero exit and noisy re-run list that obscures real failures, and
2. wasted retries (3 tries × ~115 wikis, ~05:00–07:15 each day). Medium priority: worth fixing to keep the daily job green and alerting meaningful.
### References
- Timer/wrapper: operations/puppet
`modules/profile/manifests/analytics/refinery/job/sqoop_mediawiki.pp` +
`modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-production-daily.sh.erb`
- Table def: analytics/refinery `python/refinery/sqoop.py`
(`queries['discussiontools_subscription']`)
- Related: T432208 (daily logging/cu_log sqoops), T425385 (monthly wiki-list gate)