Two options to implement this:
For snapshots, we could pull from sitematrix like we're doing now, maybe more frequently
For event-based, we could get notified by this maintenance bot: T292419: Post-creation work for pwnwiki
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • fdans | T187414 Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid" | |||
| Open | None | T190700 Automate creation of sqoop list of wikis to import data for from sitematrix |
Event Timeline
Comment Actions
Tasked this briefly, want to coordinate with T239589 and make T239136 not necessary. To that end,
- Use project namespace map table as the main source of truth when it comes to wiki lists (maybe rename)
- Add a boolean column to project namespace map to indicate whether the wiki database exists on cloud dbs
- Coordinate with research and product analytics to share this metadata as part of their metadata database
- Consider a table holding settings like per-table weights that we could use from sqoop to fine-tune how data is transferred
Comment Actions
I think this should be much higher priority now. Usually we would move tasks back to incoming to re-prioritize, but not sure how to do that in the new process. cc @EChetty
Comment Actions
This would only work for the snapshots, but a simple solution would be to just pull the sqoop list from canonical_data.wikis. There's no automated process keeping that up to date, but:
- I've been pretty diligent about keeping it up to date (adding new wikis within a few weeks)
- We are motivated to switch it to automatic generation as soon as possible, since the manual updates are a source of toil for the Movement Insights team
- It centralizes us further on a single canonical list of wikis, easing the maintenance burden (see T339928 for another example)