Documentation on adding new wikis: https://wikitech.wikimedia.org/wiki/Data_Engineering/Ops_week#Adding_new_wikis_to_the_sqoop_list

Let's evaluate whether to make improvements to the process as part of this task or create a new one that will make this process faster.

This has been a manual process so far, and my opinion is that we should take this opportunity to fix technical debt:

We maintain lists of wikis for different purposes in canonical_data.wikis, wmf_raw.mediawiki_project_namespace_map, and flat files that sqoop works from. Some of this is explained here. We already proposed, but delayed multiple times, to establish a canonical list that combines the information that consumers of these separate lists needs. To my knowledge, the difference is roughly:

1. canonical_data.wikis has more information about the wikis
2. wmf_raw.mediawiki_project_namespace_map has information about namespaces
3. static files with lists for sqoop has two extra pieces of information: estimated size of the wiki and the implied fact that the wiki's database is available to query on cloud and analytics replicas

Proposal

We update the script that generates mediawiki_project_namespace_map to include everything canonical_data.wikis includes, by joining to the manually curated list. We also update the script to ping each database for existence in cloud and analytics replicas, and to get an estimated size (in some reasonably efficient way, from information_schema). We then put the output of this into `canonical_data.wikis and make sure that's used in all the datasets we maintain (I believe this is the case already)

Change 765578 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Add wikis to sqoop list

https://gerrit.wikimedia.org/r/765578

Change 765578 merged by Milimetric:

[analytics/refinery@master] Add wikis to sqoop list

https://gerrit.wikimedia.org/r/765578

The wikis were sqooped and are available in wikistats 2:

https://stats.wikimedia.org/#/nia.wiktionary.org
https://stats.wikimedia.org/#/nia.wikipedia.org
https://stats.wikimedia.org/#/mnw.wiktionary.org
https://stats.wikimedia.org/#/mni.wiktionary.org
https://stats.wikimedia.org/#/mni.wikipedia.org
https://stats.wikimedia.org/#/wa.wikisource.org
https://stats.wikimedia.org/#/bcl.wiktionary.org
https://stats.wikimedia.org/#/ban.wikisource.org
https://stats.wikimedia.org/#/skr.wiktionary.org
https://stats.wikimedia.org/#/skr.wikipedia.org
https://stats.wikimedia.org/#/pwn.wikipedia.org
https://stats.wikimedia.org/#/ami.wikipedia.org
https://stats.wikimedia.org/#/alt.wikipedia.org
https://stats.wikimedia.org/#/shi.wikipedia.org
https://stats.wikimedia.org/#/smn.wikipedia.org
https://stats.wikimedia.org/#/mad.wikipedia.org
https://stats.wikimedia.org/#/trv.wikipedia.org
https://stats.wikimedia.org/#/jv.wikisource.org
https://stats.wikimedia.org/#/tay.wikipedia.org
https://stats.wikimedia.org/#/lmo.wiktionary.org
https://stats.wikimedia.org/#/diq.wiktionary.org
https://stats.wikimedia.org/#/dag.wikipedia.org

Hello data engineering people, I'm one of the people who are usually involved with wiki creation. We have a maintenance bot that (among other things) can notify various teams about the fact a wiki was created by making a ticket. You can see it at T292419: Post-creation work for pwnwiki and subtasks. Would that help the team with managing new wikis created? If so, can a ticket be filled under Phabricator maintenance bot?

@Urbanecm thanks much for the offer, while that would be cool I think we can also pull this data from the sitematrix for now, until we need more "real time" updates. Mentioned in T190700 where we're tracking this automation work.