Create a database on all relevant wikireplica hosts called "datasets_p" that will be used for importing managed Dataset(TM) tables via a process around the soon-to-be-created phabricator tag "wikireplica-datasets" (T173512).
This database will be used to host a subset of the tables that are currently hosted in userDBs on the old labsDB replicas.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jcrespo | T140788 Labs databases rearchitecture (tracking) | |||
| Resolved | bd808 | T166402 Program 7 Outcome 3: data services | |||
| Resolved | bd808 | T142807 Migrate all users to new Wiki Replica cluster and decommission old hardware | |||
| Declined | None | T180636 Make Dispenser's principle_links table accessible in new Wiki replica cluster | |||
| Open | Feature | None | T173511 Implement technical details and process for "datasets_p" on wikireplica hosts | ||
| Declined | Halfak | T179187 Reload monthly article quality dataset into wikireplica "datasets_p" | |||
| Duplicate | None | T173513 Create a database on the wikireplica servers called "datasets_p" |
The ask here as I understand is to provide a database that is co-located with the wiki replicas available via the wikireplica-web.eqiad.wmnet and wikireplica-analytics.eqiad.wmnet service names that can contain N tables of curated data produced by ORES, some analytics project, or other ETL/aggregation methods. These are similar to adhoc tool databases in that they are not canonical MediaWiki metadata, but different in that each table would have a clear owner and a well documented process for how the data it contains is produced and replicated across the pool of hosts that comprise the logical cluster.
I can see the value in this, but I want the DBA team (@jcrespo, @Marostegui) to help design a system that they see as scalable and maintainable. I don't want to see the hard work of T153743: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts accidentally undone by adding in new complexity without careful consideration of how it will effect:
See T173511: Implement technical details and process for "datasets_p" on wikireplica hosts for higher level discussion.