Program 7 Outcome 3: data services
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	May 26 2017, 5:51 PM

Description

See https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q1#Cloud_Services

Program

Smart tools for better data

Goal

Make Wikimedia data easily available for both the Foundation and the different Wiki communities by providing better tools, infrastructure, and access to data for editors, communities, and Foundation staff.

Outcome

Wikimedia Cloud Services users have easy access to public data.

Objectives, Activities, Milestones, and Targets

Objective 1: Provide reliable and available access to Wikimedia database dumps by upgrading the hardware used and consolidating access by internal teams, Cloud Services users, external mirrors, and HTTPS downloaders to the new canonical location.
Objective 2: Complete migration of production database replica access for Cloud services customers to the new high-availability cluster, which uses 'row based' replication technology to provide a more consistent view of production data.
Objective 3: Collaborate with Wikimedia Cloud Services customers to publish new applicable data sets

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		bd808	T166402 Program 7 Outcome 3: data services
Resolved		• madhuvishy	T168486 Migrate customer-facing Dumps endpoints to Cloud Services
Resolved		• madhuvishy	T171508 Investigate and implement alternative for showmount based check at instance boot time
Resolved		• madhuvishy	T174590 Revert patch that adds a temporary exception to the block-for-export check for the testlabs project
Resolved		• madhuvishy	T171539 Puppetize and setup initial lvms and directory structures for labstore1006\|7
Resolved		ArielGlenn	T171541 Setup periodic rsync jobs from dumps generation hosts to labstore1006\|7
Resolved		ArielGlenn	T188726 make sure all datasets in xmldatadumps/public/other on dataset1001 are accounted for on new labs boxes
Resolved		• ezachte	T189283 Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches
Resolved		ArielGlenn	T189284 Stop serving slowparse logs from dumps distribution servers
Resolved		• madhuvishy	T181431 Setup NFS on dumps servers
Resolved		• madhuvishy	T171540 Figure out how NFS failovers will work for the dumps servers - labstore1006\|7
Resolved		• madhuvishy	T188073 Maintain symlinks for WMCS NFS Dumps users with new directory structure
Resolved		• madhuvishy	T188641 Set up the web service that serves dumps.wikimedia.org
Resolved		• madhuvishy	T188642 Set up labstore1006\|7 as the source for rsync mirror sites
Resolved		• madhuvishy	T188643 Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7
Resolved		• madhuvishy	T188644 Migrate the stat* mount from dataset1001 to labstore1006/7
Resolved		• madhuvishy	T188645 Get all the rsync mirror sites to to switch over to labstore1006,7
Resolved		• madhuvishy	T188646 Point dumps.wikimedia.org to labstore1006/7
Resolved		• madhuvishy	T185101 Labstore1006/7 profile for meltdown kernel
Resolved		• madhuvishy	T188647 Announce/Communicate dumps migration to labstore1006\|7 to stakeholders
			Restricted Task
Resolved		bd808	T142807 Migrate all users to new Wiki Replica cluster and decommission old hardware
Resolved		bd808	T172704 Promote initial use of new Wiki Replica servers
Resolved		bd808	T174860 Define naming scheme for connecting to new wiki replica cluster
Resolved		• madhuvishy	T168584 Labsdb* servers need to be rebooted
Resolved		bd808	T175086 Create and announce timeline for shutting down labsdb100[13]
Resolved		bd808	T175096 Identify tools hosting databases on labsdb100[13] and notify maintainers
Resolved		bd808	T176688 Update `sql` command to use new wiki replica servers
Resolved		zhuyifei1999	T176694 Switch Quarry to use *.analytics.db.svc.eqiad.wmflabs as replica database host
Open		None	T176886 Update meta_p database for new service names
Resolved		jcrespo	T177096 Some queries to new replica hosts are dramatically slower than labsdb; missing indexes?
Resolved		bd808	T177223 Determine schema differences between labsdb1001 and labsdb1009
Resolved		eranroz	T179227 Migrate copypatrol & plagiabot to use tools.labsdb
Resolved		• Marostegui	T179464 labsdb1001 crashed - storage issue
Declined		None	T179628 Consider granting `CREATE TEMPORARY TABLES` to labsdbuser
Open		None	T180558 Include namespace IDs and their names to mysql wikireplicas (meta_p database)
Declined		None	T180636 Make Dispenser's principle_links table accessible in new Wiki replica cluster
Open	Feature	None	T173511 Implement technical details and process for "datasets_p" on wikireplica hosts
Declined		None	T173512 Create a phabricator project called "wikireplica-datasets"
Duplicate		None	T173513 Create a database on the wikireplica servers called "datasets_p"
Duplicate		None	T173514 Document the process for importing a new "datasets_p" table
Resolved		zhuyifei1999	T181492 sql command should point to the new labsdb servers
Declined		None	T182948 Create method for accessing user watchlists in database queries
Resolved		• madhuvishy	T183029 Stop managing account creation for labsdb1001 and 1003 through the maintain-dbusers script
Resolved		jcrespo	T186585 Review m5 backups
Resolved		bd808	T183651 centralauth database service name missing from new replicas
Resolved		• Marostegui	T183758 Create backups of user tables from decommissioned database servers
Resolved		• Banyek	T183983 Re-institute query killer for the analytics WikiReplica
Resolved		• Banyek	T203674 Debian package or files managed my puppet for pt-kill-wmf
Resolved		• Cmjohnson	T184832 Decommission labsdb1001 and labsdb1003

Event Timeline

bd808 created this task.May 26 2017, 5:51 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2017, 5:51 PM

bd808 moved this task from Backlog to Outcomes on the cloud-services-team (FY2017-18) board.May 26 2017, 5:51 PM

bd808 renamed this task from Program 7 Outcome 3: Wikimedia Cloud Services users have easy access to public data to Program 7 Outcome 3: data services.May 26 2017, 6:02 PM

bd808 triaged this task as Medium priority.May 26 2017, 6:06 PM

bd808 created subtask T168486: Migrate customer-facing Dumps endpoints to Cloud Services.Jun 21 2017, 12:24 AM

bd808 added a project: Data-Services.Sep 11 2017, 9:57 PM

bd808 added a subtask: T142807: Migrate all users to new Wiki Replica cluster and decommission old hardware.Sep 29 2017, 10:45 PM

Krinkle updated the task description. (Show Details)Oct 5 2017, 11:04 PM

• madhuvishy closed subtask T168486: Migrate customer-facing Dumps endpoints to Cloud Services as Resolved.Apr 24 2018, 11:57 PM

bd808 closed subtask T142807: Migrate all users to new Wiki Replica cluster and decommission old hardware as Resolved.Jun 29 2018, 9:10 PM

bd808 closed this task as Resolved.Nov 6 2018, 11:11 PM

bd808 claimed this task.