FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Dec 16 2020, 2:08 AM

Description

Tracking work happening in FY2021-2022 to finish up remaining work required before basic Multi-DC operations can be enabled.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		aaron	T252951 ResourceLoader DepStore lock acquired twice?
Resolved		aaron	T88445 MediaWiki active/active datacenter investigation and work (tracking)
Resolved		Krinkle	T270223 FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Resolved		tstarling	T267270 Determine multi-dc strategy for CentralAuth
Resolved		aaron	T278392 Storage solution for cross-datacenter tokens
Resolved	PRODUCTION ERROR	tstarling	T311590 ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}
Resolved		tstarling	T313496 Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask
Resolved		tstarling	T91820 Create HTTP verb and sticky cookie DC routing in VCL
Resolved		aaron	T91816 Add code to enable setting sticky DC cookies for POST requests
Resolved		aaron	T121440 Dedicated post-edit cache busting cookie to prevent stale reads (session consistency)
Resolved		Krinkle	T113916 Switch ResourceLoader file dependency tracking to MultiDC-friendly backend
Resolved		aaron	T247028 Database 'INSERT' query rate doubled (module_deps regression?)
Resolved		tstarling	T253433 Web installer CSS fails with ServiceDisabledException
Declined		aaron	T253697 Support hash-based deduplication in KeyValueDependencyStore
Resolved		aaron	T212129 Move MainStash out of Redis to a simpler multi-dc aware solution
Resolved		Eevans	T222851 Improve Echo seentime code for multi-DC access
Resolved		aaron	T229062 Look into a simple way to have global keys with db-replicated
Resolved		Krinkle	T254634 Determine and implement multi-dc strategy for ChronologyProtector
			Unknown Object (Task)
Resolved		Papaul	T267041 (Need By: 2020-11-29) rack/setup/install db214[234]
Resolved		• Marostegui	T269324 Productionize x2 databases
Resolved		aaron	T274174 Add modtoken field and flags to objectcache table
Resolved		Krinkle	T288998 Significant ParserCache space increase after 2021-08-12 (1.37.0-wmf.18 regression)
Resolved		tstarling	T306118 Notify DBA prior to sending db traffic to x2
Resolved		tstarling	T315271 db1151, db2144 X2 masters error: Could not execute Delete_rows_v1 event on table mainstash.objectstash
Resolved	PRODUCTION ERROR	• jcrespo	T315274 Wikimedia\Rdbms\DBTransactionError: Explicit transaction still active; a caller might have failed to call endAtomic() or cancelAtomic().
Resolved		tstarling	T315427 Create a puppet role for x2 hosts
Resolved		• Marostegui	T315853 reclone x2 codfw hosts
Resolved		tstarling	T315995 Document how to disable x2 per DC
Resolved		Krinkle	T270225 Finish session storage to actually meet multi-DC requirements
Resolved		Eevans	T222990 Audit session storage to determine max age of un-GC'd sessions
Resolved		tstarling	T279664 Progressive Multi-DC roll out
Resolved		Ladsgroup	T314486 Add support for multidc to auto_schema
Resolved		tstarling	T314750 Clean up testwiki experiments (Aug 2022)
Resolved		Vgutierrez	T315434 multi-dc.lua ATS script failing in production
Resolved		tstarling	T134809 App servers <=> mariadb SSL/TLS for cross-datacenter writes
Resolved		aaron	T136218 Audit mysql database class and hhvm binding support of SSL
Resolved		• jcrespo	T111654 Set up TLS for MariaDB replication
Resolved		• jcrespo	T120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic)
Resolved		• Cmjohnson	T120689 es1019 and its management interface are unresponsive
Resolved		Gehel	T128077 Create a PKI that can be used by Puppet and for general purpose certificates
Resolved		• jcrespo	T151995 Rolling restart of external storage servers for TLS certificate update
Resolved		• jcrespo	T152029 Rolling restart of parsercache servers for TLS certificate update
Resolved		• jcrespo	T152188 Restart pending mysql hosts with old TLS cert
Resolved		• Marostegui	T152364 db1047 out of disk space, eventlogging_sync spam
Duplicate		None	T152595 Implement TLS expiration/validation checking for MariaDB certificates
Resolved		• jcrespo	T156005 Reimage db1065 and db1066
Resolved		aaron	T171071 Perform testing for TLS effect on connection rate
Resolved		• jcrespo	T175672 Evaluate proxysql and native php-mysql client for TLS connection between Apache and mariadb
Resolved		tstarling	T196378 Investigate solutions for MySQL connection pooling
Resolved		tstarling	T313578 Make OAuth work in Multi-DC active/active mode
Resolved		aaron	T320995 Create blog entry about multi-dc
Resolved		Krinkle	T329692 Story idea for Blog: Around the world: how we serve traffic from multiple data centers

Event Timeline

Krinkle created this task.Dec 16 2020, 2:08 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 16 2020, 2:08 AM

Krinkle added subtasks: T254634: Determine and implement multi-dc strategy for ChronologyProtector, T267270: Determine multi-dc strategy for CentralAuth, T91820: Create HTTP verb and sticky cookie DC routing in VCL .Dec 16 2020, 2:08 AM

Krinkle added a subtask: T113916: Switch ResourceLoader file dependency tracking to MultiDC-friendly backend.

Krinkle added a subtask: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Dec 16 2020, 2:10 AM

Krinkle removed a subtask: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.

Krinkle added a subtask: T254422: Move CentralAuth sessions from redis backend to kask.

Krinkle removed a subtask: T254422: Move CentralAuth sessions from redis backend to kask.Dec 16 2020, 2:13 AM

Krinkle renamed this task from FY2020-2012: Enable basic Multi-DC operations for read traffic (tracking) to FY2020-2021: Enable basic Multi-DC operations for read traffic (tracking).Dec 19 2020, 12:24 AM

Krinkle edited projects, added Performance-Team; removed Performance-Team (Radar).

Krinkle moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.Dec 21 2020, 7:56 PM

Krinkle added a subtask: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Dec 22 2020, 8:46 PM

Krinkle updated the task description. (Show Details)Dec 22 2020, 8:52 PM

• ema updated the task description. (Show Details)Jan 4 2021, 8:58 AM

greg subscribed.Jan 4 2021, 6:03 PM

nshahquinn-wmf subscribed.Jan 7 2021, 12:54 PM

RhinosF1 subscribed.Jan 12 2021, 11:22 AM

Jdforrester-WMF subscribed.Jan 19 2021, 9:49 PM

Krinkle triaged this task as High priority.Jan 25 2021, 7:55 PM

Krinkle mentioned this in T270101: Grants not working with DB hosts with to ipv6.Mar 10 2021, 9:29 PM

Izno subscribed.Mar 19 2021, 1:53 AM

Ltrlg subscribed.Mar 19 2021, 12:09 PM

Krinkle closed subtask T254634: Determine and implement multi-dc strategy for ChronologyProtector as Resolved.Mar 19 2021, 8:29 PM

Krinkle renamed this task from FY2020-2021: Enable basic Multi-DC operations for read traffic (tracking) to FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking).Apr 8 2021, 1:38 PM

Krinkle updated the task description. (Show Details)

• taavi subscribed.Apr 13 2021, 1:06 PM

Krinkle moved this task from To-do: Goals prioritized current Quarter to To-do: Goals, prioritized next 4 Quarters on the Performance-Team board.Sep 20 2021, 6:23 PM

Krinkle added a parent task: T88445: MediaWiki active/active datacenter investigation and work (tracking).Oct 20 2021, 1:06 AM

Krinkle added a subtask: T134809: App servers <=> mariadb SSL/TLS for cross-datacenter writes.May 31 2022, 11:40 PM

tstarling closed subtask T134809: App servers <=> mariadb SSL/TLS for cross-datacenter writes as Resolved.Jun 14 2022, 12:00 AM

tstarling closed subtask T212129: Move MainStash out of Redis to a simpler multi-dc aware solution as Resolved.Jun 17 2022, 1:38 AM

Krinkle closed subtask T91820: Create HTTP verb and sticky cookie DC routing in VCL as Resolved.Jul 6 2022, 12:57 AM

Krinkle closed subtask T113916: Switch ResourceLoader file dependency tracking to MultiDC-friendly backend as Resolved.Jul 15 2022, 6:50 AM

Krinkle updated the task description. (Show Details)

Krinkle closed subtask T277831: Evaluate WMF's ParserCache database setup for Multi DC as Declined.Jul 21 2022, 7:48 AM

Ladsgroup subscribed.Jul 21 2022, 12:59 PM