FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Open, HighPublic
Actions

Assigned To

Authored By

	Krinkle
	Dec 16 2020, 2:08 AM

Description

Tracking work happening in FY2021-2022 to finish up remaining work required before basic Multi-DC operations can be enabled.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		aaron	T252951 ResourceLoader DepStore lock acquired twice?
Open		None	T88445 MediaWiki active/active datacenter investigation and work (tracking)
Open		Krinkle	T270223 FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Open		tstarling	T267270 Determine multi-dc strategy for CentralAuth
Resolved		aaron	T278392 Storage solution for cross-datacenter tokens
Resolved	PRODUCTION ERROR	tstarling	T311590 ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}
Resolved		tstarling	T313496 Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask
Resolved		tstarling	T91820 Create HTTP verb and sticky cookie DC routing in VCL
Resolved		aaron	T91816 Add code to enable setting sticky DC cookies for POST requests
Resolved		aaron	T121440 Dedicated post-edit cache busting cookie to prevent stale reads (session consistency)
Resolved		Krinkle	T113916 Switch ResourceLoader file dependency tracking to MultiDC-friendly backend
Resolved		aaron	T247028 Database 'INSERT' query rate doubled (module_deps regression?)
Resolved		tstarling	T253433 Web installer CSS fails with ServiceDisabledException
Declined		aaron	T253697 Support hash-based deduplication in KeyValueDependencyStore
Resolved		aaron	T212129 Move MainStash out of Redis to a simpler multi-dc aware solution
Resolved		Eevans	T222851 Improve Echo seentime code for multi-DC access
Resolved		aaron	T229062 Look into a simple way to have global keys with db-replicated
Resolved		Krinkle	T254634 Determine and implement multi-dc strategy for ChronologyProtector
			Unknown Object (Task)
Resolved		Papaul	T267041 (Need By: 2020-11-29) rack/setup/install db214[234]
Resolved		• Marostegui	T269324 Productionize x2 databases
Resolved		aaron	T274174 Add modtoken field and flags to objectcache table
Resolved		Krinkle	T288998 Significant ParserCache space increase after 2021-08-12 (1.37.0-wmf.18 regression)
Resolved		tstarling	T306118 Notify DBA prior to sending db traffic to x2
Resolved		Krinkle	T270225 Finish session storage to actually meet multi-DC requirements
Resolved		Eevans	T222990 Audit session storage to determine max age of un-GC'd sessions
Declined		None	T277831 Evaluate WMF's ParserCache database setup for Multi DC
Open		None	T279664 Progressive Multi-DC roll out
Open		Ladsgroup	T314486 Add support for multidc to auto_schema
Resolved		tstarling	T134809 App servers <=> mariadb SSL/TLS for cross-datacenter writes
Resolved		aaron	T136218 Audit mysql database class and hhvm binding support of SSL
Resolved		jcrespo	T111654 Set up TLS for MariaDB replication
Resolved		jcrespo	T120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic)
Resolved		Cmjohnson	T120689 es1019 and its management interface are unresponsive
Resolved		Gehel	T128077 Create a PKI that can be used by Puppet and for general purpose certificates
Resolved		jcrespo	T151995 Rolling restart of external storage servers for TLS certificate update
Resolved		jcrespo	T152029 Rolling restart of parsercache servers for TLS certificate update
Resolved		jcrespo	T152188 Restart pending mysql hosts with old TLS cert
Resolved		• Marostegui	T152364 db1047 out of disk space, eventlogging_sync spam
Duplicate		None	T152595 Implement TLS expiration/validation checking for MariaDB certificates
Resolved		jcrespo	T156005 Reimage db1065 and db1066
Resolved		aaron	T171071 Perform testing for TLS effect on connection rate
Resolved		jcrespo	T175672 Evaluate proxysql and native php-mysql client for TLS connection between Apache and mariadb
Resolved		tstarling	T196378 Investigate solutions for MySQL connection pooling
Resolved		tstarling	T313578 Make OAuth work in Multi-DC active/active mode

Event Timeline

Krinkle created this task.Dec 16 2020, 2:08 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 16 2020, 2:08 AM

Krinkle added subtasks: T254634: Determine and implement multi-dc strategy for ChronologyProtector, T267270: Determine multi-dc strategy for CentralAuth, T91820: Create HTTP verb and sticky cookie DC routing in VCL .Dec 16 2020, 2:08 AM

Krinkle added a subtask: T113916: Switch ResourceLoader file dependency tracking to MultiDC-friendly backend.

Krinkle added a subtask: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Dec 16 2020, 2:10 AM

Krinkle removed a subtask: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.

Krinkle added a subtask: T254422: Move CentralAuth sessions from redis backend to kask.

Krinkle removed a subtask: T254422: Move CentralAuth sessions from redis backend to kask.Dec 16 2020, 2:13 AM

Krinkle renamed this task from FY2020-2012: Enable basic Multi-DC operations for read traffic (tracking) to FY2020-2021: Enable basic Multi-DC operations for read traffic (tracking).Dec 19 2020, 12:24 AM

Krinkle edited projects, added Performance-Team; removed Performance-Team (Radar).

Krinkle moved this task from Inbox to Next: Goal-oriented on the Performance-Team board.Dec 21 2020, 7:56 PM

Krinkle added a subtask: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Dec 22 2020, 8:46 PM

Krinkle updated the task description. (Show Details)Dec 22 2020, 8:52 PM

• ema updated the task description. (Show Details)Jan 4 2021, 8:58 AM

greg added a subscriber: greg.Jan 4 2021, 6:03 PM

nshahquinn-wmf added a subscriber: nshahquinn-wmf.Jan 7 2021, 12:54 PM

RhinosF1 added a subscriber: RhinosF1.Jan 12 2021, 11:22 AM

Jdforrester-WMF added a subscriber: Jdforrester-WMF.Jan 19 2021, 9:49 PM

Krinkle triaged this task as High priority.Jan 25 2021, 7:55 PM

Krinkle mentioned this in T270101: Grants not working with DB hosts with to ipv6.Mar 10 2021, 9:29 PM

Izno added a subscriber: Izno.Mar 19 2021, 1:53 AM

Ltrlg added a subscriber: Ltrlg.Mar 19 2021, 12:09 PM

Krinkle closed subtask T254634: Determine and implement multi-dc strategy for ChronologyProtector as Resolved.Mar 19 2021, 8:29 PM

Krinkle renamed this task from FY2020-2021: Enable basic Multi-DC operations for read traffic (tracking) to FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking).Apr 8 2021, 1:38 PM

Krinkle updated the task description. (Show Details)

taavi added a subscriber: taavi.Apr 13 2021, 1:06 PM

Krinkle moved this task from Next: Goal-oriented to Backlog: Future Goals on the Performance-Team board.Sep 20 2021, 6:23 PM

Krinkle added a parent task: T88445: MediaWiki active/active datacenter investigation and work (tracking).Oct 20 2021, 1:06 AM

Krinkle added a subtask: T134809: App servers <=> mariadb SSL/TLS for cross-datacenter writes.May 31 2022, 11:40 PM

tstarling closed subtask T134809: App servers <=> mariadb SSL/TLS for cross-datacenter writes as Resolved.Jun 14 2022, 12:00 AM

tstarling closed subtask T212129: Move MainStash out of Redis to a simpler multi-dc aware solution as Resolved.Jun 17 2022, 1:38 AM

Krinkle closed subtask T91820: Create HTTP verb and sticky cookie DC routing in VCL as Resolved.Jul 6 2022, 12:57 AM

Krinkle closed subtask T113916: Switch ResourceLoader file dependency tracking to MultiDC-friendly backend as Resolved.Fri, Jul 15, 6:50 AM

Krinkle updated the task description. (Show Details)

Krinkle closed subtask T277831: Evaluate WMF's ParserCache database setup for Multi DC as Declined.Thu, Jul 21, 7:48 AM

Ladsgroup added a subscriber: Ladsgroup.Thu, Jul 21, 12:59 PM