Maniphest T321885

Resync stale maps postgres replicas
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	awight
	Oct 28 2022, 11:25 AM

Tags

Referenced Files

None

Subscribers

Tokens

"Barnstar" token, awarded by awight.

Description

Three (of five) of our maps server nodes have database contents which haven't been replicated since roughly March 2022: graph. Let's restore these servers by depooling and resyncing with the master, one at a time.

The affected servers are:

maps2005.codfw.wmnet
maps2006.codfw.wmnet
maps2008.codfw.wmnet

A script already exists to do the postgres part of the work: resync_replica.sh

We think this fix might last for some time, since the other good servers have maintained up-to-date replicas for as long as we have metrics in grafana (since late June 2022).

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T321959 Tech Wishes - Maps service infrastructure deprecations
		Resolved		None	T316365 Restore the map data health and parity between clusters
		Resolved		jijiki	T321885 Resync stale maps postgres replicas

Event Timeline

awight created this task.Oct 28 2022, 11:25 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2022, 11:25 AM

lilients_WMDE moved this task from Backlog to In sprint on the WMDE-GeoInfo-FocusArea board.Nov 1 2022, 1:41 PM

awight moved this task from Sprint Backlog to Watching / Epic / Stalled on the WMDE-TechWish-Sprint-2022-10-26 board.Nov 4 2022, 12:28 PM

WMDE-Fisch added a project: WMDE-TechWish-Sprint-2022-11-09.Nov 9 2022, 12:39 PM

Lena_WMDE moved this task from Sprint Backlog to Watching / Epic / Stalled on the WMDE-TechWish-Sprint-2022-11-09 board.Nov 9 2022, 12:43 PM

TheDJ mentioned this in T322104: Variable blue tiles in Ethiopia.Nov 14 2022, 3:32 PM

See also the longer-term fix T290149: Configure replication slots on Postgres masters.

@hnowlan synced all replicas, I believe we can mark this as done

Amazing, thank you! I can confirm that the postgres replica sync lag graphs have dropped to 0 for codfw.

awight awarded a token.Nov 21 2022, 8:59 AM