Page MenuHomePhabricator

Resync stale maps postgres replicas
Closed, ResolvedPublic


Three (of five) of our maps server nodes have database contents which haven't been replicated since roughly March 2022: graph. Let's restore these servers by depooling and resyncing with the master, one at a time.

The affected servers are:

  • maps2005.codfw.wmnet
  • maps2006.codfw.wmnet
  • maps2008.codfw.wmnet

A script already exists to do the postgres part of the work:

We think this fix might last for some time, since the other good servers have maintained up-to-date replicas for as long as we have metrics in grafana (since late June 2022).

Event Timeline

jijiki claimed this task.
jijiki added a project: serviceops.
jijiki added subscribers: hnowlan, jijiki.

@hnowlan synced all replicas, I believe we can mark this as done

Amazing, thank you! I can confirm that the postgres replica sync lag graphs have dropped to 0 for codfw.