Page MenuHomePhabricator

Document infrastructure considerations
Closed, ResolvedPublic

Description

As we move towards implementing the new solution, we should reappraise our storage requirements. Currently we have 9 Postgres nodes and 9 Cassandra instances in each datacentre. This is almost certainly overkill and we should reexamine the capacity we require, and how those hosts are configured.

We should also examine capacity requirements for moving to Redis - can we simply migrate to using the RDB clusters in existence or will our significant volume of data cause issues?

Acceptance Criteria

Event Timeline

After some benchmarking on the tegola deployment on k8s it doesn't look like we have any sort of capacity problems. Here is our setup:

  • We built a karotherian module to be able to mirror traffic to different backends
  • We gradually increased from 0% to 100% of the production traffic mirrored from eqiad kartotherian to codfw tegola

We didn't find any latency/io/load issue to any component:

  • Postgres (maps2005..2010)
  • Swift (thanos)
  • Kubernetes

Some findings regarding resource usage:

  • We removed the dependency to Cassandra and we now rely on an existing Swift cluster (thanos).
  • Based on the load testing i don' think we can reduce resources on Postgres nodes.
  • The new maps cluster consists of less machines compared to the cluster we decommisioned.

Given that we already have some docs around infrastructure for the new architecture I would consider this ticket resolved.