Re-import full planet data into maps1009 and then maps2009
Process documented at Run a planet import
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T321959 Tech Wishes - Maps service infrastructure deprecations | |||
| Resolved | None | T316365 Restore the map data health and parity between clusters | |||
| Resolved | None | T323920 Deploy recent map server work to production ( Jan 2023 ) | |||
| Resolved | jijiki | T314472 Re-import full planet data into eqiad and codfw | |||
| Resolved | Jgiannelos | T324459 Enable traffic mirroring from codfw to eqiad | |||
| Resolved | jijiki | T325309 High error rate on maps.wikimedia.org |
Hi @Jgiannelos,
Which of the above steps would require an SRE to intervene? Merging puppet changes is one that comes to mind, but others may too?
One question that does come to mind is whether the machine have enough disk space to have both schemas around. A quick
ssh maps1009.eqiad.wmnet df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-root 73G 37G 33G 54% / /dev/mapper/vg0-srv 2.7T 2.0T 591G 78% /srv
appears to suggest that no?
I think that eqiad is left in a stale state and we never deleted the backup schema after an issue we had with the import.
On maps1009:
schema_name | pg_size_pretty --------------------+---------------- import | 728 GB information_schema | 96 kB pg_catalog | 9000 kB pg_toast | 95 GB public | 805 GB (5 rows)
and maps2009:
schema_name | pg_size_pretty --------------------+---------------- information_schema | 96 kB pg_catalog | 7584 kB pg_toast | 53 GB public | 800 GB
On maps1009 there are 700G of import data that are going to be replaced on next import, so i think we are good with storage.
Regarding SRE support, planet import is probably the most common task we do for maps maintenance so I think it might be a good opportunity for folks that are or will be involved with maps operation to get familiarized with. I was thinking that it can be a good example of knowledge transfer so we can do it together (me or @MSantos) with somebody from serviceops.
Mentioned in SAL (#wikimedia-operations) [2022-11-30T15:33:58Z] <effie> jiji@maps1009 imposm-removebackup-import - T314472
@Jgiannelos and I have successfully completed re-imported full planet data on eqiad. Next up, we are working on ways we can warm up eqiad's tile cache; we are leaning towards mirroring a percentage of codfw's (live) traffic to eqiad, which will trigger tile creation
Change 866377 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/deployment-charts@master] tegola-vector-tiles: use new tegola swift container in eqiad
Change 866379 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] hieradata: enable maps replication and tile_generation timers
Change 866377 merged by Effie Mouzeli:
[operations/deployment-charts@master] tegola-vector-tiles: use new tegola swift container in eqiad
Change 866379 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable maps replication and tile_generation timers
Change 866442 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[operations/puppet@production] maps: Use new swift container for eqiad pregeneration
Change 866442 merged by Effie Mouzeli:
[operations/puppet@production] maps: Use new swift container for eqiad pregeneration
Change 868343 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/deployment-charts@master] tegola-vector-tiles: add more replicas
Change 868343 merged by jenkins-bot:
[operations/deployment-charts@master] tegola-vector-tiles: add more replicas
Change 879556 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] hieradata: disable maps tile_generation timers for planet import
Change 879556 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: disable maps tile_generation timers for planet import
Mentioned in SAL (#wikimedia-operations) [2023-01-12T15:28:47Z] <effie> Planet import in codfw (on maps2009) started at 15:26 UTC - T314472
Change 880933 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/deployment-charts@master] tegola-vector-tiles: use new tegola swift container in codfw
Change 880934 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/deployment-charts@master] tegola-vector-tiles: disable tile pregeneration in codfw
Change 880933 merged by jenkins-bot:
[operations/deployment-charts@master] tegola-vector-tiles: use new tegola swift container in codfw
Change 880934 merged by jenkins-bot:
[operations/deployment-charts@master] tegola-vector-tiles: disable tile pregeneration in codfw
Change 881597 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/deployment-charts@master] tegola-vector-tiles: tile pregeneration and replicas bump
Change 881599 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/puppet@production] hieradata: enable maps timers in codfw
Change 881599 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable maps timers in codfw
Change 881597 merged by jenkins-bot:
[operations/deployment-charts@master] tegola-vector-tiles: tile pregeneration and replicas bump
Import to codfw has been completed, and we have bootstrapped its tile storage using https://gerrit.wikimedia.org/r/c/operations/puppet/+/875973
Lots hope it’s finally more predictable from this point forward. Thank you for the perseverance team!