Page MenuHomePhabricator

Re-import full planet data into codfw
Open, Needs TriagePublic

Description

Background information

Re-import full planet data into maps2009.

executing the import

  • disable disable_replication_cron and disable_tile_generation_cron in hieradata
  • execute initial-imposm-import script using the following command
# change -d value accordingly when performing the import with the most updated values
# from https://planet.openstreetmap.org/pbf/ -> latest planet date
# this step should take ~13 hours and potentially trigger WAL errors
sudo -s
screen -DR
imposm-initial-import -d 220530 -x webproxy.codfw.wmnet:8080
  • Post a log message in IRC #wikimedia-operations !log maintenance: trigger full planet re-import for maps codfw
  • monitor the full planet import
    • Did the script finished properly?
    • Are the logs sufficient?

When the import is done, the data won't be available in production, the data will be in the import schema

  • deploy data to the public schema by running:
sudo -u osmupdater imposm-deploy-import
  • REINDEX DATABASE gis;
  • monitor and check the APIs, if needed rollback with the following command
sudo -u osmupdater imposm-rollback-import
  • check if sql functions and indexes are applied (if not apply them manually)
  • re-enable the replication service and check for its health
  • re-enable the tile-generation cron and check for its health
  • clean-up old data in backup
    • Post a log message in IRC #wikimedia-operations !log <your_username>@maps2009 imposm-removebackup-import (T305845)
imposm-removebackup-import

Open questions

  • How much SRE intervention will be needed?

Acceptance criteria

  • DB is refreshed with most updated data

Event Timeline

Jgiannelos updated the task description. (Show Details)
Jgiannelos updated the task description. (Show Details)

What is the status of this? I see that replication has been disabled for a while on both codfw and eqiad. I suspect that some user-facing components of it haven't been properly updated since January when I tagged e.g. this OSM object that still isn't available via geoshape. Is there some more general task about this?

MSantos updated the task description. (Show Details)
MSantos updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2022-06-21T11:59:04Z] <mbsantos> mbsantos@maps2009 imposm-removebackup-import (T305845)

Change 807108 had a related patch set uploaded (by MSantos; author: MSantos):

[operations/puppet@production] maps: re-enable tile generation cron in codfw

https://gerrit.wikimedia.org/r/807108

Next steps to failover to codfw:

  • Re-enable kubernetes cronjobs for codfw env
  • Reset kafka offsets to start pregeneration with fresh events
  • Bootstrap with tiles from current swift container
    • Extract tiles
    • Create tileset
    • Send events
  • Merge patch from https://phabricator.wikimedia.org/T307182
  • Re-enable tile pregeneration cron

Change 807561 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/deployment-charts@master] tegola: Re-enable tile pregeneration on codfw

https://gerrit.wikimedia.org/r/807561

Change 807567 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/deployment-charts@master] tegola: Point codfw to a new swift container

https://gerrit.wikimedia.org/r/807567

Change 807561 merged by jenkins-bot:

[operations/deployment-charts@master] tegola: Re-enable tile pregeneration on codfw

https://gerrit.wikimedia.org/r/807561

Change 807567 merged by jenkins-bot:

[operations/deployment-charts@master] tegola: Point codfw to a new swift container

https://gerrit.wikimedia.org/r/807567