Background information
The final step for the imposm3 migration in production, planned for beginning of Q3/20-21.
How (imposm3 migration plan)
pre-import checks
- Run REINDEX for the database in order to decrease disk space and remove dangling indexes
- Is there enough free space?
Database will have a fresh start in maps1009.eqiad.wmnet
executing the import
This is a checklist for the imposm3 migration. Because it's the first time executing this functionality in production, we need SRE supervision through all the steps. The proposal migration plan avoids the need for depool and repool on the production machines because it doesn't delete data and have a rollback mechanism.
- merge needed puppet rules https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482
- apply new puppet rules for maps::osm_master
- enable imposm parameter in hieradata
- disable disable_replication_service and disable_tile_generation_cron in hieradata
- execute initial-imposm-import script using the following command
# change -d and -s values accordingly when performing the import with the most updated values sudo -u osmupdater imposm-initial-import \ -d 201019 \ latest dump -s https://planet.openstreetmap.org/replication/hour/000/071/112.state.txt \ hourly replication -x webproxy.eqiad.wmnet:8080
- monitor the full planet import
- Is there any disk space issue?
- Did the script finished properly?
- Are the logs sufficient?
When the import is done, the data won't be available in production, the data will be in the import schema
- execute check scripts?
- deploy data to the public schema by running:
sudo -u osmupdater imposm-deploy-import
- monitor and check the APIs, if needed rollback with the following command
sudo -u osmupdater imposm-rollback-import
- re-enable the replication service and check for its health
- re-enable the tile-generation cron and check for its health
- clean-up old osm2pgsql data running the following SQL script
sudo -u postgres psql -f /etc/imposm/cleanup-old-osm2pgsql-tables.sql sudo -u osmupdater imposm-removebackup-import
Open questions
- Is it possible to execute the import without deleting current data? Is there enough disk space for that?
- How much SRE intervention will be needed?
Acceptance criteria
- imposm3 replaces osm2pgsql in the production environment