Change Details

= Background information Final step for the imposm3 migration in production = How (imposm3 migration plan) == pre-import checks - [ ] Run REINDEX for the database in order to decrease disk space and remove dangling indexes - [ ] Is there enough free space? - Still needs checking, we don't know yet how much free space is needed to perform the initial import with imposm == executing the import This is a checklist for the imposm3 migration. Because it's the first time executing this functionality in production, we need SRE supervision through all the steps. The proposal migration plan avoids the need for depool and repool on the production machines because it doesn't delete data and have a rollback mechanism. - [ ] merge needed puppet rules https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482 - [ ] apply new puppet rules for maps::osm_master - [ ] enable `imposm` parameter in hieradata - [ ] disable `disable_replication_service` and `disable_tile_generation_cron` in hieradata - [ ] execute `initial-imposm-import` script using the following command ``` # change -d and -s values accordingly when performing the import with the most updated values sudo -u osmupdater imposm-initial-import \ -d 201019 \ latest dump -s https://planet.openstreetmap.org/replication/hour/000/071/112.state.txt \ hourly replication -x webproxy.eqiad.wmnet:8080 ``` - [ ] monitor the full planet import - Is there any disk space issue? - Did the script finished properly? - Are the logs sufficient? > When the import is done, the data won't be available in production, the data will be in the import schema - [ ] execute check scripts? - [ ] deploy data to the public schema by running: ``` sudo -u osmupdater imposm-deploy-import ``` - [ ] monitor and check the APIs, if needed rollback with the following command ``` sudo -u osmupdater imposm-rollback-import ``` - [ ] re-enable the replication service and check for its health - [ ] re-enable the tile-generation cron and check for its health - [ ] clean-up old osm2pgsql data running the following SQL script ``` sudo -u postgres psql -f /etc/imposm/cleanup-old-osm2pgsql-tables.sql sudo -u osmupdater imposm-removebackup-import ``` = Open questions - Is it possible to execute the import without deleting current data? Is there enough disk space for that? - How much SRE intervention will be needed? = Acceptance criteria - [] imposm3 replaces osm2pgsql in the production environment

= Background information The final step for the imposm3 migration in production, planned for beginning of Q3/20-21. = How (imposm3 migration plan) == pre-import checks - [ ] Run REINDEX for the database in order to decrease disk space and remove dangling indexes - [ ] Is there enough free space? - Still needs checking, we don't know yet how much free space is needed to perform the initial import with imposm == executing the import This is a checklist for the imposm3 migration. Because it's the first time executing this functionality in production, we need SRE supervision through all the steps. The proposal migration plan avoids the need for depool and repool on the production machines because it doesn't delete data and have a rollback mechanism. - [ ] merge needed puppet rules https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482 - [ ] apply new puppet rules for maps::osm_master - [ ] enable `imposm` parameter in hieradata - [ ] disable `disable_replication_service` and `disable_tile_generation_cron` in hieradata - [ ] execute `initial-imposm-import` script using the following command ``` # change -d and -s values accordingly when performing the import with the most updated values sudo -u osmupdater imposm-initial-import \ -d 201019 \ latest dump -s https://planet.openstreetmap.org/replication/hour/000/071/112.state.txt \ hourly replication -x webproxy.eqiad.wmnet:8080 ``` - [ ] monitor the full planet import - Is there any disk space issue? - Did the script finished properly? - Are the logs sufficient? > When the import is done, the data won't be available in production, the data will be in the import schema - [ ] execute check scripts? - [ ] deploy data to the public schema by running: ``` sudo -u osmupdater imposm-deploy-import ``` - [ ] monitor and check the APIs, if needed rollback with the following command ``` sudo -u osmupdater imposm-rollback-import ``` - [ ] re-enable the replication service and check for its health - [ ] re-enable the tile-generation cron and check for its health - [ ] clean-up old osm2pgsql data running the following SQL script ``` sudo -u postgres psql -f /etc/imposm/cleanup-old-osm2pgsql-tables.sql sudo -u osmupdater imposm-removebackup-import ``` = Open questions - Is it possible to execute the import without deleting current data? Is there enough disk space for that? - How much SRE intervention will be needed? = Acceptance criteria - [] imposm3 replaces osm2pgsql in the production environment

= Background information FThe final step for the imposm3 migration in production, planned for beginning of Q3/20-21. = How (imposm3 migration plan) == pre-import checks - [ ] Run REINDEX for the database in order to decrease disk space and remove dangling indexes - [ ] Is there enough free space? - Still needs checking, we don't know yet how much free space is needed to perform the initial import with imposm == executing the import This is a checklist for the imposm3 migration. Because it's the first time executing this functionality in production, we need SRE supervision through all the steps. The proposal migration plan avoids the need for depool and repool on the production machines because it doesn't delete data and have a rollback mechanism. - [ ] merge needed puppet rules https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482 - [ ] apply new puppet rules for maps::osm_master - [ ] enable `imposm` parameter in hieradata - [ ] disable `disable_replication_service` and `disable_tile_generation_cron` in hieradata - [ ] execute `initial-imposm-import` script using the following command ``` # change -d and -s values accordingly when performing the import with the most updated values sudo -u osmupdater imposm-initial-import \ -d 201019 \ latest dump -s https://planet.openstreetmap.org/replication/hour/000/071/112.state.txt \ hourly replication -x webproxy.eqiad.wmnet:8080 ``` - [ ] monitor the full planet import - Is there any disk space issue? - Did the script finished properly? - Are the logs sufficient? > When the import is done, the data won't be available in production, the data will be in the import schema - [ ] execute check scripts? - [ ] deploy data to the public schema by running: ``` sudo -u osmupdater imposm-deploy-import ``` - [ ] monitor and check the APIs, if needed rollback with the following command ``` sudo -u osmupdater imposm-rollback-import ``` - [ ] re-enable the replication service and check for its health - [ ] re-enable the tile-generation cron and check for its health - [ ] clean-up old osm2pgsql data running the following SQL script ``` sudo -u postgres psql -f /etc/imposm/cleanup-old-osm2pgsql-tables.sql sudo -u osmupdater imposm-removebackup-import ``` = Open questions - Is it possible to execute the import without deleting current data? Is there enough disk space for that? - How much SRE intervention will be needed? = Acceptance criteria - [] imposm3 replaces osm2pgsql in the production environment