Page MenuHomePhabricator

[OSM] perform imposm3 migration in production
Closed, ResolvedPublic

Description

Background information

The final step for the imposm3 migration in production, planned for beginning of Q3/20-21.

How (imposm3 migration plan)

pre-import checks

  • Run REINDEX for the database in order to decrease disk space and remove dangling indexes
  • Is there enough free space?

Database will have a fresh start in maps1009.eqiad.wmnet

executing the import

This is a checklist for the imposm3 migration. Because it's the first time executing this functionality in production, we need SRE supervision through all the steps. The proposal migration plan avoids the need for depool and repool on the production machines because it doesn't delete data and have a rollback mechanism.

  • merge needed puppet rules https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482
  • apply new puppet rules for maps::osm_master
  • enable imposm parameter in hieradata
  • disable disable_replication_service and disable_tile_generation_cron in hieradata
  • execute initial-imposm-import script using the following command
# change -d and -s values accordingly when performing the import with the most updated values
sudo -u osmupdater imposm-initial-import \
   -d 201019 \ latest dump
   -s  https://planet.openstreetmap.org/replication/hour/000/071/112.state.txt \ hourly replication
   -x webproxy.eqiad.wmnet:8080
  • monitor the full planet import
    • Is there any disk space issue?
    • Did the script finished properly?
    • Are the logs sufficient?

When the import is done, the data won't be available in production, the data will be in the import schema

  • execute check scripts?
  • deploy data to the public schema by running:
sudo -u osmupdater imposm-deploy-import
  • monitor and check the APIs, if needed rollback with the following command
sudo -u osmupdater imposm-rollback-import
  • re-enable the replication service and check for its health
  • re-enable the tile-generation cron and check for its health
  • clean-up old osm2pgsql data running the following SQL script
sudo -u postgres psql -f /etc/imposm/cleanup-old-osm2pgsql-tables.sql
sudo -u osmupdater imposm-removebackup-import

Open questions

  • Is it possible to execute the import without deleting current data? Is there enough disk space for that?
  • How much SRE intervention will be needed?

Acceptance criteria

  • imposm3 replaces osm2pgsql in the production environment

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+40 -241
operations/puppetproduction+0 -9
operations/puppetproduction+3 -1
operations/puppetproduction+32 -145
operations/puppetproduction+57 -30
operations/puppetproduction+0 -3
operations/puppetproduction+2 -0
operations/puppetproduction+37 -6
operations/puppetproduction+34 -6
operations/puppetproduction+0 -15
operations/puppetproduction+30 -5
operations/puppetproduction+37 -6
operations/puppetproduction+28 -6
operations/puppetproduction+35 -6
operations/puppetproduction+9 -0
operations/puppetproduction+38 -7
operations/puppetproduction+33 -10
operations/puppetproduction+39 -6
operations/puppetproduction+23 -7
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedhnowlan
Opensdkim
Resolvedhnowlan
OpenMSantos
ResolvedMSantos
ResolvedMSantos
ResolvedMSantos
ResolvedMSantos
OpenMSantos
OpenMSantos
OpenBUG REPORTMSantos
OpenMSantos
OpenBUG REPORTJgiannelos
OpenNone
OpenNone
ResolvedMSantos
Resolveddcaro
Resolvedhnowlan
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

The puppet changes introduced in https://gerrit.wikimedia.org/r/c/operations/puppet/+/644482 are breaking the puppet runs on the clouddb host, can you advice (T276039)?

Change 696418 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: make maps2009 a buster imposm-based master in codfw

https://gerrit.wikimedia.org/r/696418

Change 696418 merged by Hnowlan:

[operations/puppet@production] maps: make maps2009 a buster imposm-based master in codfw

https://gerrit.wikimedia.org/r/696418

Script wmf-auto-reimage was launched by hnowlan on cumin2001.codfw.wmnet for hosts:

['maps2009.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106091536_hnowlan_32077.log.

Completed auto-reimage of hosts:

['maps2009.codfw.wmnet']

and were ALL successful.

Change 699044 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] osm: create missing imposm directories

https://gerrit.wikimedia.org/r/699044

Change 699782 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: make maps1007 a buster replica of the new imposm cluster

https://gerrit.wikimedia.org/r/699782

Change 699782 merged by Hnowlan:

[operations/puppet@production] maps: make maps1007 a buster replica of the new imposm cluster

https://gerrit.wikimedia.org/r/699782

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['maps1007.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106161020_hnowlan_29876.log.

Completed auto-reimage of hosts:

['maps1007.eqiad.wmnet']

Of which those FAILED:

['maps1007.eqiad.wmnet']

Change 700087 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: make maps2007 a buster replica of maps2009

https://gerrit.wikimedia.org/r/700087

Change 699044 merged by Hnowlan:

[operations/puppet@production] osm: create missing imposm directories, add mirror support to import

https://gerrit.wikimedia.org/r/699044

Change 700087 merged by Hnowlan:

[operations/puppet@production] maps: make maps2007 a buster replica of maps2009

https://gerrit.wikimedia.org/r/700087

Script wmf-auto-reimage was launched by hnowlan on cumin2002.codfw.wmnet for hosts:

['maps2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106281007_hnowlan_3226701.log.

Completed auto-reimage of hosts:

['maps2007.codfw.wmnet']

and were ALL successful.

Change 702102 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: make maps1008 a buster replica of maps1009

https://gerrit.wikimedia.org/r/702102

Change 702615 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: make maps2010 a buster replica of maps2009

https://gerrit.wikimedia.org/r/702615

Change 702619 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: reimage maps1010 as buster replica of maps1009

https://gerrit.wikimedia.org/r/702619

Change 702687 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: standardise the maps2.0 config in codfw, remove old nodes

https://gerrit.wikimedia.org/r/702687

Change 702984 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: standardised the maps2.0 config in eqiad, remove old nodes

https://gerrit.wikimedia.org/r/702984

Change 704296 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: disable OSM sync and tilerator in codfw

https://gerrit.wikimedia.org/r/704296

Script wmf-auto-reimage was launched by hnowlan on cumin2002.codfw.wmnet for hosts:

['maps2008.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202107141122_hnowlan_1582736.log.

Completed auto-reimage of hosts:

['maps2008.codfw.wmnet']

and were ALL successful.

Change 704827 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: disable tilerator in the new cluster

https://gerrit.wikimedia.org/r/704827

Change 704827 merged by Hnowlan:

[operations/puppet@production] maps: disable tilerator in the new cluster

https://gerrit.wikimedia.org/r/704827

Change 705684 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: reenable tilerator on codfw new cluster

https://gerrit.wikimedia.org/r/705684

Change 702615 merged by Hnowlan:

[operations/puppet@production] maps: make maps2010 a buster replica of maps2009

https://gerrit.wikimedia.org/r/702615

Script wmf-auto-reimage was launched by hnowlan on cumin2002.codfw.wmnet for hosts:

['maps2010.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202107221608_hnowlan_2963454.log.

Completed auto-reimage of hosts:

['maps2010.codfw.wmnet']

and were ALL successful.

Change 702102 merged by Hnowlan:

[operations/puppet@production] maps: make maps1008 a buster replica of maps1009

https://gerrit.wikimedia.org/r/702102

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['maps1008.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108050919_hnowlan_5261.log.

Completed auto-reimage of hosts:

['maps1008.eqiad.wmnet']

and were ALL successful.

Change 710231 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: reimage maps2006 as a buster replica of maps2009

https://gerrit.wikimedia.org/r/710231

Change 710234 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: reimage maps2005 as buster replica of maps2009

https://gerrit.wikimedia.org/r/710234

Change 710231 merged by Hnowlan:

[operations/puppet@production] maps: reimage maps2006 as a buster replica of maps2009

https://gerrit.wikimedia.org/r/710231

Script wmf-auto-reimage was launched by hnowlan on cumin2002.codfw.wmnet for hosts:

['maps2006.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108051353_hnowlan_1044416.log.

Change 702619 merged by Hnowlan:

[operations/puppet@production] maps: reimage maps1010 as buster replica of maps1009

https://gerrit.wikimedia.org/r/702619

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['maps1010.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108051402_hnowlan_1737.log.

Completed auto-reimage of hosts:

['maps2006.codfw.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['maps1010.eqiad.wmnet']

and were ALL successful.

Change 710286 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: reenable tilerator in imposm cluster

https://gerrit.wikimedia.org/r/710286

Change 710286 merged by Hnowlan:

[operations/puppet@production] maps: reenable tilerator in imposm cluster

https://gerrit.wikimedia.org/r/710286

Change 710509 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: make maps1006 a buster replica of maps1009

https://gerrit.wikimedia.org/r/710509

Change 710509 merged by Hnowlan:

[operations/puppet@production] maps: make maps1006 a buster replica of maps1009

https://gerrit.wikimedia.org/r/710509

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['maps1006.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108061059_hnowlan_17309.log.

Completed auto-reimage of hosts:

['maps1006.eqiad.wmnet']

and were ALL successful.

Change 710234 merged by Hnowlan:

[operations/puppet@production] maps: reimage maps2005 as buster replica of maps2009

https://gerrit.wikimedia.org/r/710234

Change 710544 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: add missing postgres entry for maps2005

https://gerrit.wikimedia.org/r/710544

Change 710544 merged by Hnowlan:

[operations/puppet@production] maps: add missing postgres entry for maps2005

https://gerrit.wikimedia.org/r/710544

Script wmf-auto-reimage was launched by hnowlan on cumin2002.codfw.wmnet for hosts:

['maps2005.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108061407_hnowlan_1214654.log.

Completed auto-reimage of hosts:

['maps2005.codfw.wmnet']

and were ALL successful.

Change 710582 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: reimage maps1005 as buster imposm replica

https://gerrit.wikimedia.org/r/710582

Change 710591 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] maps: reenable tilerator on maps2005

https://gerrit.wikimedia.org/r/710591

Change 710591 merged by Hnowlan:

[operations/puppet@production] maps: reenable tilerator on maps2005

https://gerrit.wikimedia.org/r/710591

Change 710582 merged by Hnowlan:

[operations/puppet@production] maps: reimage maps1005 as buster imposm replica

https://gerrit.wikimedia.org/r/710582

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['maps1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108090929_hnowlan_14581.log.

Completed auto-reimage of hosts:

['maps1005.eqiad.wmnet']

and were ALL successful.

Change 702984 abandoned by Hnowlan:

[operations/puppet@production] maps: standardised the maps2.0 config in eqiad, remove old nodes

Reason:

Superseded by another change

https://gerrit.wikimedia.org/r/702984

Change 704296 abandoned by Hnowlan:

[operations/puppet@production] maps: disable OSM sync and tilerator in codfw

Reason:

No longer needed

https://gerrit.wikimedia.org/r/704296

Change 705684 abandoned by Hnowlan:

[operations/puppet@production] maps: reenable tilerator on codfw new cluster

Reason:

https://gerrit.wikimedia.org/r/705684

Change 702687 merged by Hnowlan:

[operations/puppet@production] maps: standardise the maps2.0 config in codfw, remove old nodes

https://gerrit.wikimedia.org/r/702687