Page MenuHomePhabricator
Create Task
Maniphest T198622

migrate maps servers to stretch with the current style
Closed, ResolvedPublic
Actions

Description

Since the new maps style is delayed, we need to start working on the migration to stretch with the current style.

Migration plan:

  • prepare puppet change to isolate maps2004 from current cluster
  • reduce cassandra replication to allow running on a 3 nodes cluster
  • depool maps2004
  • install new disks in maps2004
  • reimage maps2004 as stretch, with new partition scheme
  • initial OSM import on maps2004
  • regenerate tiles on maps2004
  • wait tile generation (probably a long, long time)
  • validate that everything looks good
  • depool maps2003
  • reimage maps2003 as maps slave
  • increase cassandra replication on the new cluster
  • validate
  • depool maps200[12], pool maps200[34]

Notes:

  • since this upgrade does not bring a new style, the tiles generated by both new and old install should be equivalent, having tiles served from both stretch and jessie, from the same DC or from different DC should not be an issue.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Mholloway added subtasks: T198485: Verify loading instructions for existing styles on Stretch, T172090: move maps-test cluster to cloud infrastructure.Jul 23 2018, 7:31 PM
• Mholloway changed the status of subtask T198485: Verify loading instructions for existing styles on Stretch from Stalled to Open.
LGoto added a project: Product-Infrastructure-Team-Backlog-Deprecated.Jul 25 2018, 3:41 PM
• Jhernandez moved this task from Needs triage to Upcoming on the Product-Infrastructure-Team-Backlog-Deprecated board.Aug 1 2018, 3:36 PM
• Mholloway closed subtask T198485: Verify loading instructions for existing styles on Stretch as Resolved.Aug 2 2018, 7:08 PM
• Mholloway added a comment.Aug 2 2018, 7:19 PM

Now that the Beta Cluster is back in good working order with both Jessie (deployment-maps03) and Stretch (deployment-maps04) instances, I think we're ready to go forward with this. @Gehel, how soon do you think this could happen?

@MSantos is currently working on finishing the dependency updates to use Mapnik 3.7.2 (T188674) and on merging the 'stretch' branches of the -package and -deploy repos back into master. That means our next production deployment will be after the move to Stretch.

All that remains to be done for T172090 is to decom the maps-test cluster. That can be done anytime. Should it wait until after the migration to stretch?

• Mholloway moved this task from In progress to To-do on the Maps-Sprint board.Aug 2 2018, 7:37 PM
Gehel updated the task description. (Show Details)Aug 20 2018, 12:13 PM
• Mholloway mentioned this in T152196: Kartotherian snapshots always return low DPI images.Aug 22 2018, 8:46 PM
• Mholloway mentioned this in T186732: Decide on Cache-Control headers for map tiles.Aug 22 2018, 8:51 PM
• Jhernandez added a parent task: T152196: Kartotherian snapshots always return low DPI images.Aug 23 2018, 4:49 PM
• Jhernandez closed subtask T172090: move maps-test cluster to cloud infrastructure as Resolved.Aug 23 2018, 4:54 PM
gerritbot subscribed.Sep 3 2018, 10:18 AM

Change 457408 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: migrate maps2004 to stretch

https://gerrit.wikimedia.org/r/457408

gerritbot added a project: Patch-For-Review.Sep 3 2018, 10:18 AM
gerritbot added a comment.Sep 10 2018, 11:32 AM

Change 459535 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: migrate maps1004 to stretch

https://gerrit.wikimedia.org/r/459535

MSantos moved this task from To-do to In progress on the Maps-Sprint board.Sep 12 2018, 5:11 PM
• Mholloway mentioned this in T109776: Tilerator should purge Varnish cache.Sep 17 2018, 2:29 PM
• Mholloway assigned this task to Gehel.Sep 24 2018, 5:53 AM
gerritbot added a comment.Sep 25 2018, 8:22 AM

Change 459535 merged by Gehel:
[operations/puppet@production] maps: migrate maps1004 to stretch

https://gerrit.wikimedia.org/r/459535

ops-monitoring-bot subscribed.Sep 25 2018, 11:59 AM

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201809251159_gehel_3263.log.

ops-monitoring-bot added a comment.Sep 25 2018, 11:59 AM

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201809251159_gehel_3447.log.

gerritbot added a comment.Sep 25 2018, 1:22 PM

Change 462702 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: migrate maps1004 to stretch

https://gerrit.wikimedia.org/r/462702

gerritbot added a comment.Sep 25 2018, 1:25 PM

Change 462702 merged by Gehel:
[operations/puppet@production] maps: migrate maps1004 to stretch

https://gerrit.wikimedia.org/r/462702

ops-monitoring-bot added a comment.Sep 25 2018, 1:30 PM

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201809251330_gehel_24458.log.

ops-monitoring-bot added a comment.Sep 25 2018, 2:32 PM

Completed auto-reimage of hosts:

['maps1004.eqiad.wmnet']

and were ALL successful.

MSantos removed a project: Maps-Sprint.Sep 25 2018, 7:29 PM
MSantos moved this task from Upcoming to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.
• Mholloway added a subtask: T205462: Load OSM data into maps1004.Oct 2 2018, 3:52 AM
gerritbot added a comment.Jan 11 2019, 4:24 PM

Change 483798 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps: migrate maps1003 to stretch

https://gerrit.wikimedia.org/r/483798

Stashbot subscribed.Jan 17 2019, 1:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-17T13:38:28Z] <gehel> starting upgrade to stretch for maps1003 - T198622

gerritbot added a comment.Jan 17 2019, 1:42 PM

Change 483798 merged by Gehel:
[operations/puppet@production] maps: migrate maps1003 to stretch

https://gerrit.wikimedia.org/r/483798

Stashbot added a comment.Jan 17 2019, 2:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-17T14:01:11Z] <gehel> pooling maps1004 (first time after stretch upgrade) - T198622

ops-monitoring-bot added a comment.Jan 17 2019, 2:18 PM

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['maps1003.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201901171418_gehel_162898.log.

ops-monitoring-bot added a comment.Jan 17 2019, 2:51 PM

Completed auto-reimage of hosts:

['maps1003.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jan 17 2019, 5:55 PM

Change 485072 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps: change cassandra version

https://gerrit.wikimedia.org/r/485072

gerritbot added a comment.Jan 17 2019, 6:22 PM

Change 485072 merged by Gehel:
[operations/puppet@production] maps: change cassandra version

https://gerrit.wikimedia.org/r/485072

gerritbot added a comment.Jan 18 2019, 9:28 AM

Change 485164 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] kartotherian: install nodejs-legacy module

https://gerrit.wikimedia.org/r/485164

• Mathew.onipe mentioned this in T214153: Fix node vs nodejs dependency issue.Jan 18 2019, 10:16 AM
gerritbot added a comment.Jan 18 2019, 10:57 AM

Change 485164 merged by Gehel:
[operations/puppet@production] kartotherian: install nodejs-legacy module

https://gerrit.wikimedia.org/r/485164

gerritbot added a comment.Jan 18 2019, 3:05 PM

Change 485192 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps: increase wal_size for postgres 9.6 on stretch

https://gerrit.wikimedia.org/r/485192

gerritbot added a comment.Jan 18 2019, 3:55 PM

Change 485192 merged by Gehel:
[operations/puppet@production] maps: increase wal_size for postgres 9.6 on stretch

https://gerrit.wikimedia.org/r/485192

Stashbot added a comment.Jan 19 2019, 12:34 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-19T12:34:43Z] <onimisionipe> pool maps1003 - stretch migration is complete T198622

gerritbot added a comment.Jan 21 2019, 7:54 AM

Change 485584 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps: migrate maps1002 to stretch

https://gerrit.wikimedia.org/r/485584

Pikne mentioned this in T214350: mapframe dynamic maps (maplink) don't always work.Jan 22 2019, 8:52 AM
Stashbot added a comment.Jan 22 2019, 9:55 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-22T09:55:04Z] <gehel> repooling maps1003 after upgrade to stretch - T198622

Stashbot added a comment.Jan 22 2019, 12:39 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-22T12:39:58Z] <gehel> start stretch upgrade for maps1002 - T198622

gerritbot added a comment.Jan 22 2019, 12:57 PM

Change 485584 merged by Gehel:
[operations/puppet@production] maps: migrate maps1002 to stretch

https://gerrit.wikimedia.org/r/485584

ops-monitoring-bot added a comment.Jan 22 2019, 1:06 PM

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['maps1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201901221306_gehel_241795.log.

ops-monitoring-bot added a comment.Jan 22 2019, 1:46 PM

Completed auto-reimage of hosts:

['maps1002.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jan 22 2019, 2:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-22T14:45:48Z] <onimisionipe> starting init of postgres replication on maps1002 - T198622

• Mathew.onipe mentioned this in T214434: Kartotherian service on maps100[2-4] timed out on when trying to get tiles..Jan 22 2019, 8:40 PM
gerritbot added a comment.Jan 23 2019, 11:33 AM

Change 486062 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps: migrate maps1001 to stretch

https://gerrit.wikimedia.org/r/486062

Stashbot added a comment.Jan 24 2019, 10:37 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-24T10:37:23Z] <gehel> starting stretch upgrade on maps1001 - T198622

gerritbot added a comment.Jan 24 2019, 10:43 AM

Change 486062 merged by Gehel:
[operations/puppet@production] maps: migrate maps1001 to stretch

https://gerrit.wikimedia.org/r/486062

ops-monitoring-bot added a comment.Jan 24 2019, 10:47 AM

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['maps1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201901241047_gehel_58889.log.

ops-monitoring-bot added a comment.Jan 24 2019, 11:30 AM

Completed auto-reimage of hosts:

['maps1001.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jan 24 2019, 6:37 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-24T18:37:14Z] <onimisionipe> pooling maps1003 - stretch migration is complete. T198622

gerritbot added a comment.Jan 25 2019, 9:10 AM

Change 486436 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: re-enable OSM lag check

https://gerrit.wikimedia.org/r/486436

gerritbot added a comment.Jan 25 2019, 10:26 AM

Change 486436 merged by Gehel:
[operations/puppet@production] maps: re-enable OSM lag check

https://gerrit.wikimedia.org/r/486436

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:08 PM
gerritbot added a comment.Jan 31 2019, 3:33 PM

Change 487360 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps: migrate maps2004 to stretch

https://gerrit.wikimedia.org/r/487360

Gehel closed subtask T205462: Load OSM data into maps1004 as Resolved.Feb 7 2019, 3:53 PM
Stashbot added a comment.Feb 7 2019, 3:55 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-07T15:55:05Z] <gehel> starting reimage of maps2004 - T198622

gerritbot added a comment.Feb 7 2019, 3:59 PM

Change 487360 merged by Gehel:
[operations/puppet@production] maps: migrate maps2004 to stretch

https://gerrit.wikimedia.org/r/487360

Gehel added a subtask: T215521: Load OSM data into maps2004.Feb 7 2019, 4:03 PM
ops-monitoring-bot added a comment.Feb 7 2019, 4:07 PM

Script wmf-auto-reimage was launched by gehel on cumin2001.codfw.wmnet for hosts:

['maps2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201902071606_gehel_23617.log.

Stashbot added a comment.Feb 8 2019, 1:39 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-08T13:39:53Z] <onimisionipe> starting osm-initial-import for maps2004 which is the newly migrated to stretch master - T198622

gerritbot added a comment.Feb 18 2019, 1:38 AM

Change 491191 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps: migrate maps2003 to stretch

https://gerrit.wikimedia.org/r/491191

MSantos added a subtask: T219849: Tilerator crashed on maps200[1-3].codfw.wmnet.Apr 2 2019, 2:26 PM
gerritbot added a comment.Apr 9 2019, 9:20 AM

Change 491191 had a related patch set uploaded (by Gehel; owner: Mathew.onipe):
[operations/puppet@production] maps: migrate maps2003 to stretch

https://gerrit.wikimedia.org/r/491191

gerritbot added a comment.Apr 9 2019, 9:23 AM

Change 491191 merged by Gehel:
[operations/puppet@production] maps: migrate maps2003 to stretch

https://gerrit.wikimedia.org/r/491191

ops-monitoring-bot added a comment.Apr 9 2019, 9:25 AM

Script wmf-auto-reimage was launched by gehel on cumin2001.codfw.wmnet for hosts:

['maps2003.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201904090925_gehel_30748.log.

gerritbot added a comment.Apr 9 2019, 3:00 PM

Change 502514 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[maps/kartotherian/deploy@master] Insert maps2003 into stretch environment

https://gerrit.wikimedia.org/r/502514

Stashbot added a comment.Apr 10 2019, 7:12 AM

Mentioned in SAL (#wikimedia-operations) [2019-04-10T07:12:21Z] <onimisionipe> depooling maps200[34] to increase cassandra replication factor - T198622

gerritbot added a comment.Apr 10 2019, 8:13 AM

Change 502514 merged by Gehel:
[maps/kartotherian/deploy@master] Insert maps2003 into stretch environment

https://gerrit.wikimedia.org/r/502514

gerritbot added a comment.Apr 10 2019, 9:41 AM

Change 502768 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] maps migrate maps2002 to stretch

https://gerrit.wikimedia.org/r/502768

gerritbot added a comment.Apr 11 2019, 9:39 AM

Change 502768 merged by Gehel:
[operations/puppet@production] maps migrate maps2002 to stretch

https://gerrit.wikimedia.org/r/502768

ops-monitoring-bot added a comment.Apr 11 2019, 9:42 AM

Script wmf-auto-reimage was launched by gehel on cumin2001.codfw.wmnet for hosts:

['maps2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201904110941_gehel_28422.log.

gerritbot added a comment.Apr 11 2019, 12:01 PM

Change 502977 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] migrate maps2001 to stretch

https://gerrit.wikimedia.org/r/502977

ops-monitoring-bot added a comment.Apr 11 2019, 12:16 PM

Completed auto-reimage of hosts:

['maps2002.codfw.wmnet']

Of which those FAILED:

['maps2002.codfw.wmnet']
gerritbot added a comment.Apr 11 2019, 12:49 PM

Change 502977 merged by Gehel:
[operations/puppet@production] migrate maps2001 to stretch

https://gerrit.wikimedia.org/r/502977

gerritbot added a comment.Apr 11 2019, 12:52 PM

Change 502988 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: maps2001 is now a slave after migrating to stretch

https://gerrit.wikimedia.org/r/502988

gerritbot added a comment.Apr 11 2019, 12:59 PM

Change 502988 merged by Gehel:
[operations/puppet@production] maps: maps2001 is now a slave after migrating to stretch

https://gerrit.wikimedia.org/r/502988

ops-monitoring-bot added a comment.Apr 11 2019, 1:00 PM

Script wmf-auto-reimage was launched by gehel on cumin2001.codfw.wmnet for hosts:

['maps2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201904111300_gehel_10390.log.

ops-monitoring-bot added a comment.Apr 11 2019, 1:08 PM

Script wmf-auto-reimage was launched by gehel on cumin2001.codfw.wmnet for hosts:

['maps2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201904111308_gehel_11769.log.

• Mathew.onipe subscribed.Apr 12 2019, 10:00 AM

This task is now complete and the lessons learnt have been documented here: https://wikitech.wikimedia.org/wiki/Maps-migration

• Mathew.onipe closed this task as Resolved.Apr 12 2019, 10:00 AM
gerritbot added a comment.Apr 15 2019, 7:45 AM

Change 503932 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: check OSM replication lag on all nodes in codfw

https://gerrit.wikimedia.org/r/503932

gerritbot added a comment.Apr 15 2019, 7:52 AM

Change 503932 merged by Gehel:
[operations/puppet@production] maps: check OSM replication lag on all nodes in codfw

https://gerrit.wikimedia.org/r/503932

Gehel closed subtask T219849: Tilerator crashed on maps200[1-3].codfw.wmnet as Resolved.Apr 16 2019, 3:08 PM
gerritbot added a comment.May 1 2019, 4:52 PM

Change 457408 abandoned by Gehel:
maps: migrate maps2004 to stretch

Reason:
already done!

https://gerrit.wikimedia.org/r/457408

• Mathew.onipe closed subtask T215521: Load OSM data into maps2004 as Resolved.May 14 2019, 1:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits