Page MenuHomePhabricator

Deploy maps internationalization to production
Closed, ResolvedPublic

Description

GOAL: Document a deployment plan for the maps internationalization feature that we can confidently execute without causing any downtime and that supports rollback if anything goes wrong.

Proposed plan:

  1. 1. Introduce 'kartotherian_storage_id' and 'tilerator_storage_id' hiera variables with the current value of 'storage_id' (v3) and use them for their respective services.
  2. 2. Deploy tilerator@i18n (it is using keyspace v4)
  3. 3. Regenerate all tiles. In the meantime, kartotherian is serving tiles from keyspace vN, they may be getting a bit stale but that's ok
  4. 4. Deploy kartotherian with i18n code with existing sources serving tiles from keyspace vN+1 WITHOUT Babel and new sources serving from vN+1 WITH Babel
  5. To do official release on a wiki, set $wgKartographerUsePageLanguage = true and purge pages in Category:Pages_with_maps
  6. Eventually drop keyspace vN to save space

Alternate ending (after: Deploy kartotherian@i18n):

  1. Set kartotherian_storage_id to vN (v3) to serve tiles from old keyspace
  2. Set tilerator_storage_id to vN (v3), rewind and restart osm updates

All questions/comment/simplifications/warnings/etc are welcome.

Details

Related Gerrit Patches:
operations/mediawiki-config : masterRemove unused wgKartographerDfltStyle setting
operations/mediawiki-config : masterEnable maps i18n everywhere
operations/mediawiki-config : masterMake test and test2 using maps i18n correctly
maps/kartotherian/deploy : masterRemove temporary i18n source
maps/kartotherian/deploy : masterUse storage_id instead of tilerator_storage_id
operations/puppet : productionBump kartotherian_storage_id to v4
operations/puppet : productionmaps: re-enable OSM replication
operations/puppet : productionmaps: disable OSM replication during tile regeneration
maps/kartotherian/deploy : mastererb syntax but if still closes with endif
maps/kartotherian/deploy : masterUse erb syntax for "if"
operations/puppet : productionMake tilerator_storage_id to kartotherian
operations/puppet : productionmaps: increment cassandra keyspace to v4 for tilerator
operations/puppet : productionAllow for different storage_id in kartotherian and tilerator

Event Timeline

SBisson created this task.Apr 6 2018, 6:21 PM
Restricted Application added a project: Maps. · View Herald TranscriptApr 6 2018, 6:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Restricted Application added a project: Discovery. · View Herald TranscriptApr 6 2018, 8:47 PM
MaxSem added a subscriber: MaxSem.Apr 6 2018, 9:21 PM
  1. Should also include updating the style which will be dependent on vectors.
Gehel added a comment.Apr 9 2018, 9:05 AM

Quick check on disk space shows that we have ~500GB free on maps[12]00[1-4] nodes, and that the current keyspace is using bewteen 100GB and 200GB, so we seem to have enough space to keep both for some time. Note that the maps-test* nodes have slightly smaller disks, keeping both keyspaces there is probably going to be an issue. But since we now have test instances on WMCS, that's ok.

  1. Should also include updating the style which will be dependent on vectors.

kartotherian@i18n includes a version of the style that renders 'name' instead of 'name_en'. It's currently pulling it from the 'render-name' branch. Once that is merged I'll update kartotherian@i18n to pull it from a proper version.

Quick check on disk space shows that we have ~500GB free on maps[12]00[1-4] nodes, and that the current keyspace is using bewteen 100GB and 200GB, so we seem to have enough space to keep both for some time. Note that the maps-test* nodes have slightly smaller disks, keeping both keyspaces there is probably going to be an issue. But since we now have test instances on WMCS, that's ok.

Good to know. Thanks for checking.

@Gehel pointed out on irc that we don't really have to stop and restart the job that process new OSM data.

Fewer steps is better. I'll update the plan accordingly.

SBisson updated the task description. (Show Details)Apr 9 2018, 1:54 PM
SBisson updated the task description. (Show Details)Apr 9 2018, 2:34 PM
SBisson updated the task description. (Show Details)Apr 9 2018, 3:19 PM
SBisson updated the task description. (Show Details)Apr 9 2018, 3:25 PM
SBisson updated the task description. (Show Details)

Change 425092 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/puppet@production] Allow for different storage_id in kartotherian and tilerator

https://gerrit.wikimedia.org/r/425092

Change 425092 merged by Gehel:
[operations/puppet@production] Allow for different storage_id in kartotherian and tilerator

https://gerrit.wikimedia.org/r/425092

SBisson updated the task description. (Show Details)Apr 10 2018, 12:34 PM

Quick check on disk space shows that we have ~500GB free on maps[12]00[1-4] nodes, and that the current keyspace is using bewteen 100GB and 200GB, so we seem to have enough space to keep both for some time. Note that the maps-test* nodes have slightly smaller disks, keeping both keyspaces there is probably going to be an issue. But since we now have test instances on WMCS, that's ok.

The WMCS test instance is pretty small, it only has 80 GB of disk space, and for that reason we've only imported Belgium, Switzerland, Israel and Palestine into it. maps-test2001 has 245GB free so it might just barely have enough space to regenerate a second copy of the tiles.

SBisson updated the task description. (Show Details)Apr 12 2018, 10:59 AM

Change 425801 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: increment cassandra keyspace to v4 for tilerator

https://gerrit.wikimedia.org/r/425801

Mentioned in SAL (#wikimedia-operations) [2018-04-12T13:22:04Z] <gehel> deploying maps internationalization, including new keyspace and generating new tiles - T191655

Change 425801 merged by Gehel:
[operations/puppet@production] maps: increment cassandra keyspace to v4 for tilerator

https://gerrit.wikimedia.org/r/425801

Mentioned in SAL (#wikimedia-operations) [2018-04-12T13:40:15Z] <gehel> dropping leftover keyspace v2 and v5 on maps / eqiad - T191655

SBisson updated the task description. (Show Details)Apr 12 2018, 3:04 PM

This is how step 2 was executed:

  1. Merge deploy repo (https://gerrit.wikimedia.org/r/#/c/425609/) [done]
  2. Set tilerator_storage_id to v4 (https://gerrit.wikimedia.org/r/#/c/425801/) [gehel] [done]
  3. Force a puppet run on tileratorui nodes [gehel] [done]
  4. Use tileshell to force creation of keyspace and tiles table (https://wikitech.wikimedia.org/wiki/Maps#Cassandra) [stephane] [done]
  5. Deploy tilerator everywhere [stephane] [done]
  6. Create extract_names function in PG (https://github.com/kartotherian/osm-bright.tm2source/blob/master/sql/names.sql) sudo -u postgres psql -Xd gis -f /srv/deployment/tilerator/deploy/node_modules/@kartotherian/osm-bright-source/sql/names.sql [gehel] [done]
  7. Restart tilerator and tileratorui (not automatic) [gehel] [done]
  8. grant EXECUTE on FUNCTION extract_names(hstore) to public; [gehel] [done]
SBisson triaged this task as High priority.Apr 13 2018, 2:33 PM
SBisson updated the task description. (Show Details)Apr 18 2018, 4:49 PM

This is how #4 could be done:

  1. Propagate tilerator_storage_id to kartotherian here and here
  2. Update kartotherian sources
    1. Hardcode (temporarily) style of existing sources as @kartotherian/osm-bright-style-prei18n. This package correspond to v2.1.4 and was already released to npm
    2. Add new sources to serve from tilerator_storage_id (the condition is to prevent it from failing on maps-test* where there is no new storage)
{% if tilerator_storage_id != storage_id %}
v4:
  uri: cassandra://
  params:
    maxzoom: 16
    keyspace: v4
    cp: {var: cassandra-servers}
    username: {var: cassandra-user}
    password: {var: cassandra-pswd}
    setLastModified: true

osm-localized:
  uri: babel://
  params:
    source: {ref: v4}
    tag: name
    combineName: false
    defaultLanguage: en

osm-pbf-i18n:
  public: true
  formats: [pbf]
  uri: overzoom://
  params:
    source: {ref: osm-localized}
    maxzoom: 19
  overrideInfo:
    attribution: 'Map data &copy; <a href="http://openstreetmap.org/copyright">OpenStreetMap contributors</a>'
    tiles: ['https://maps.wikimedia.org/osm-pbf-i18n/{z}/{x}/{y}.pbf']
  defaultHeaders:
    Cache-Control: 'public, max-age=3600'

# can be accessed with ?s=osm-intl-i18n
osm-intl-i18n:
  public: true
  formats: [png,json,headers,svg,jpeg]
  scales: [1.3, 1.5, 2, 2.6, 3]
  maxzoom: 19
  static: true
  maxheight: 2048
  maxwidth: 2048
  uri: vector://
  xml:
    loader: '@kartotherian/osm-bright-style'
  xmlSetParams:
    source: {ref: osm-pbf-i18n}
  overrideInfo:
    attribution: 'Map data &copy; <a href="http://openstreetmap.org/copyright">OpenStreetMap contributors</a>'
    tiles: ['https://maps.wikimedia.org/osm-intl-i18n/{z}/{x}/{y}.png']
  defaultHeaders:
    Cache-Control: 'public, max-age=3600'
{% endif %}

Change 427631 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/puppet@production] Make tilerator_storage_id to kartotherian

https://gerrit.wikimedia.org/r/427631

Change 427631 merged by Gehel:
[operations/puppet@production] Make tilerator_storage_id to kartotherian

https://gerrit.wikimedia.org/r/427631

Change 427685 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] Use erb syntax for "if"

https://gerrit.wikimedia.org/r/427685

Change 427685 merged by Gehel:
[maps/kartotherian/deploy@master] Use erb syntax for "if"

https://gerrit.wikimedia.org/r/427685

Change 427689 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] erb syntax but if still closes with endif

https://gerrit.wikimedia.org/r/427689

Change 427689 merged by Gehel:
[maps/kartotherian/deploy@master] erb syntax but if still closes with endif

https://gerrit.wikimedia.org/r/427689

Change 427691 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: disable OSM replication during tile regeneration

https://gerrit.wikimedia.org/r/427691

Change 427691 merged by Gehel:
[operations/puppet@production] maps: disable OSM replication during tile regeneration

https://gerrit.wikimedia.org/r/427691

Mentioned in SAL (#wikimedia-operations) [2018-04-19T16:20:33Z] <gehel> shutting down tilerator on maps[12].* for maintenance - T191655

Mentioned in SAL (#wikimedia-operations) [2018-04-19T16:28:00Z] <gehel> restarting tilerator on maps[12].* - T191655

SBisson updated the task description. (Show Details)Apr 20 2018, 7:46 PM
Catrope added a comment.EditedMay 1 2018, 6:47 PM

For the MediaWiki side of this, we need to:

Then when we're ready, for each set of wikis we want to deploy to:

  • Set the page language config var to true
  • Run the purge script

Change 430893 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: re-enable OSM replication

https://gerrit.wikimedia.org/r/430893

Change 430893 merged by Gehel:
[operations/puppet@production] maps: re-enable OSM replication

https://gerrit.wikimedia.org/r/430893

Change 431550 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/puppet@production] Bump kartotherian_storage_id to v4

https://gerrit.wikimedia.org/r/431550

SBisson updated the task description. (Show Details)May 7 2018, 12:20 PM
SBisson added a comment.EditedMay 7 2018, 12:23 PM

Cleanup steps for kartotherian config (May 7th)

  1. Set kartotherian_storage_id to v4
  2. Update kartotherian/deploy/scap/templates/config.yaml.j2 to use storage_id, which correspond to kartotherian_storage_id, instead of tilerator_storage_id. Both should have the same value (v4) so this should be a no-op.

Change 431557 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] Use storage_id instead of tilerator_storage_id

https://gerrit.wikimedia.org/r/431557

Change 431550 merged by Gehel:
[operations/puppet@production] Bump kartotherian_storage_id to v4

https://gerrit.wikimedia.org/r/431550

Change 431557 merged by Gehel:
[maps/kartotherian/deploy@master] Use storage_id instead of tilerator_storage_id

https://gerrit.wikimedia.org/r/431557

Change 431561 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/mediawiki-config@master] Make test and test2 using maps i18n correctly

https://gerrit.wikimedia.org/r/431561

Change 431561 merged by jenkins-bot:
[operations/mediawiki-config@master] Make test and test2 using maps i18n correctly

https://gerrit.wikimedia.org/r/431561

Mentioned in SAL (#wikimedia-operations) [2018-05-07T13:16:55Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:431561|Make test and test2 using maps i18n correctly (T191655)]] (duration: 01m 00s)

Change 431565 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] Remove temporary i18n source

https://gerrit.wikimedia.org/r/431565

Change 431565 merged by Gehel:
[maps/kartotherian/deploy@master] Remove temporary i18n source

https://gerrit.wikimedia.org/r/431565

For the MediaWiki side of this, we need to:

All done.

SBisson updated the task description. (Show Details)May 7 2018, 4:28 PM
SBisson closed this task as Resolved.May 7 2018, 4:31 PM

AFAICT the work here is finished. All the temporary variable values and hacks have been cleaned up.

Actual releases of i18n will be handled by setting wgKartographerUsePageLanguage

Change 431628 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/mediawiki-config@master] Enable maps i18n everywhere

https://gerrit.wikimedia.org/r/431628

Change 431628 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable maps i18n everywhere

https://gerrit.wikimedia.org/r/431628

Mentioned in SAL (#wikimedia-operations) [2018-05-08T13:25:28Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:431628|Enable maps i18n everywhere (T191655)]] (duration: 01m 00s)

Change 431609 had a related patch set uploaded (by Sbisson; owner: Catrope):
[operations/mediawiki-config@master] Remove unused wgKartographerDfltStyle setting

https://gerrit.wikimedia.org/r/431609

Change 431609 merged by jenkins-bot:
[operations/mediawiki-config@master] Remove unused wgKartographerDfltStyle setting

https://gerrit.wikimedia.org/r/431609

Mentioned in SAL (#wikimedia-operations) [2018-05-09T13:18:40Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:431609|Remove unused wgKartographerDfltStyle setting (T191655)]] (duration: 01m 20s)