7. Should also include updating the style which will be dependent on vectors.

Quick check on disk space shows that we have ~500GB free on maps[12]00[1-4] nodes, and that the current keyspace is using bewteen 100GB and 200GB, so we seem to have enough space to keep both for some time. Note that the maps-test* nodes have slightly smaller disks, keeping both keyspaces there is probably going to be an issue. But since we now have test instances on WMCS, that's ok.

In T191655#4113343, @MaxSem wrote:

7. Should also include updating the style which will be dependent on vectors.

kartotherian@i18n includes a version of the style that renders 'name' instead of 'name_en'. It's currently pulling it from the 'render-name' branch. Once that is merged I'll update kartotherian@i18n to pull it from a proper version.

In T191655#4115991, @Gehel wrote:

Quick check on disk space shows that we have ~500GB free on maps[12]00[1-4] nodes, and that the current keyspace is using bewteen 100GB and 200GB, so we seem to have enough space to keep both for some time. Note that the maps-test* nodes have slightly smaller disks, keeping both keyspaces there is probably going to be an issue. But since we now have test instances on WMCS, that's ok.

Good to know. Thanks for checking.

@Gehel pointed out on irc that we don't really have to stop and restart the job that process new OSM data.

Fewer steps is better. I'll update the plan accordingly.

Change 425092 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/puppet@production] Allow for different storage_id in kartotherian and tilerator

https://gerrit.wikimedia.org/r/425092

Change 425092 merged by Gehel:
[operations/puppet@production] Allow for different storage_id in kartotherian and tilerator

https://gerrit.wikimedia.org/r/425092

In T191655#4115991, @Gehel wrote:

Quick check on disk space shows that we have ~500GB free on maps[12]00[1-4] nodes, and that the current keyspace is using bewteen 100GB and 200GB, so we seem to have enough space to keep both for some time. Note that the maps-test* nodes have slightly smaller disks, keeping both keyspaces there is probably going to be an issue. But since we now have test instances on WMCS, that's ok.

The WMCS test instance is pretty small, it only has 80 GB of disk space, and for that reason we've only imported Belgium, Switzerland, Israel and Palestine into it. maps-test2001 has 245GB free so it might just barely have enough space to regenerate a second copy of the tiles.

Change 425801 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: increment cassandra keyspace to v4 for tilerator

https://gerrit.wikimedia.org/r/425801

Mentioned in SAL (#wikimedia-operations) [2018-04-12T13:22:04Z] <gehel> deploying maps internationalization, including new keyspace and generating new tiles - T191655

Change 425801 merged by Gehel:
[operations/puppet@production] maps: increment cassandra keyspace to v4 for tilerator

https://gerrit.wikimedia.org/r/425801

Mentioned in SAL (#wikimedia-operations) [2018-04-12T13:40:15Z] <gehel> dropping leftover keyspace v2 and v5 on maps / eqiad - T191655

This is how step 2 was executed:

1. Merge deploy repo (https://gerrit.wikimedia.org/r/#/c/425609/) [done]
2. Set tilerator_storage_id to v4 (https://gerrit.wikimedia.org/r/#/c/425801/) [gehel] [done]
3. Force a puppet run on tileratorui nodes [gehel] [done]
4. Use tileshell to force creation of keyspace and tiles table (https://wikitech.wikimedia.org/wiki/Maps#Cassandra) [stephane] [done]
5. Deploy tilerator everywhere [stephane] [done]
6. Create extract_names function in PG (https://github.com/kartotherian/osm-bright.tm2source/blob/master/sql/names.sql) sudo -u postgres psql -Xd gis -f /srv/deployment/tilerator/deploy/node_modules/@kartotherian/osm-bright-source/sql/names.sql [gehel] [done]
7. Restart tilerator and tileratorui (not automatic) [gehel] [done]
8. grant EXECUTE on FUNCTION extract_names(hstore) to public; [gehel] [done]

This is how #4 could be done:

1. Propagate tilerator_storage_id to kartotherian here and here
2. Update kartotherian sources
   1. Hardcode (temporarily) style of existing sources as @kartotherian/osm-bright-style-prei18n. This package correspond to v2.1.4 and was already released to npm
   2. Add new sources to serve from tilerator_storage_id (the condition is to prevent it from failing on maps-test* where there is no new storage)

{% if tilerator_storage_id != storage_id %}
v4:
  uri: cassandra://
  params:
    maxzoom: 16
    keyspace: v4
    cp: {var: cassandra-servers}
    username: {var: cassandra-user}
    password: {var: cassandra-pswd}
    setLastModified: true

osm-localized:
  uri: babel://
  params:
    source: {ref: v4}
    tag: name
    combineName: false
    defaultLanguage: en

osm-pbf-i18n:
  public: true
  formats: [pbf]
  uri: overzoom://
  params:
    source: {ref: osm-localized}
    maxzoom: 19
  overrideInfo:
    attribution: 'Map data &copy; <a href="http://openstreetmap.org/copyright">OpenStreetMap contributors</a>'
    tiles: ['https://maps.wikimedia.org/osm-pbf-i18n/{z}/{x}/{y}.pbf']
  defaultHeaders:
    Cache-Control: 'public, max-age=3600'

# can be accessed with ?s=osm-intl-i18n
osm-intl-i18n:
  public: true
  formats: [png,json,headers,svg,jpeg]
  scales: [1.3, 1.5, 2, 2.6, 3]
  maxzoom: 19
  static: true
  maxheight: 2048
  maxwidth: 2048
  uri: vector://
  xml:
    loader: '@kartotherian/osm-bright-style'
  xmlSetParams:
    source: {ref: osm-pbf-i18n}
  overrideInfo:
    attribution: 'Map data &copy; <a href="http://openstreetmap.org/copyright">OpenStreetMap contributors</a>'
    tiles: ['https://maps.wikimedia.org/osm-intl-i18n/{z}/{x}/{y}.png']
  defaultHeaders:
    Cache-Control: 'public, max-age=3600'
{% endif %}

Change 427631 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/puppet@production] Make tilerator_storage_id to kartotherian

https://gerrit.wikimedia.org/r/427631

Change 427631 merged by Gehel:
[operations/puppet@production] Make tilerator_storage_id to kartotherian

https://gerrit.wikimedia.org/r/427631

Change 427685 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] Use erb syntax for "if"

https://gerrit.wikimedia.org/r/427685

Change 427685 merged by Gehel:
[maps/kartotherian/deploy@master] Use erb syntax for "if"

https://gerrit.wikimedia.org/r/427685

Change 427689 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] erb syntax but if still closes with endif

https://gerrit.wikimedia.org/r/427689

Change 427689 merged by Gehel:
[maps/kartotherian/deploy@master] erb syntax but if still closes with endif

https://gerrit.wikimedia.org/r/427689

Change 427691 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: disable OSM replication during tile regeneration

https://gerrit.wikimedia.org/r/427691

Change 427691 merged by Gehel:
[operations/puppet@production] maps: disable OSM replication during tile regeneration

https://gerrit.wikimedia.org/r/427691

Mentioned in SAL (#wikimedia-operations) [2018-04-19T16:20:33Z] <gehel> shutting down tilerator on maps[12].* for maintenance - T191655

Mentioned in SAL (#wikimedia-operations) [2018-04-19T16:28:00Z] <gehel> restarting tilerator on maps[12].* - T191655

For the MediaWiki side of this, we need to:

• Write a maintenance script that purges all pages with map tags (T193525: Write maintenance script that purges pages using maps)
• As the patch for T192955: Add config setting for default map language rolls out on the train, run the script for each group; now all maps will point to ?lang=local
• In Kartotherian, add Babel to the main source (osm-intl)
• Change the testwiki+test2wiki config to remove the custom source, and set the page language config var to true. Then purge all map pages on these wikis.
• In Kartotherian, remove the custom source (osm-intl-i18n)

Then when we're ready, for each set of wikis we want to deploy to:

• Set the page language config var to true
• Run the purge script

Change 430893 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: re-enable OSM replication

https://gerrit.wikimedia.org/r/430893

Change 430893 merged by Gehel:
[operations/puppet@production] maps: re-enable OSM replication

https://gerrit.wikimedia.org/r/430893

Change 431550 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/puppet@production] Bump kartotherian_storage_id to v4

https://gerrit.wikimedia.org/r/431550

Cleanup steps for kartotherian config (May 7th)

1. Set kartotherian_storage_id to v4
2. Update kartotherian/deploy/scap/templates/config.yaml.j2 to use storage_id, which correspond to kartotherian_storage_id, instead of tilerator_storage_id. Both should have the same value (v4) so this should be a no-op.

Change 431557 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] Use storage_id instead of tilerator_storage_id

https://gerrit.wikimedia.org/r/431557

Change 431550 merged by Gehel:
[operations/puppet@production] Bump kartotherian_storage_id to v4

https://gerrit.wikimedia.org/r/431550

Change 431557 merged by Gehel:
[maps/kartotherian/deploy@master] Use storage_id instead of tilerator_storage_id

https://gerrit.wikimedia.org/r/431557

Change 431561 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/mediawiki-config@master] Make test and test2 using maps i18n correctly

https://gerrit.wikimedia.org/r/431561

Change 431561 merged by jenkins-bot:
[operations/mediawiki-config@master] Make test and test2 using maps i18n correctly

https://gerrit.wikimedia.org/r/431561

Mentioned in SAL (#wikimedia-operations) [2018-05-07T13:16:55Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:431561|Make test and test2 using maps i18n correctly (T191655)]] (duration: 01m 00s)

Change 431565 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[maps/kartotherian/deploy@master] Remove temporary i18n source

https://gerrit.wikimedia.org/r/431565

Change 431565 merged by Gehel:
[maps/kartotherian/deploy@master] Remove temporary i18n source

https://gerrit.wikimedia.org/r/431565

In T191655#4171979, @Catrope wrote:

For the MediaWiki side of this, we need to:

• Write a maintenance script that purges all pages with map tags (T193525: Write maintenance script that purges pages using maps)
• As the patch for T192955: Add config setting for default map language rolls out on the train, run the script for each group; now all maps will point to ?lang=local
• In Kartotherian, add Babel to the main source (osm-intl)
• Change the testwiki+test2wiki config to remove the custom source, and set the page language config var to true. Then purge all map pages on these wikis.
• In Kartotherian, remove the custom source (osm-intl-i18n)

All done.

Change 431628 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/mediawiki-config@master] Enable maps i18n everywhere

https://gerrit.wikimedia.org/r/431628

Change 431628 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable maps i18n everywhere

https://gerrit.wikimedia.org/r/431628

Mentioned in SAL (#wikimedia-operations) [2018-05-08T13:25:28Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:431628|Enable maps i18n everywhere (T191655)]] (duration: 01m 00s)

Change 431609 had a related patch set uploaded (by Sbisson; owner: Catrope):
[operations/mediawiki-config@master] Remove unused wgKartographerDfltStyle setting

https://gerrit.wikimedia.org/r/431609

Change 431609 merged by jenkins-bot:
[operations/mediawiki-config@master] Remove unused wgKartographerDfltStyle setting

https://gerrit.wikimedia.org/r/431609

Mentioned in SAL (#wikimedia-operations) [2018-05-09T13:18:40Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:431609|Remove unused wgKartographerDfltStyle setting (T191655)]] (duration: 01m 20s)