Page MenuHomePhabricator

Deploy new vector tiles on maps-test2004.codfw.wmnet
Closed, ResolvedPublic

Description

As part of preparing for production, we're testing on maps-test2004.

After testing it, we want to wipe the machine, and test again to be sure it works on a clean slate.

Key points to check are

  • We can pre-generate some tiles
  • Live generation of vector tiles work
  • Vector to raster works
  • Multilingual stuff works
  • Geoshapes works

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
InvalidNone
StalledPnorman
OpenPnorman
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
DeclinedNone
ResolvedGehel
ResolvedSBisson
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedGehel
ResolvedGehel
ResolvedGehel
ResolvedPnorman
ResolvedMholloway
ResolvedGehel

Event Timeline

Pnorman created this task.May 8 2018, 1:33 AM
Pnorman triaged this task as Normal priority.

Change 432694 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[operations/puppet@production] Set hieradata variables for maps tm2source

https://gerrit.wikimedia.org/r/432694

Change 432694 abandoned by Pnorman:
Set hieradata variables for maps tm2source

https://gerrit.wikimedia.org/r/432694

Pnorman moved this task from Backlog to In progress on the Maps-Sprint board.May 12 2018, 6:10 PM

Change 432741 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[maps/kartotherian/package@master] Make Kartotherian depend on new schema work

https://gerrit.wikimedia.org/r/432741

Change 432741 abandoned by Pnorman:
Make Kartotherian depend on new schema work

Reason:
I need to do the docker rebuild stuff

https://gerrit.wikimedia.org/r/432741

Change 432741 restored by Pnorman:
Make Kartotherian depend on new schema work

Reason:
Docker stuff doesn't result in a repo change, so this is a valid commit

https://gerrit.wikimedia.org/r/432741

Change 432741 merged by Gehel:
[maps/kartotherian/package@master] Make Kartotherian depend on new schema work

https://gerrit.wikimedia.org/r/432741

Change 433217 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[maps/kartotherian/deploy@master] Switch maps-test2004 to ClearTable based styles

https://gerrit.wikimedia.org/r/433217

Change 433276 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[maps/tilerator/deploy@master] Make maps-test2004 use cleartables

https://gerrit.wikimedia.org/r/433276

Change 433276 merged by Gehel:
[maps/tilerator/deploy@master] Make maps-test2004 use cleartables

https://gerrit.wikimedia.org/r/433276

Change 433217 abandoned by Pnorman:
Switch maps-test2004 to ClearTable based styles

Reason:
I ended up doing this in https://gerrit.wikimedia.org/r/434887

https://gerrit.wikimedia.org/r/433217

Getting close!

This is with the genview source, so no tile store in Cassandra.

v3 source and everything that derives from it isn't working. I'm getting T195476: Unable to create source "v3"self._closeAsync is not a function error, which I suspect is an error masking a Cassandra error.

T195476: Unable to create source "v3"self._closeAsync is not a function error goes away when Cassandra is running. Instead we get a TypeError: Cannot read property 'length' of undefined message which we're debugging

Pnorman added a subscriber: mojodna.Jun 1 2018, 2:44 AM

I worked with @Mholloway and @mojodna, author of tilelive-tmsource, at debugging the errors.

The first step was to get service-runner to produce stack traces, which was done with this patch

--- a/node_modules/service-runner/lib/logger.js
+++ b/node_modules/service-runner/lib/logger.js
@@ -202,7 +202,7 @@ class Logger {
         const logUnhandledException = (err) => {
             if (!inLogger) {
                 inLogger = true;
-                this.log('fatal/service-runner/unhandled', err);
+                this.log('fatal/service-runner/unhandled', err.stack);
                 inLogger = false;
             }
         };

This showed the problem was in tilelive-tmstyle, when it tries to match up its layer list against that coming from the source. This was worked around by adding a tilejson to meddo and pointing at it with source: "tilejson+file://./temp.json"

Next, after some dead ends, we disabled some of the sources, and the v3 source works. It must be some later source that's causing the next error, but figuring out that will have to wait for another day.

Change 437297 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[operations/puppet@production] Remove duplicate osm2pgsql parameter

https://gerrit.wikimedia.org/r/437297

Change 437297 merged by Gehel:
[operations/puppet@production] Remove duplicate osm2pgsql parameter

https://gerrit.wikimedia.org/r/437297

Pnorman updated the task description. (Show Details)Jun 4 2018, 8:28 PM
Pnorman updated the task description. (Show Details)
Pnorman updated the task description. (Show Details)Jun 4 2018, 10:55 PM
Pnorman updated the task description. (Show Details)Jun 4 2018, 11:15 PM

Served from the Kartotherian frontend.

A couple of notes

  • Truncating tables in Cassandra is useful when doing this stuff.
  • Disabling browser cache is essential

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['maps-test2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806131529_gehel_14236.log.

Gehel added a subscriber: Gehel.Jun 13 2018, 5:33 PM

reimage is failing, scap has a reference to tin. See T197159 for details.

Completed auto-reimage of hosts:

['maps-test2004.codfw.wmnet']

and were ALL successful.

Gehel added a comment.Jun 13 2018, 7:06 PM

reimage of maps-test2004.cofdw.wmnet is completed. There is an open point (T197159) that will be fixed separately.

@Mholloway / @Pnorman your turn for the data import.

Gehel closed this task as Resolved.Jun 15 2018, 8:25 PM
Gehel claimed this task.

@Gehel you tagged this as resolved but T197059 isn't done

Vvjjkkii renamed this task from Deploy new vector tiles on maps-test2004.codfw.wmnet to hfdaaaaaaa.Jul 1 2018, 1:12 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii removed Gehel as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
WhitePhosphorus renamed this task from hfdaaaaaaa to Deploy new vector tiles on maps-test2004.codfw.wmnet.Jul 1 2018, 2:47 AM
WhitePhosphorus closed this task as Resolved.
WhitePhosphorus lowered the priority of this task from High to Normal.
WhitePhosphorus assigned this task to Gehel.
WhitePhosphorus updated the task description. (Show Details)
WhitePhosphorus added subscribers: Aklapper, gerritbot.