| Pnorman | |
| May 8 2018, 1:33 AM |
| F18762220: image.png | |
| Jun 4 2018, 11:16 PM |
| F18624855: image.png | |
| May 31 2018, 2:32 AM |
As part of preparing for production, we're testing on maps-test2004.
After testing it, we want to wipe the machine, and test again to be sure it works on a clean slate.
Key points to check are
Change 432694 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[operations/puppet@production] Set hieradata variables for maps tm2source
Change 432741 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[maps/kartotherian/package@master] Make Kartotherian depend on new schema work
Change 432741 abandoned by Pnorman:
Make Kartotherian depend on new schema work
Reason:
I need to do the docker rebuild stuff
Change 432741 restored by Pnorman:
Make Kartotherian depend on new schema work
Reason:
Docker stuff doesn't result in a repo change, so this is a valid commit
Change 432741 merged by Gehel:
[maps/kartotherian/package@master] Make Kartotherian depend on new schema work
Change 433217 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[maps/kartotherian/deploy@master] Switch maps-test2004 to ClearTable based styles
Change 433276 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[maps/tilerator/deploy@master] Make maps-test2004 use cleartables
Change 433276 merged by Gehel:
[maps/tilerator/deploy@master] Make maps-test2004 use cleartables
Change 433217 abandoned by Pnorman:
Switch maps-test2004 to ClearTable based styles
Reason:
I ended up doing this in https://gerrit.wikimedia.org/r/434887
Getting close!
This is with the genview source, so no tile store in Cassandra.
v3 source and everything that derives from it isn't working. I'm getting T195476: Unable to create source "v3"self._closeAsync is not a function error, which I suspect is an error masking a Cassandra error.
T195476: Unable to create source "v3"self._closeAsync is not a function error goes away when Cassandra is running. Instead we get a TypeError: Cannot read property 'length' of undefined message which we're debugging
I worked with @Mholloway and @mojodna, author of tilelive-tmsource, at debugging the errors.
The first step was to get service-runner to produce stack traces, which was done with this patch
--- a/node_modules/service-runner/lib/logger.js +++ b/node_modules/service-runner/lib/logger.js @@ -202,7 +202,7 @@ class Logger { const logUnhandledException = (err) => { if (!inLogger) { inLogger = true; - this.log('fatal/service-runner/unhandled', err); + this.log('fatal/service-runner/unhandled', err.stack); inLogger = false; } };
This showed the problem was in tilelive-tmstyle, when it tries to match up its layer list against that coming from the source. This was worked around by adding a tilejson to meddo and pointing at it with source: "tilejson+file://./temp.json"
Next, after some dead ends, we disabled some of the sources, and the v3 source works. It must be some later source that's causing the next error, but figuring out that will have to wait for another day.
Change 437297 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[operations/puppet@production] Remove duplicate osm2pgsql parameter
Change 437297 merged by Gehel:
[operations/puppet@production] Remove duplicate osm2pgsql parameter
Served from the Kartotherian frontend.
A couple of notes
Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:
['maps-test2004.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201806131529_gehel_14236.log.
Completed auto-reimage of hosts:
['maps-test2004.codfw.wmnet']
and were ALL successful.
reimage of maps-test2004.cofdw.wmnet is completed. There is an open point (T197159) that will be fixed separately.
@Mholloway / @Pnorman your turn for the data import.