We depooled maps masters during the migration to imposm as we were seeing an unsustainable number of connections to postgresql at the time. This was around the time of the transition to Tegola, meaning that the primaries ran tilerator in addition to kartotherian, and we also had issues with connection pooling in Kartotherian. Resource usage on maps primaries is now extremely low, and it seems that we could get a little bit of extra capacity and uniformity by returning them to use as kartotherian hosts in pybal.
Description
Details
Related Objects
- Mentioned Here
- T356756: maps1009 is failing to start after scap deploy
Event Timeline
Change 993700 had a related patch set uploaded (by Hnowlan; author: Hnowlan):
[operations/deployment-charts@master] tegola-vector-tiles: add maps primaries to config
Change 993702 had a related patch set uploaded (by Hnowlan; author: Hnowlan):
[operations/puppet@production] conftool: restore maps primary servers to kartotherian pool
Change 993702 merged by Hnowlan:
[operations/puppet@production] conftool: restore maps primary servers to kartotherian pool
Change 993700 merged by jenkins-bot:
[operations/deployment-charts@master] tegola-vector-tiles: add maps primaries to config
This change looks like is causing an issue. From apps team:
We seem to be getting intermittent 404s for certain urls, e.g. https://maps.wikimedia.org/static/webgl/wikisprites%402x.json
From a quick look running this request in all maps nodes only maps1009 is returning 404s for this URL. I think we should depool maps1009 and 2009 again and figure out whats wrong.
Probably its running some older version of kartotherian that doesn't expose this static file.
More specifically maps1009 is missing the static folder to serve:
jgiannelos@maps1009:/srv/deployment/kartotherian/deploy/src$ stat static stat: cannot stat 'static': No such file or directory
Depooled for now. It looks like the primaries are still configured in scap for kartotherian, what caused them to be missing these files?