Page MenuHomePhabricator

Repool maps primaries in Kartotherian
Closed, ResolvedPublic

Description

We depooled maps masters during the migration to imposm as we were seeing an unsustainable number of connections to postgresql at the time. This was around the time of the transition to Tegola, meaning that the primaries ran tilerator in addition to kartotherian, and we also had issues with connection pooling in Kartotherian. Resource usage on maps primaries is now extremely low, and it seems that we could get a little bit of extra capacity and uniformity by returning them to use as kartotherian hosts in pybal.

Event Timeline

Change 993700 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] tegola-vector-tiles: add maps primaries to config

https://gerrit.wikimedia.org/r/993700

Change 993702 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] conftool: restore maps primary servers to kartotherian pool

https://gerrit.wikimedia.org/r/993702

hnowlan changed the task status from Open to In Progress.Jan 29 2024, 2:50 PM
hnowlan claimed this task.
hnowlan triaged this task as Low priority.

Change 993702 merged by Hnowlan:

[operations/puppet@production] conftool: restore maps primary servers to kartotherian pool

https://gerrit.wikimedia.org/r/993702

Change 993700 merged by jenkins-bot:

[operations/deployment-charts@master] tegola-vector-tiles: add maps primaries to config

https://gerrit.wikimedia.org/r/993700

This change looks like is causing an issue. From apps team:

We seem to be getting intermittent 404s for certain urls, e.g. https://maps.wikimedia.org/static/webgl/wikisprites%402x.json

From a quick look running this request in all maps nodes only maps1009 is returning 404s for this URL. I think we should depool maps1009 and 2009 again and figure out whats wrong.
Probably its running some older version of kartotherian that doesn't expose this static file.

More specifically maps1009 is missing the static folder to serve:

jgiannelos@maps1009:/srv/deployment/kartotherian/deploy/src$ stat static
stat: cannot stat 'static': No such file or directory

This change looks like is causing an issue. From apps team:

We seem to be getting intermittent 404s for certain urls, e.g. https://maps.wikimedia.org/static/webgl/wikisprites%402x.json

From a quick look running this request in all maps nodes only maps1009 is returning 404s for this URL. I think we should depool maps1009 and 2009 again and figure out whats wrong.
Probably its running some older version of kartotherian that doesn't expose this static file.

More specifically maps1009 is missing the static folder to serve:

jgiannelos@maps1009:/srv/deployment/kartotherian/deploy/src$ stat static
stat: cannot stat 'static': No such file or directory

Depooled for now. It looks like the primaries are still configured in scap for kartotherian, what caused them to be missing these files?

For reference here is the ticket tracking the actual problem with maps1009 T356756

maps1009,2009 should be ready to be repooled