This seems to be because it's trying to display map tiles from https://maps-beta.wmflabs.org/, which is down.

@awight I'm afraid that you were the last person to touch this configuration: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/803932. Do you know what happened to the service?

The maps-beta.wmflabs.org proxy points at the deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud instance. That node was created by @hnowlan in mid-2024.

I forced a Puppet run on deployment-maps-master02 to see what would happen:

bd808@deployment-maps-master02:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(1716bf547b) gitpuppet - MediaWiki: Only proxy existing .php files, otherwise return nice 404'
Notice: /Stage[main]/Osm::Imposm3/Systemd::Service[imposm]/Service[imposm]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Osm::Imposm3/Systemd::Service[imposm]/Service[imposm]: Unscheduling refresh on Service[imposm]
Notice: Applied catalog in 11.48 seconds

imposm being restarted there is a good sign that something is broken.

bd808@deployment-maps-master02:~$ sudo journalctl -u imposm --no-page --since "15 minutes ago"
Mar 19 15:25:47 deployment-maps-master02 systemd[1]: Started imposm.service - "imposm service for OSM sync".
Mar 19 15:25:47 deployment-maps-master02 imposm[2810534]: [2026-03-19T15:25:47Z] 0:00:00 [fatal] Unable to read last.state.txt:open /srv/osm/diff/last.state.txt: no such file or directory
Mar 19 15:25:47 deployment-maps-master02 systemd[1]: imposm.service: Main process exited, code=exited, status=1/FAILURE
Mar 19 15:25:47 deployment-maps-master02 systemd[1]: imposm.service: Failed with result 'exit-code'.
Mar 19 15:32:09 deployment-maps-master02 systemd[1]: Started imposm.service - "imposm service for OSM sync".
Mar 19 15:32:09 deployment-maps-master02 imposm[2811178]: [2026-03-19T15:32:09Z] 0:00:00 [fatal] Unable to read last.state.txt:open /srv/osm/diff/last.state.txt: no such file or directory
Mar 19 15:32:09 deployment-maps-master02 systemd[1]: imposm.service: Main process exited, code=exited, status=1/FAILURE
Mar 19 15:32:09 deployment-maps-master02 systemd[1]: imposm.service: Failed with result 'exit-code'.
bd808@deployment-maps-master02:~$ ls -lh /srv/osm/diff/
total 0

I wonder if anyone from the Content-Transform-Team can help figure out who to poke for help getting the tile server stack running again?

I don't have any clues, but +1 to bd808's suggested next stop!

The front proxy is pointed at port 6533 on the backing server. 6533 is the port documented on https://wikitech.wikimedia.org/wiki/Maps/Kartotherian, but https://wikitech.wikimedia.org/wiki/Kubernetes/Service_ports lists 6543. Neither port is present in netstat -tunl output on the host.

I have a hunch that part of the problem here, possibly all of the problem, is that Kartotherian moved from being a scap3 deployed service to a Kubernetes service in production and nobody setup a Docker service in Beta to match. This hunch has some alignment with the diagram at https://wikitech.wikimedia.org/wiki/Maps/v2/Infrastructure#Diagram and the note "The diagram is still very useful but not up-to-date, since Kartotherian now runs on Kubernetes and not on the maps bare metal nodes."

I think that the issue here is that kartotherian used to be deployed using puppet on mapsXXXX nodes but we moved it to k8s so now puppet doesn't configure the node kartotherian service.

I think to bring again deployment-prep up to speed with prod we need:

• A new node for tegola (no need for caching)
• A new node for kartotherian
• A new node for postgres/osm (no need for replication, just an initial bootstrap is enough)

And then:

• Run a bootstrap of a small part of OSM (full maps import is probably too much for the instances we have)
• Point the proxy to kartotherian
• Remove the maps-master node

On a different note (cc @ihurbain) since we barely do any development in kartotherian, there is no need for maps on beta. Can we point mw on beta to use maps in prod ?

In T420299#11732460, @Jgiannelos wrote:

Can we point mw on beta to use maps in prod ?

This feels like a reasonable question to ask. If we can get the tiles or whatever are needs from elsewhere and maintaining the stack here is not helpful then we should save the effort.

I agree with @Jgiannelos and @ihurbain. We should reduce complexity with the beta cluster and use maps in prod. That was the case in the past if I'm not mistaken.

In T420299#11733039, @MSantos wrote:

I agree with @Jgiannelos and @ihurbain. We should reduce complexity with the beta cluster and use maps in prod. That was the case in the past if I'm not mistaken.

It has gone back and forth over time:

• rOMWC5ce61a32ce04: Enable Kartographer ext in labs
• rOMWC535cc47cb490: Labs: Kartographer is already loaded in prod
• rOMWC948eafb0d50c: Use maps-beta to serve maps in labs
• rOMWC4bbffd3c14a4: Remove reference to no-longer-existing maps-beta.wmflabs.org
• rOMWC0dedf96d7ea2: [beta] Maps will be provided by the maps-experiments kartotherian

We checked wth CTT team and it doesn't look like anyone knows how MW is maintained on beta. Ideally the maintainers of beta can change the kartotheiran config to point to production maps endpoint.

Change #1261515 had a related patch set uploaded (by Arlolra; author: Arlolra):

[operations/mediawiki-config@master] Use prod to serve maps in labs

https://gerrit.wikimedia.org/r/1261515

Change #1261515 merged by jenkins-bot:

[operations/mediawiki-config@master] Use prod to serve maps in labs

https://gerrit.wikimedia.org/r/1261515

Thanks y'all.

I wonder if we could delete the non-working parts of the old setup? Apart from the proxy "maps-beta.wmflabs.org", there is also an instance "deployment-maps-master02", and maybe other things that I don't even know to look for. Are they worth keeping in case someone wanted to have separate maps infrastructure in beta again, or would that be easier to set up from scratch?

I can cleanup the stale instances. Also might be a good idea to remove the maps cloudvps project that is unused.

In T420299#11756968, @Jgiannelos wrote:

I can cleanup the stale instances.

See T421449: Clean up broken legacy maps stack in Beta Cluster, and thank you.

Also might be a good idea to remove the maps cloudvps project that is unused.

The Maps Cloud VPS project is used, just not for your maps. It predates the WMF having any maps projects at all. WikiMiniAtlas runs there if nothing else.