List of steps to reproduce (step by step, including full links if applicable):
- Some of the lakes are missing again.
Description
Description
Related Objects
Related Objects
- Mentioned In
- T316365: Restore the map data health and parity between clusters
T313177: Lake Erie is missing again.
T296021: Borders between countries are not always depicted in Wikimedia Maps
T218097: [Bug] OSM DB degradation during sync as a result of missing features - Mentioned Here
- T306424: Tegola pods are crashing because swift doesnt allow connections
Event Timeline
Comment Actions
Its a known issue. We had to fallback to old tiles because of the latest incident: https://phabricator.wikimedia.org/T306424
Soon we are going to failover again to the latest known good state but meanwhile this was the only way to manage the without a backing tile storage
Comment Actions
Confirmed still broken:
https://maps.wikimedia.org/#8/45.066/-81.557
Seems to be fixed now. \o/
Comment Actions
Tiles did look fine earlier when the data was served from eqiad but it seems old tiles resurfaced now after switching to codfw. I suppose the regeneration of all tiles was forced in eqiad but not in codfw, or something like that? @Jgiannelos could you please check.
Comment Actions
At first I thought it was varnish needing some time, but indeed if I hit directly the service eqiad looks like its serving the right tiles for the maps on the link were codfw is failing.
The interesting part is that both envs are freshly imported with the exact same config and fresh tiles pregenerated.
I think the problem is that tegola on k8s is running some old pods and the issue might be that it is still pointing to the old storage. Respawning fresh ones will do the trick.
cc @jijiki
From a quick look pods look pretty old:
kube_env tegola-vector-tiles codfw NAME READY STATUS RESTARTS AGE tegola-vector-tiles-main-796b7f9588-2qlk6 2/2 Running 1 6d21h tegola-vector-tiles-main-796b7f9588-497hz 2/2 Running 1 8d tegola-vector-tiles-main-796b7f9588-5hswz 2/2 Running 1 56d tegola-vector-tiles-main-796b7f9588-9kf4w 2/2 Running 1 8d tegola-vector-tiles-main-796b7f9588-dnglj 2/2 Running 1 56d tegola-vector-tiles-main-796b7f9588-dwj8b 2/2 Running 1 6d21h tegola-vector-tiles-main-796b7f9588-fvbwr 2/2 Running 1 56d tegola-vector-tiles-main-796b7f9588-g42x6 2/2 Running 1 56d tegola-vector-tiles-main-796b7f9588-lbvgd 2/2 Running 1 6d21h tegola-vector-tiles-main-796b7f9588-lhcjz 2/2 Running 1 56d tegola-vector-tiles-main-796b7f9588-p2l24 2/2 Running 1 56d tegola-vector-tiles-main-796b7f9588-r4qbh 2/2 Running 1 8d
Comment Actions
I kicked the pods and it looks like hitting the service directly on maps shows the lakes:
kubectl get pods NAME READY STATUS RESTARTS AGE tegola-vector-tiles-main-796b7f9588-5zvbs 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-7q8qx 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-7w9b5 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-7zp9v 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-gkrd8 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-glx62 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-lcvws 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-mbkmq 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-pr8fp 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-q48pb 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-wx92s 2/2 Running 1 10m tegola-vector-tiles-main-796b7f9588-xc7fd 2/2 Running 1 10m
We need to wait for varnish now