@Evad37 I've tested every machine currently serving production traffic without caching, it looked fine for me. Maybe it was fixed after T267296#6609192. Is someone still able to reproduce it?

I also observe that lately it takes hours or days after page save until map content matching dynamic view becomes available on snapshot image. E.g. in this article map was added three days ago and on snapshot I currently still don't see OSM object that is auto-positioned in dynamic view. Or, in this article I observed the same yesterday, while now snapshot highlights the OSM object, but the object has become missing in dynamic view.

Could the issue be similar to T266807? Per Grafana 100% CPU usage remains to be more or less an issue, If I'm not mistaken. I suppose users seeing different output is due to different level of unstability in eqiad and codfw?

In T268927#6655678, @Pikne wrote:

I also observe that lately it takes hours or days after page save until map content matching dynamic view becomes available on snapshot image. E.g. in this article map was added three days ago and on snapshot I currently still don't see OSM object that is auto-positioned in dynamic view. Or, in this article I observed the same yesterday, while now snapshot highlights the OSM object, but the object has become missing in dynamic view.

Could the issue be similar to T266807? Per Grafana 100% CPU usage remains to be more or less an issue, If I'm not mistaken. I suppose users seeing different output is due to different level of unstability in eqiad and codfw?

@Pikne, this specific case was cache, but thank you for reporting, maybe we shouldn't cache geoshape responses too long.

Thanks to @Evad37 for reporting. I just want to add that the marker/shape sometimes does show initially in article space, but subsequently it can disappear and then reappear. That's even after I link the Wikidata item to the OSM item and it does that initial cache which shows it. This article for instance I am sure showed it previously, but doesn't presently.

Thanks for your assistance.

@The_Equalizer it looks like there are still some cache from the maps outage, I'm tagging Traffic to help us understand if it's possible to clean-up cache from maps.wikimedia.org/geoshape and /geoline endpoints created before T267296#6609192 (Nov 6~7th).

The max lifetime of any object in the Traffic CDN is 24 hours. Are you sure they're being cached there? Can you give a full example URL?

In T268927#6662840, @CDanis wrote:

The max lifetime of any object in the Traffic CDN is 24 hours. Are you sure they're being cached there? Can you give a full example URL?

The full example URL is https://maps.wikimedia.org/geoshape?getgeojson=1&ids=Q2073436

I just tested every machine in production (after your comment) and it looks like maps1001.eqiad.wmnet is the one serving the same content cached for this URL, all other machines are serving a different output (the correct one)

Sounds like you chaps have a very quick handle on things - great stuff and keep up the good work.

maps1001 is now in sync and serving data consistent with the other nodes.