Link for verification: https://maps.wikimedia.org/#15/55.8054/37.5379

Added some more translations to the task description to get the sense of urgency of this issue. Hopefully not a lot of people read articles about that area of Moscow, but I think that assumption will probably be proven incorrect soon. This is *Unbreak Now* level but I do not want to move the triage level myself.

It's on entire map of Russia...

I'll note that the attack against OSM is repeated, ongoing, and automated, coming in waves. The OSM folks responding do not have sophisticated anti-vandalism tools, so they must wait for the accounts to be blocked by members of the Data Working Group. There are not many people with that access, so it can take a while for accounts to be blocked and then reverted.

In sum: Until OSM develops better anti-vandalism tools or the attacker stops, planet updates are likely to contain vandalism to name:ru tags depending on timing.

Can somebody highlight a feature that is vandalized on the map so we can understand whats happening on the upstream OSM data? On a more practical note as long as the upstream OSM data are fixed our hourly diffs are going to be updated and eventually tiles are going to be regenerated.

On the image above, it is a bunch of street names and a bunch of area names. РУБЛЬ СДОХ in the corner means ‘Rouble is dead’ etc.

Examples are in https://www.openstreetmap.org/changeset/140149289 (revert of vandalism) or https://www.openstreetmap.org/changeset/140198815 (actual vandalism) it seems

If I understand correctly from the OSM edit history the vandalism fix happened 21 hours ago so it should be part of our next pregeneration run in the next few hours (cronjob schedule "0 12 * * *")

What we could try is to manually trigger the pregeneration jobs earlier.

For example: https://www.openstreetmap.org/node/1274540186/history
Version #12 is the vandalized version that we serve.

I just verified in one of the postgres replicas in eqiad that the feature that has been reverted is reflected in our OSM data:

gis=# select name from layer_transportation_name(ST_MakeEnvelope(4174443.4825,7518727.7715,4175089.6142,7518974.3567, 3857), 18);
      name
-----------------
 улица Левитана
 улица Поленова
 улица Алабяна
 улица Алабяна
 улица Панфилова
 улица Алабяна

Just a heads up while I was checking for tiles pregeneration I saw this failure on tegola:

	Aug 23, 2023 @ 12:00:26.633
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
Aug 23, 2023 @ 12:00:26.6332023-08-23 12:00:26 [INFO] root.go:62: Loading config file: /etc/tegola/config.toml
Aug 23, 2023 @ 12:00:26.6332023-08-23 12:00:26 [INFO] config.go:306: loading local config (/etc/tegola/config.toml)
Aug 23, 2023 @ 12:00:26.632  File "/opt/lib/python/site-packages/poppy/messsaging.py", line 22, in _get_engine
Aug 23, 2023 @ 12:00:26.632    return KafkaEngine(self.config)
Aug 23, 2023 @ 12:00:26.632  File "/opt/lib/python/site-packages/poppy/engine.py", line 124, in __init__
Aug 23, 2023 @ 12:00:26.632    self.consumer: KafkaConsumer = KafkaConsumer(
Aug 23, 2023 @ 12:00:26.632  File "/opt/lib/python/site-packages/kafka/consumer/group.py", line 356, in __init__
Aug 23, 2023 @ 12:00:26.632    self._client = KafkaClient(metrics=self._metrics, **self.config)
Aug 23, 2023 @ 12:00:26.632  File "/opt/lib/python/site-packages/kafka/client_async.py", line 244, in __init__
Aug 23, 2023 @ 12:00:26.631  File "/opt/lib/python/site-packages/poppy/cli.py", line 169, in dequeue
Aug 23, 2023 @ 12:00:26.631    with closing(Queue(ctx.obj)) as queue:
Aug 23, 2023 @ 12:00:26.631  File "/opt/lib/python/site-packages/poppy/messsaging.py", line 16, in __init__
Aug 23, 2023 @ 12:00:26.631    self.engine = self._get_engine()
Aug 23, 2023 @ 12:00:26.631    return f(get_current_context(), *args, **kwargs)
Aug 23, 2023 @ 12:00:26.630    return ctx.invoke(self.callback, **ctx.params)
Aug 23, 2023 @ 12:00:26.630  File "/opt/lib/python/site-packages/click/core.py", line 783, in invoke
Aug 23, 2023 @ 12:00:26.630    rv = self.invoke(ctx)
Aug 23, 2023 @ 12:00:26.630  File "/opt/lib/python/site-packages/click/core.py", line 1434, in invoke
Aug 23, 2023 @ 12:00:26.630    return __callback(*args, **kwargs)
Aug 23, 2023 @ 12:00:26.630  File "/opt/lib/python/site-packages/click/core.py", line 1688, in invoke
Aug 23, 2023 @ 12:00:26.630    return _process_result(sub_ctx.command.invoke(sub_ctx))
Aug 23, 2023 @ 12:00:26.630  File "/opt/lib/python/site-packages/click/decorators.py", line 33, in new_func
Aug 23, 2023 @ 12:00:26.629    sys.exit(main())
Aug 23, 2023 @ 12:00:26.629  File "/opt/lib/python/site-packages/click/core.py", line 1157, in __call__
Aug 23, 2023 @ 12:00:26.629    return self.main(*args, **kwargs)
Aug 23, 2023 @ 12:00:26.629  File "/opt/lib/python/site-packages/click/core.py", line 1078, in main
Aug 23, 2023 @ 12:00:26.629Traceback (most recent call last):
Aug 23, 2023 @ 12:00:26.629  File "/opt/lib/python/site-packages/bin/poppy", line 8, in <module>

That means the pregeneration failed and tiles are stuck in the vandalized version.
For more logs: https://logstash.wikimedia.org/goto/6051764c55891c3384c4b173ea0fdc38

In T344753#9113224, @Jgiannelos wrote:

That means the pregeneration failed and tiles are stuck in the vandalized version.
For more logs: https://logstash.wikimedia.org/goto/6051764c55891c3384c4b173ea0fdc38

Thanks. According to the logs that error was seen once in the last two days on one of the 12 tegola replicas. I don't think that's a generic issue tbh

It looks like kicking the cronjobs manually did the trick cc @jijiki

Can't you just freeze the map as it was three days ago, while the task is in progress?

Unfortunately reverting the map tiles to a specific state from scratch is going to take days.

Ideally, we’d have an api, where trusted ppl can enter a node id or something, and trigger a purge and regenerate for the item and its derivatives. But thats probably gonna be a tad involved to develop.

Looks like wiki maps are fixed now, I can't find this kind of vandalism on them.

...but attacks on OSM are still going, bots running every night, many labels was vandalised on OSM last evening by ruwiki users reports.

Unless we plan to invest some time to make ad-hoc regional tile invalidation thats relatively fast I don't think we have any method other than wait for upstream for fixes and wait for our pipeline to run.

Some features worth trying in the future is:

• Move away from daily tile refresh and instead use stream processing and refresh tiles hourly (same schedule with our OSM sync pipeline)
• Have a command to generate tile refresh events for a given BBOX manually so SREs have a way to respond in such cases