ORES/LiftWing infrastructure is down in the Russian Wikipedia
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Q-bit-array
	Apr 15 2024, 7:37 AM

Description

Error

Since about 16-18 hours the ORES/LiftWing infrastructure is completely down in the Russian Wikipedia.

All ORES/LiftWing-related services are not operational:

LiftWing API endpoint is down (https://api.wikimedia.org/service/lw/inference/v1/models/ruwiki-damaging:predict)
ORES-legacy API endpoint is down (https://ores-legacy.wikimedia.org/v3/scores/ruwiki/?models=damaging)
ORES/LiftWing scores in the MediaWiki database are not present
Highlighting of the problematic/damaging edits on the "Recent changes" page does not work (https://ru.wikipedia.org/wiki/Служебная:Свежие_правки?userExpLevel=unregistered&hidebots=1&hidecategorization=1&hideWikibase=1&hidelog=1&limit=1000&days=7&damaging__likelybad_color=c4&damaging__verylikelybad_color=c5&urlversion=2)

A sample Curl request to the LiftWing API returns a HTTP 504 error code. The same API request works for the English Wikipedia

curl output examples

$ curl https://api.wikimedia.org/service/lw/inference/v1/models/ruwiki-damaging:predict -X POST -d '{"rev_id": 137258595}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    78  100    57    0    21      1      0  0:00:57  0:00:30  0:00:27    13{"httpReason":"upstream request timeout","httpCode":504}


$ curl https://api.wikimedia.org/service/lw/inference/v1/models/enwiki-damaging:predict -X POST -d '{"rev_id": 137258595}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   213  100   192  100    21    351     38 --:--:-- --:--:-- --:--:--   391{"enwiki":{"models":{"damaging":{"version":"0.5.1"}},"scores":{"137258595":{"damaging":{"score":{"prediction":false,"probability":{"false":0.9931303628447447,"true":0.006869637155255367}}}}}}}

Impact

Because of the outage, almost all our antivandal tools, which use ORES/LiftWing, are currently not operational. We are also not able to use the recent changes page to manually monitor problematic edits, as they are not highlighted any more.

Thanks in advance for fixing the issue!