ml-serve2005 is off and ready

This has been solved for now, though needs better docs and possibly simplification, as an extra step is needed:

SLO Dashboard now available here: https://grafana.wikimedia.org/d/slo-Lift_Wing_Recommendation_API_NG/lift-wing-recommendation-api-ng-slo-s?orgId=1

I think we can remove the SRE-Access-Requests tag, since this likely can be entirely covered on the k8s permission level.

Unfortunately, Istio does not export a le=7500 bucket, but only these (the base unit is milliseconds)

Amir set the mentioned repo to archived, so I think we're now done here.

I've created https://gerrit.wikimedia.org/r/c/mediawiki/services/ores/deploy/+/988488 for the deploy repo. While it's unlikely anyone would ever want to use that repo, better safe than sorry.

Both changes have been merged and pushed to ml-staging.

In T347262#9406216, @elukey wrote:

We decided to use 95% for experimental new services, see https://wikitech.wikimedia.org/wiki/SLO/Lift_Wing#Calculate_the_realistic_targets

99% seems to be a lot for the first step, even if the ceiling is higher.

For the availability of the service, I think an SLO similar to our Revscoring and Revertrisk services would be a good fit.

This is complete. I'll track the slash-vs-no-slash matter mentioned above in a separate ticket.

The above mentioned GH PRs have all been merged.

From my workstation at home, both the API endpoint and the spec query work, but there are still things that could improve (more on that below):

Opened these pull requests:

Final version. Unless there are any objections today, I will update the linked PR today and after review,
we can merge it tomorrow. I'll then add this text to the other repos Aiko has mentioned as well.

New version:

Warning: The ORES infrastructure and Revscoring models are being deprecated by the WMF 
Machine Learning team, please check https://wikitech.wikimedia.org/wiki/ORES for more info.

My proposed big warning for those repos:

We're all done here. The to-be-archived repos we'll handle in the separate ticket.

These are the remaining hits in the puppet repo.

@calbon I think this task is outdated, should we just close it?

Images have been built and published:

0.11.2 was released recently. I will update the images to that version before proceeding.

In T347278#9340974, @elukey wrote:

@klausman everything should be done, except the work in T349632, lemme know if anything is missing, otherwise this is done.

Ah, correction: Mikhail mentions that the actual querying of LW is only done one request at a time (the Spark/Large bits are for getting the input data, IIUC). So the delay option seems the most promising.

In T351114#9331004, @mpopov wrote:

@klausman: would introducing a slight delay (e.g. 0.5s) between each of the 34K requests help?

Added to the talk page:

For technical/simplcity matters, the model_info query was simplified and doesn't support all the parameters anymore. I am not all that familiar with the PHP script, but does the output of https://ores.wikimedia.org/v3/scores/?model_info (note absence of =version) still have all the necessary information? If so, this should be easily fixable.

@calbon Can you weigh in on this? AIUI, this would be nontrivial update to Revscoring.