I deployed a change today for this. It is not everything we hoped, alas :).

In T367194#9881178, @Krinkle wrote:

Can/does the site in question communicate their enforced limit in an HTTP standardised and machine-readable way? For example, a 429 Status with Retry-After header, or (even better, to avoid hitting a failure first) a crawl limit in robots.txt.

@akosiaris Do you know what the infrastructure for this might look like? Is this something we could do with url downloader? Or would this be built into citoid?

Splitting from the main thread, to inquire about this specifically:

Should be reported upstream to: https://github.com/zotero/translators

I think it's going to be hard to determine pro grammatically with some of these.

I recently noticed the Zotero connectors (the browser extension, so not what we use) has an "unproxify" function:

I deployed a quick change today that reports a 415 if Zotero reports an unsupported media type (T365583) It also supposedly prevents us from re-scraping the page, avoiding downloading the pdf twice once Zotero fails.

I deployed a quick change today that reports a 415 if Zotero reports an unsupported media type (T365583) It also supposedly prevents us from re-scraping the page, avoiding downloading the pdf twice once Zotero fails.

You can now see 415 errors seperate from 404s: https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=1716376689141&to=1716380289141&viewPanel=13

Could this be the cause of T361728? Had failures upgrading Zotero to 18 and it also appears to use mesh 1.4.

In T362379#9805721, @akosiaris wrote:

In T362379#9804663, @SCherukuwada wrote:

@akosiaris

@Mvolz correct me if I am wrong, but Citoid relies almost exclusively to Zotero for workloads and all does thing itself if Zotero fails, right ? I find it difficult that health checks could create so much traffic, hence my asking.

I still think what you said makes sense - I've seen at least one bug where Zotero was stuck in a loop, so knowing what the egress from a single citoid request is would be useful. https://forums.zotero.org/discussion/102507/zotero-causing-continuous-article-download-request-loop

Agreed. But both egress and ingress. This 10MB/s traffic is weird.

In T362379#9804663, @SCherukuwada wrote:

@akosiaris

In the Grafana dash, Saturation -> Total Network says it's about 10 MB/s. Does this count everything the job is doing, including what Zotero might be sending out?

I still think what you said makes sense - I've seen at least one bug where Zotero was stuck in a loop, so knowing what the egress from a single citoid request is would be useful. https://forums.zotero.org/discussion/102507/zotero-causing-continuous-article-download-request-loop

In T300470#9787766, @Patafisik wrote:

Tested today, this is not resolved for me. The automatic citation fails, so I tried switching to manual mode (to fill out the template manually), but the url disappears when I chose the template Cite web.

In T361728#9757621, @Jdforrester-WMF wrote:

Do we have further insight into why the failure rate went up? Anything I can do to help? (A user is asking for a translator update, which led me here.)

In T350880#9762887, @Sebastian_Berlin-WMSE wrote:

Thanks for merging, @Mvolz. How long will it take for this to go live?

See also: T214038

This is stalled I think because it's not clear what we should do. At the very least there should be an error message because PDFs remain our greatest source of failures.

We should definitely change the label to "replace citation".

In T362379#9729595, @jeremyb-phone wrote:

besides statistics about which tools have this much volume would you also drop some kind of time series graph dashboard with request volume and error volume? thank you.

In T362379#9729826, @AlexisJazz wrote:

How about scraping RSS feeds every 10 minutes and caching the results? That's very little traffic. Includes titles, links, author and date.

• https://feeds.npr.org/1001/rss.xml
• https://liaison.reuters.com/
• https://www.nytimes.com/rss

In T362379#9729558, @Samwalton9-WMF wrote:

In T362379#9729541, @Mvolz wrote:

The NYTimes has been blocking us for a while, it briefly worked when we changed datacenters and ergo IP, but they've understandably reblocked us after a few weeks' reprieve!

There's not a whole lot we can do except to ask for IP exemptions - @Samwalton9-WMF would this be something partnerships could try?

Possibly! It's easiest in cases where The Wikipedia Library has an ongoing dialogue, as is the case with Elsevier, who we're currently talking with about this issue.

This is partly a consequence of the fact that over the last few years our traffic has increased a lot, we didn't used to trigger IP blocks as often.

A possible solution would be to close off the API, but that would mean we'd no longer support things like reftoolbar.

We may also want to look into adding blacklists for websites who have expressed they definitely do not want us accessing them to be respectful of that.

Could we set up a system whereby API keys are manually distributed for tools which are going to be used on Wikimedia projects? I'd hope we could find a middle ground between 'fully open' and 'fully closed'. Unless of course tools like reftoolbar are the primary culprit of this increased traffic.

In T350880#9739286, @Sebastian_Berlin-WMSE wrote:

@Mvolz @Jdforrester-WMF, with the new translator added I'd like to have it deployed so it works on WP. How do I do that? As far as I understand the translators used are the ones in the Gitlab repo. Can I just make a merge request to that? It looks like you've done squash merges of the updates in the past.

Zotero doesn't do much validation, normally we end up doing validation in citoid.

The NYTimes has been blocking us for a while, it briefly worked when we changed datacenters and ergo IP, but they've understandably reblocked us after a few weeks' reprieve!

I'm having this issue again.

I finally found where the Zotero alerts live, here's the snapshot for the failure: https://logstash.wikimedia.org/app/dashboards#/view/8b1907c0-2062-11ec-85b7-9d1831ce7631?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2024-04-02T12:31:03.210Z',to:'2024-04-04T12:30:53.863Z'))&_a=(description:'Investigate%20alert%20trends%20from%20Icinga%20and%20Alertmanager.',filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'0fade920-6712-11eb-8327-370b46f9e7a5',key:labels.status,negate:!f,params:(query:firing),type:phrase),query:(match_phrase:(labels.status:firing))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'0fade920-6712-11eb-8327-370b46f9e7a5',key:labels.instance,negate:!f,params:(query:'zotero:4969'),type:phrase),query:(match_phrase:(labels.instance:'zotero:4969')))),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:''),timeRestore:!t,title:'Alerts%20overview',viewMode:view)

In T213689#9703471, @Clement_Goubert wrote:

Summing up the discussion on the patch set, this is not what is wanted, turning monitoring on in the service would turn on prometheus metrics scraping, and zotero doesn't expose any metrics. What may be wanted instead is to add a probe of type swagger to the service definition in service.yaml, but I am unsure if x-amples are needed for this to work correctly.

I notice that Zotero is not part of this dashboard: https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?orgId=1

In T361576#9687817, @hnowlan wrote:

After spending some time on this, I think we might have some difficulty with supporting the third citoid pattern of rewriting $site/v1/data/citation/$format/$search to /api?format=$format&search=$search. Envoy's support for this kind of mangling isn't very complete as it's something it kinda expects services to do elsewhere.

I'll still keep looking at it but if there were a way we could just go on with the first two url patterns of $lang/v1/data/citation/api?format=$format&search=$search and $lang/v1/data/citation?format=$format&search=$search being rewritten as /api?format=$format&search=$search it'd be easier.

In T338425#9681026, @hnowlan wrote:

In T338425#9680124, @Mvolz wrote:

So yeah, this might be something we could add to Rest Gateway. Honestly it's more nice to have than anything... alternatively could have people put in their own accept-language header instead of passing on the one in the wiki automatically. Basically affects how people are using the gui api endpoint, not how the service functions inherently.

What sort of logic would this involve? REST Gateway will always pass on headers unless otherwise told to, but having service-specific logic in the gateway outside of generalisable features is something we'd like to avoid if possible.

Things are looking okay after the zotero rollback, so that was probably it: https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad&orgId=1&from=1712086993941&to=1712173393941

Here's a sample log from the probe failing: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2024.04.03?id=5F7rpI4BeyUZP0OuxTwY

More alerts fired since the citoid rollback, so rolled back Zotero at ~7 UTC

More info: I did two deploys at the time, Zotero as well. Zotero can affect the citoid probe since it's called by citoid.

In T309772#9575630, @kostajh wrote:

In T309772#9558607, @akosiaris wrote:

Which teams are aware for this in the WMF? I see service-runner and Security tagged on the task, but I don't know if further communication efforts have happened. Does anyone know?

This is a high usage library in the WMF, it appearing abandoned is pretty bad and some serious effort to fix it should be made before things like T357950 go forward.

I don't know which teams are formally aware of it, but anyone who has built with service-runner or attempted to update its dependencies lately would know.

I believe the project used to be owned by a previous iteration of the MW Platform team, but not sure (looking at Developers/Maintainers and service-runner description didn't provide any answers).

• Remove various methods of stats reporting(statsd and Prometheus) and use one and only one way of reporting stats using a statsd client

In T338425#9679450, @MSantos wrote:

@Mvolz can you elaborate on the reasoning for not adding any compatibility changes? Looking at the resbase code I see that RESTBase does a sanitisation on the accept-language header.

Does that mean REST Gateway will cover that functionality or this is already covered by Citoid and RESTBase code is redundant?

I think this is done here:

Instead of editing the url, Citoid should keep the original url.

I've put the fix into today's afternoon backport window, hopefully we'll get it through (the window is looking a bit full). https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_11

Possibly this backport? https://phabricator.wikimedia.org/T359527