Zotero doesn't do much validation, normally we end up doing validation in citoid.

The NYTimes has been blocking us for a while, it briefly worked when we changed datacenters and ergo IP, but they've understandably reblocked us after a few weeks' reprieve!

I'm having this issue again.

I finally found where the Zotero alerts live, here's the snapshot for the failure: https://logstash.wikimedia.org/app/dashboards#/view/8b1907c0-2062-11ec-85b7-9d1831ce7631?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2024-04-02T12:31:03.210Z',to:'2024-04-04T12:30:53.863Z'))&_a=(description:'Investigate%20alert%20trends%20from%20Icinga%20and%20Alertmanager.',filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'0fade920-6712-11eb-8327-370b46f9e7a5',key:labels.status,negate:!f,params:(query:firing),type:phrase),query:(match_phrase:(labels.status:firing))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'0fade920-6712-11eb-8327-370b46f9e7a5',key:labels.instance,negate:!f,params:(query:'zotero:4969'),type:phrase),query:(match_phrase:(labels.instance:'zotero:4969')))),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:''),timeRestore:!t,title:'Alerts%20overview',viewMode:view)

In T213689#9703471, @Clement_Goubert wrote:

Summing up the discussion on the patch set, this is not what is wanted, turning monitoring on in the service would turn on prometheus metrics scraping, and zotero doesn't expose any metrics. What may be wanted instead is to add a probe of type swagger to the service definition in service.yaml, but I am unsure if x-amples are needed for this to work correctly.

I notice that Zotero is not part of this dashboard: https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?orgId=1

In T361576#9687817, @hnowlan wrote:

After spending some time on this, I think we might have some difficulty with supporting the third citoid pattern of rewriting $site/v1/data/citation/$format/$search to /api?format=$format&search=$search. Envoy's support for this kind of mangling isn't very complete as it's something it kinda expects services to do elsewhere.

I'll still keep looking at it but if there were a way we could just go on with the first two url patterns of $lang/v1/data/citation/api?format=$format&search=$search and $lang/v1/data/citation?format=$format&search=$search being rewritten as /api?format=$format&search=$search it'd be easier.

In T338425#9681026, @hnowlan wrote:

In T338425#9680124, @Mvolz wrote:

So yeah, this might be something we could add to Rest Gateway. Honestly it's more nice to have than anything... alternatively could have people put in their own accept-language header instead of passing on the one in the wiki automatically. Basically affects how people are using the gui api endpoint, not how the service functions inherently.

What sort of logic would this involve? REST Gateway will always pass on headers unless otherwise told to, but having service-specific logic in the gateway outside of generalisable features is something we'd like to avoid if possible.

Things are looking okay after the zotero rollback, so that was probably it: https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad&orgId=1&from=1712086993941&to=1712173393941

Here's a sample log from the probe failing: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2024.04.03?id=5F7rpI4BeyUZP0OuxTwY

More alerts fired since the citoid rollback, so rolled back Zotero at ~7 UTC

More info: I did two deploys at the time, Zotero as well. Zotero can affect the citoid prob since it's called by citoid.

In T309772#9575630, @kostajh wrote:

In T309772#9558607, @akosiaris wrote:

Which teams are aware for this in the WMF? I see service-runner and Security tagged on the task, but I don't know if further communication efforts have happened. Does anyone know?

This is a high usage library in the WMF, it appearing abandoned is pretty bad and some serious effort to fix it should be made before things like T357950 go forward.

I don't know which teams are formally aware of it, but anyone who has built with service-runner or attempted to update its dependencies lately would know.

I believe the project used to be owned by a previous iteration of the MW Platform team, but not sure (looking at Developers/Maintainers and service-runner description didn't provide any answers).

• Remove various methods of stats reporting(statsd and Prometheus) and use one and only one way of reporting stats using a statsd client

In T338425#9679450, @MSantos wrote:

@Mvolz can you elaborate on the reasoning for not adding any compatibility changes? Looking at the resbase code I see that RESTBase does a sanitisation on the accept-language header.

Does that mean REST Gateway will cover that functionality or this is already covered by Citoid and RESTBase code is redundant?

I think this is done here:

Instead of editing the url, Citoid should keep the original url.

I've put the fix into today's afternoon backport window, hopefully we'll get it through (the window is looking a bit full). https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_11

Possibly this backport? https://phabricator.wikimedia.org/T359527

I was hoping today's redeploy of zotero would fix this but it seems to have fixed it only for eqiad but not codfw which is... weird.

Declining, because I was lead to believe this wasn't necessary after all (see abandoned change where I implemented it). Feel free to re-open if that is not longer the case as I can un-abandon it.

Issue was T358597

In T332542#9580358, @Jdforrester-WMF wrote:

CC @Mvolz; we currently can't publish any new versions because all the npm releasers have left, and no-one transferred it to the Wikimedia org.

In T262129#9579262, @YitNat wrote:

This issue also occurs on English wikipedia.. None of the DOI links work today.

It was likely just that you edited it at all that pulled the templatedata through. In the future you can just edit and save without changing anything. Unfortunately a long standing bug.

In T356186#9500883, @Nardog wrote:

I didn't have the Navigation Popups gadget enabled and I still got the error.

It's less acronym salad, so that's good. I worry about the ambiguity of "other", though. It also eliminates the "full citation" though that's less useful now that we only have crossref and no longer have worldcat for book citations.

Reported upstream: https://github.com/zotero/translators/issues/3190

It normally should do that, yes.

Deployed the pmid only fix, but Pubmed central seems to be working as well. They must have unblocked us before I even deployed? Sort of resolved but I want to fix pmc central and book urls as well.

I agree this would be helpful, but it requires us to pick a citation template since we don't actually have any way of deciding one in the absence of the itemType field from the returned metadata.

Kind of resolved (waiting for deploy) but the PR I did only prevents the translator from failing by just skipping the author which is not ideal, so leaving this open for now. The RDF produced by wikidata with the redirect is weird as well as it duplicates authors that aren't even affected, and that went away when the redirect was removed - might also be an upstream bug with the rdf from wikidata?

In T127354#9145039, @Dnshitobu wrote:

"Website" -> Wɛbesite

"Book" -> Gɔŋɔ

"Newspaper article" -> Lasebaarɛ gɔŋɔ

"Journal paper" (as in a scientific journal article or paper)-> Alagumgube gɔŋɔ

"Other work" (any other work to be cited other than the above options) -> Tuuma asi'a

In T127354#8921833, @Dnshitobu wrote:

In T127354#8890520, @Mvolz wrote:

In T127354#8887939, @Dnshitobu wrote:

Currently, we still no manual citation with no automatic citation, we would love to have the citation options of the Dagbani Wikipedia (dag.wikipedia.org)

In T127354#8887853, @Mvolz wrote:

In T127354#8884767, @Dnshitobu wrote:

Hello,
I wish to request support with setting up citations in the Gurene Wikipedia. (gur.wikipedia.org).

In T127354#8884663, @Shahadusadik wrote:

Please enable citoid at the Gurene Wikipedia. gur.wikipedia.org

For this we need to know what citation templates gur wiki uses.

I had a look to see what templates are used there, is this the only one? https://gur.wikipedia.org/wiki/T%C9%9Bmpileti:Cite_web

I note there that historically there was a different template for papers Tɛmpileti:Tlf but this seems to have been deleted? Are there any templates for academic papers or books that I haven't found?

Yes. So, the automatic citation feature depends on what templates the community uses for manual citations.

I notice dag wiki has for example Cite news, but gur has at some point merged the two... in order to set up things we would either need to use only Cite web, or if the community agrees they want it exactly like dag wiki we could import the other templates (it looks like they just imported the ones from en wiki mostly).

Thanks for the efforts. We would love to have the cite web, cite book, cite news and all the other templates that allows for automatic bibliographic citation for references. We look forward to the deployment procedure or processes.