In theory we can support import / export as translation-server already does this and supports RIS: https://www.zotero.org/support/kb/importing_standardized_formats .

This is actually a totally different feature request so moving into a comment. This task was for the native scraper using this data, not for people to input itself; I'm going to mark this as declined (as we no longer invest in the native scraper since most of this is now done in Zotero), but feel free to make a different task with the village pump idea.

Plausibly a regression in the back end surfacing on the front. FYI mobrovac hasn't worked for the foundation in many years and Mooeypoo hasn't worked on the VE team in many years, so there's not much point in tagging either :).

This is expected behaviour. Some website block us (either passively with anti-bot methods or deliberately) and as a fallback measure we check archive.org. Unfortunately it means can't determine if the website is legitimately dead or just blocking us in order to set the dead flag, so we opted not to set it at all, see discussion in: T374831

This would have been a nice task for a google code-in student, RIP. I think we can close it unless we want to tag it as good first task? It's not really a developer task though.

In T345627#11409512, @elukey wrote:

@Mvolz all merged, the new dashboard is available here, but it seems that we are seeing a lot of errors anyway :(

Plausibly a regression; we try to add an http where the protocol is missing, but I think I touched that code base recently. Tests still pass but an edge case maybe!

Thanks for linking!

I assume the websites fixed their metadata, but closing as a dupe of removing this dependency anyway (which if a future website did the same thing, would still cause an issue until we remove it).

The dashboard now has this: https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=now-30m&to=now&timezone=utc&var-dc=000000026

Declining as we've basically stopped supporting vagrant and everything is docker these days.

In T406362#11246000, @Ponor wrote:

@Mvolz That's what I feared, though I could not imagine why it would change anything in the URL given to it.
I've created a translator for web2cit https://meta.wikimedia.org/wiki/Web2Cit/data/hr/enciklopedija/www/templates.json, which we have enabled for all users on hrwiki (though some still pick the first option, plain Citoid, it seems). Creating one for Zotero would be too much work for me (?), and I feel it would take ages to see it working on our wikis, unlike web2cit.

Declining as T370118 got there first.

Do you know why/ if there any reason why they haven't been submitted upstream to Zotero translators repository? Ideally this would be the approach as it would benefit both citoid and the Zotero project.

I've checked and it works locally, so this seems to be some sort of us - specific issue - but it's a little odd for an intentional block since it's giving us a 504 gate way timeout.

Over the past two weeks ~ 80% of requests are failing with a 401 error. About 20% are 200s.

In T391852#11271062, @elukey wrote:

In T391852#11271042, @Mvolz wrote:

In T391852#11269205, @elukey wrote:

@Mvolz I added two new panels to https://grafana-rw.wikimedia.org/d/NJkCVermz/citoid? in the traffic section, to compare the traffic by HTTP response code as seen by Istio/Envoy/Citoid. Afaics Istio and Envoy agrees on the traffic, meanwhile Citoid shows a slightly higher rate (2 rps vs 3 rps) for the 200s, that I think is the traffic for Zotero.

We'll talk about that tomorrow, but it is expected that the absolute number of 200s should be higher in citoid than in Zotero, because we ask Zotero first and then if we don't get a 200 from them we try a few more things.

For the purposes of the SLO we do not care what Zotero is doing, it's a totally private service and users do not see the end result of Zotero. It's just like querying crossRef, which we also do. That we are hosting the instance should be largely irrelevant (unless it goes down of course!)

I'm still pretty confused how these numbers are so different. All we care about is the response to the user from the citoid service. We don't care about internal calls to Zotero.

I agree 100%, my point is that the HTTP metrics emitted by Citoid that you use for your dashboards may be counting Zotero calls as well, meanwhile the other ones (Istio/Envoy) do not. It would explain the difference in volume size, and it would affect other calculations as well (like non 200 responses over total etc..).

I think I figured it out - the"citation failure" graph for nodejs was incorrectly including requests to _info which was inflating our 200 response rate.

In T391852#11269205, @elukey wrote:

@Mvolz I added two new panels to https://grafana-rw.wikimedia.org/d/NJkCVermz/citoid? in the traffic section, to compare the traffic by HTTP response code as seen by Istio/Envoy/Citoid. Afaics Istio and Envoy agrees on the traffic, meanwhile Citoid shows a slightly higher rate (2 rps vs 3 rps) for the 200s, that I think is the traffic for Zotero.

The issue seems to be with Zotero which is using the Embedded Metadata translator: https://github.com/zotero/translators/blob/master/Embedded%20Metadata.js