In T330768#8659732, @akosiaris wrote:

I 've updated https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1 a bit (more could be done, but I 'd rather do it in a consistent automated way across all of the Service folder in grafana instead of one offs) to use thanos instead of the per DC prometheus sources. The picture painted is a consistent one across the last 90d. So, while the Switchover allowed us to revisit this and unearth it once more, it's currently "expected" and "normal" (for some definition of those)

Ignoring 4xx as errors (whether that's a correct assumption or not is a good discussion, my gut says we should count 4xx - I 've updated the panels) and sticking strictly to 5xx, we have an error rate over the last 90 days that is averaging ~6%, with peaks at 12% and lows at 0.5% (those lows are unsurprisingly right on January 1st, so they can be safely discounted). 6% of an error rate is pretty high (it follows that we have an availability of 94%) and there are dependent services (RESTBase) whose health checks are occasionally flapping because of that. As far as I know, the service does not have an SLO so that (pretty high) error rate might be acceptable, but in that case, we need to adapt monitoring to account for that.

I have moved pages where I had permission. On a few wikis, moves have special restrictions. I didn't have enough edits on https://he.wikipedia.org/ and on https://bg.wikipedia.org/ it was detected as vandalism.

In T323169#8466670, @PhotographyEdits wrote:

I think this bug could be closed because it doesn't seem possible to reproduce anymore. Maybe it was some temporary rate limiting.

In T198456#8459069, @DLynch wrote:

I mean, it does interfere, you can no longer enter pmids under 1000 (generally papers before 1975). I'm not a huge fan of this approach consequently.

Sorry, I was unclear -- I meant literally entering "PMID 1" into the field, which should result in it still generating the citation for PMID 1. Just entering "1" would no longer do it, but it should still be possible to make the citation if you know that trick. (But not "pmid 1". It's very specific.)

(I've never tried sticking a gif into here before, so I'm curious how well that'll work.)

In T198456#8445060, @DLynch wrote:

As a starting point, that patch makes it so that pressing “generate” after you’ve entered any number less than 1000 in the auto input will switch to the reuse panel and fill in its search field with that number. The user would then be responsible for clicking the item in the search results that corresponds to the one they meant.

Enhancements to this that'd take slightly more work to implement would be:

1. pressing generate instantly inserting the new citation without requiring confirmation
2. making this behavior trigger when you enter any valid citation reference in the auto input (i.e. named refs would be caught as well, and numbers that aren't references to existing citations would get passed through to the regular citoid behavior)

In T321333#8436059, @ppelberg wrote:

@Mvolz: do you have a sense for what – if anything – is blocking us from enabling Citoid at the Nepali Wikipedia?

That URL works for me now. Possibly not a block, but just slow sometimes?

In T54582#8408107, @Tacsipacsi wrote:

I think there are still a lot of templates that don’t use Lua and thus support exactly as many parameters as documented. They probably don’t show error messages if unknown parameters are used, either, since that would also require Lua. This means that the extra parameters would simply not appear – which doesn’t cause visual breakage, but it’s also hard for the editors to notice. What does Citoid do now if there are more authors/whatever data than what the template – according to TemplateData – supports?

In lieu of a better plan, I think I'm just going to have the Citoid extension "guess" additional parameters from the last one rather naively by incrementing whatever integer is at the end. I.e. if it's last7, the next param will be last8

@akosiaris I can't find anything in the logs... did we stop logging 404s or is it I am bad at finding things?

I've checked, and the urls work locally for me.

In T108980#8323752, @Strainu wrote:

Hi folks, is this task being actively worked on or planned for the near future? If not, maybe we should remove links to https://www.mediawiki.org/wiki/Help:VisualEditor/User_guide/Citations-Full until it can be translated? We're sending people from wikis to these help pages (e.g. https://m.mediawiki.org/wiki/Help:VisualEditor/User_guide/ro ) and we would much rather have them localized.

In T232565#8300968, @Ghuron wrote:

Some identifiers (such as P356) are case-insensitive, but others (such as P3083) are case-sensitive. We have no control over it, since those are identifiers in external sources. As a result, when I'm searching for haswbstatement:"P3083=* b Cen" I'm getting:

1. https://www.wikidata.org/wiki/Q2072696#P3083 (* b Cen)
2. https://www.wikidata.org/wiki/Q4836952#P3083 (* B Cen)

and there is no way I can limit my search to the first result only.
Either case-sensitivity parameter in search or corresponding property text in results (so I can filter second entity by myself) would work for me.

In T242098#8314758, @diegodlh wrote:

I think this is because the examples you mentioned use JavaScript for rendering the final page, and as far as I know the Citoid server does not render JavaScript content. That is, it works on the webpages' source code as returned from the web server.

Realistically, we can't do anything about bad metadata on websites upstream, other than the workaround @diegodlh has mentioned, so closing this!

I'd guess we've been blocked for too much traffic.

I have enabled this, however it probably needs a lot of translation work - I have attempted it using google translate so I expect it has not turned out great

(Looks resolved to me now)

Copying from the other thread -

@akosiaris - looks like for the last week or so we're getting hammered for isbn requests so much it's made the current error rate like 70% -> https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&from=now-30d&to=now&refresh=5m

Another point is that having separate templates for separate wikis would allow wikilinking to pages in the fields (i.e. the publisher, for instance). (Relevant mailing list discussion here: https://lists.wikimedia.org/hyperkitty/list/webtocit@lists.wikimedia.org/thread/HRYDDXA6MLTKBEV477H6FMZ4CBSNVGS4/)

Ah, and only volume but not issue was in the suggested params - so they didn't probably notice you could add the issue separately. Fixed: https://en.wikipedia.org/w/index.php?title=Template:Cite_journal/doc&type=revision&diff=1090606937&oldid=1090605535&diffmode=source

We used to add 01 as the day. This made people mad due to "false precision". As has nearly every other way I've tried to resolve this. There is extensive discussion on the other thread if you want to weigh in there!

A potential fix could be change the template data type - currently issue and volume are both set to "line" and it could be set to "number". However if you have, for instance, a "summer" issue, then that would prevent people from using that. Other than validating it against the type, there's currently no way to validate it more specifically than that.

Now having looked at the examples, these can't be from citoid because they're both pdf links which citoid doesn't do. Potentially user error?

In T305574#7945576, @diegodlh wrote:

At Wikimedia Hackathon 2022's Web2Cit session (T308449), @Mvolz commented that having separate Web2Cit configurations per Wikipedia may be useful for the specific case described in T132308. That is, that incomplete dates returned by Citoid (e.g., 2010-12, meaning December 2010) throw an error in English Wikipedia citation templates, to avoid confusion with date ranges (i.e., 2010-2012). As described there, it was tried with -XX at the end (i.e., 2010-12-XX), but whereas accepted by English Wikipedia, it was rejected by other Wikipedias.

If this is the only case where we would benefit from having different Web2Cit/Citoid response format between Wikipedias, I think that it would be outweighed by the benefits of having one single set of common Web2Cit configurations across Wikipedias (i.e., the benefit of a larger community). I've added a comment to T132308 with an idea of how this could be sorted out at the Citoid extension's level.

@Mvolz, can you think of where we may benefit from having separate Web2Cit configurations per Wikipedia? Thanks!

Tried with 12 and it's fine now :).

In T205870#7838013, @fgiunchedi wrote:

In T205870#7837817, @Mvolz wrote:

This is now deployed for citoid.

This is great to see! Thanks for your help @Mvolz

I have updated grafana for the most part, however there are a few (minor) metrics this broke which relied on the service-runner native ones; quantiles by status and method are broken, but quantiles overall are still working. I'm not sure how to fix them but I'm not sure how essential those are since it's the same info but just broken down a bit.

Garbage collection metrics have been broken for a while, as have memory pod metrics, and that's not related to this change.

ack, please feel free to contact us (SRE o11y) for assistance with the missing/broken metrics if needed

This is now deployed for citoid.

In T291707#7810937, @akosiaris wrote:

In T291707#7401997, @Mvolz wrote:

In terms of a get endpoint, would swagger docs suffice? I started to do something like that but never got around to finishing it :/ https://github.com/zotero/translation-server/issues/76

That would work fine. Many thanks for working on that!

In review -> https://github.com/zotero/translation-server/pull/131

@Mvolz, care to ping upstream again on that one? It's 6 months already.

We had another slew of pages disrupting our weekend as SREs. Pages we could literally do nothing else than just restart things and hope the problem doesn't show up again. This is not sustainable and is demoralizing long term, I am afraid that if this continues SRE will have to pull support for zotero and stop receiving pages for incidents/outages/degradations the service faces.

Judging from the graphs, https://w.wiki/4zi8 we still had spare capacity (14 replicas and at most 9 were fully utilized) and could serve requests. So if we had this get endpoint to use as a readiness probe, kubernetes would have automatically depooled the non responsive instances until they were responsive again leading to no user-visible issues for end users.

It's probable we ran into our request limit with worldcat, but I can't find any obvious spike on the 24th like we've had in the past with bots eating up our requests. It's working for now - we should perhaps improve metrics/ tracking to alert us when we go over our request limit.

In T301510#7745430, @diegodlh wrote:

I've been looking into Citoid API request rate limits.

We access the Citoid API through Wikmedia's RESTBase proxy. I found two 429 HyperSwitch errors for exceeded request rates: https://www.mediawiki.org/wiki/HyperSwitch/errors/rate_exceeded and https://www.mediawiki.org/wiki/HyperSwitch/errors/request_rate_exceeded

Here it says that there is a global limit of up to 200 requests per second, but that individual endpoints may have specific limits. However, the Citoid API documentation doesn't seem to say anything about it.

On the other hand, I found this thread where @Mvolz mentions a "1000/10s (100/s long term, with 1000 burst)" limit.

She also refers to how long requests take and timeouts, but I'm not sure what she means. How does time to response affect request rate limit? Say we make 1000 requests at t=0s of which only 500 have returned a response at t=10s, can we make another 1000-request batch now? Or do pending requests count against our request rate limit?

This is very cool! The short answer is - it would be very difficult to integrate this into the actual service we use. It uses a web scraper (Zotero) and each website is parsed by a different translator.

Unfortunately this is a case where you would have to write a custom translator for zotero (see https://www.mediawiki.org/wiki/Citoid#My_favourite_site_isn't_recognised_by_citoid_and_only_gets_basic_information) for these sites.

In T294010#7482173, @akosiaris wrote:

In T294010#7481262, @Mvolz wrote:

It's ~70 rps at those peaks. They are most definitely violating https://www.mediawiki.org/wiki/API:Etiquette (even if we don't have hard numbers in that page) and we can take action against that. A quick look at turnilo shows a single AWS IP with a user agent of Apache-HttpClient/4.5.6 (Java/1.8.0_265) doing the vast majority of these calls in the last day (>85%).

I 've gone ahead and added them to our abuser lists (in the private repo). It will take some 30 minutes to propagate fully, but after that they should receive back a 403 asking them to contact noc@wikimedia.org.

To be fair, https://en.wikipedia.org/api/rest_v1/ says limit it to 200 r/s.

I had forgotten about that. Good point.

Unfortunately even 1r/s would eat up our quota. We could add more specific documentation to citoid on the page.

Yup, we should do that. Having a global recommendation that is incompatible with more specific services like Citoid isn't good decorum. Thanks for bringing it up!

It's ~70 rps at those peaks. They are most definitely violating https://www.mediawiki.org/wiki/API:Etiquette (even if we don't have hard numbers in that page) and we can take action against that. A quick look at turnilo shows a single AWS IP with a user agent of Apache-HttpClient/4.5.6 (Java/1.8.0_265) doing the vast majority of these calls in the last day (>85%).

I 've gone ahead and added them to our abuser lists (in the private repo). It will take some 30 minutes to propagate fully, but after that they should receive back a 403 asking them to contact noc@wikimedia.org.