Another point is that having separate templates for separate wikis would allow wikilinking to pages in the fields (i.e. the publisher, for instance). (Relevant mailing list discussion here: https://lists.wikimedia.org/hyperkitty/list/webtocit@lists.wikimedia.org/thread/HRYDDXA6MLTKBEV477H6FMZ4CBSNVGS4/)

Ah, and only volume but not issue was in the suggested params - so they didn't probably notice you could add the issue separately. Fixed: https://en.wikipedia.org/w/index.php?title=Template:Cite_journal/doc&type=revision&diff=1090606937&oldid=1090605535&diffmode=source

We used to add 01 as the day. This made people mad due to "false precision". As has nearly every other way I've tried to resolve this. There is extensive discussion on the other thread if you want to weigh in there!

A potential fix could be change the template data type - currently issue and volume are both set to "line" and it could be set to "number". However if you have, for instance, a "summer" issue, then that would prevent people from using that. Other than validating it against the type, there's currently no way to validate it more specifically than that.

Now having looked at the examples, these can't be from citoid because they're both pdf links which citoid doesn't do. Potentially user error?

In T305574#7945576, @diegodlh wrote:

At Wikimedia Hackathon 2022's Web2Cit session (T308449), @Mvolz commented that having separate Web2Cit configurations per Wikipedia may be useful for the specific case described in T132308. That is, that incomplete dates returned by Citoid (e.g., 2010-12, meaning December 2010) throw an error in English Wikipedia citation templates, to avoid confusion with date ranges (i.e., 2010-2012). As described there, it was tried with -XX at the end (i.e., 2010-12-XX), but whereas accepted by English Wikipedia, it was rejected by other Wikipedias.

If this is the only case where we would benefit from having different Web2Cit/Citoid response format between Wikipedias, I think that it would be outweighed by the benefits of having one single set of common Web2Cit configurations across Wikipedias (i.e., the benefit of a larger community). I've added a comment to T132308 with an idea of how this could be sorted out at the Citoid extension's level.

@Mvolz, can you think of where we may benefit from having separate Web2Cit configurations per Wikipedia? Thanks!

Tried with 12 and it's fine now :).

In T205870#7838013, @fgiunchedi wrote:

In T205870#7837817, @Mvolz wrote:

This is now deployed for citoid.

This is great to see! Thanks for your help @Mvolz

I have updated grafana for the most part, however there are a few (minor) metrics this broke which relied on the service-runner native ones; quantiles by status and method are broken, but quantiles overall are still working. I'm not sure how to fix them but I'm not sure how essential those are since it's the same info but just broken down a bit.

Garbage collection metrics have been broken for a while, as have memory pod metrics, and that's not related to this change.

ack, please feel free to contact us (SRE o11y) for assistance with the missing/broken metrics if needed

This is now deployed for citoid.

In T291707#7810937, @akosiaris wrote:

In T291707#7401997, @Mvolz wrote:

In terms of a get endpoint, would swagger docs suffice? I started to do something like that but never got around to finishing it :/ https://github.com/zotero/translation-server/issues/76

That would work fine. Many thanks for working on that!

In review -> https://github.com/zotero/translation-server/pull/131

@Mvolz, care to ping upstream again on that one? It's 6 months already.

We had another slew of pages disrupting our weekend as SREs. Pages we could literally do nothing else than just restart things and hope the problem doesn't show up again. This is not sustainable and is demoralizing long term, I am afraid that if this continues SRE will have to pull support for zotero and stop receiving pages for incidents/outages/degradations the service faces.

Judging from the graphs, https://w.wiki/4zi8 we still had spare capacity (14 replicas and at most 9 were fully utilized) and could serve requests. So if we had this get endpoint to use as a readiness probe, kubernetes would have automatically depooled the non responsive instances until they were responsive again leading to no user-visible issues for end users.

It's probable we ran into our request limit with worldcat, but I can't find any obvious spike on the 24th like we've had in the past with bots eating up our requests. It's working for now - we should perhaps improve metrics/ tracking to alert us when we go over our request limit.

In T301510#7745430, @diegodlh wrote:

I've been looking into Citoid API request rate limits.

We access the Citoid API through Wikmedia's RESTBase proxy. I found two 429 HyperSwitch errors for exceeded request rates: https://www.mediawiki.org/wiki/HyperSwitch/errors/rate_exceeded and https://www.mediawiki.org/wiki/HyperSwitch/errors/request_rate_exceeded

Here it says that there is a global limit of up to 200 requests per second, but that individual endpoints may have specific limits. However, the Citoid API documentation doesn't seem to say anything about it.

On the other hand, I found this thread where @Mvolz mentions a "1000/10s (100/s long term, with 1000 burst)" limit.

She also refers to how long requests take and timeouts, but I'm not sure what she means. How does time to response affect request rate limit? Say we make 1000 requests at t=0s of which only 500 have returned a response at t=10s, can we make another 1000-request batch now? Or do pending requests count against our request rate limit?

This is very cool! The short answer is - it would be very difficult to integrate this into the actual service we use. It uses a web scraper (Zotero) and each website is parsed by a different translator.

Unfortunately this is a case where you would have to write a custom translator for zotero (see https://www.mediawiki.org/wiki/Citoid#My_favourite_site_isn't_recognised_by_citoid_and_only_gets_basic_information) for these sites.

In T294010#7482173, @akosiaris wrote:

In T294010#7481262, @Mvolz wrote:

It's ~70 rps at those peaks. They are most definitely violating https://www.mediawiki.org/wiki/API:Etiquette (even if we don't have hard numbers in that page) and we can take action against that. A quick look at turnilo shows a single AWS IP with a user agent of Apache-HttpClient/4.5.6 (Java/1.8.0_265) doing the vast majority of these calls in the last day (>85%).

I 've gone ahead and added them to our abuser lists (in the private repo). It will take some 30 minutes to propagate fully, but after that they should receive back a 403 asking them to contact noc@wikimedia.org.

To be fair, https://en.wikipedia.org/api/rest_v1/ says limit it to 200 r/s.

I had forgotten about that. Good point.

Unfortunately even 1r/s would eat up our quota. We could add more specific documentation to citoid on the page.

Yup, we should do that. Having a global recommendation that is incompatible with more specific services like Citoid isn't good decorum. Thanks for bringing it up!

It's ~70 rps at those peaks. They are most definitely violating https://www.mediawiki.org/wiki/API:Etiquette (even if we don't have hard numbers in that page) and we can take action against that. A quick look at turnilo shows a single AWS IP with a user agent of Apache-HttpClient/4.5.6 (Java/1.8.0_265) doing the vast majority of these calls in the last day (>85%).

I 've gone ahead and added them to our abuser lists (in the private repo). It will take some 30 minutes to propagate fully, but after that they should receive back a 403 asking them to contact noc@wikimedia.org.

In T294010#7475082, @akosiaris wrote:

Looking at https://grafana.wikimedia.org/d/NJkCVermz/citoid?viewPanel=43&orgId=1&from=now-30d&to=now I think the ban worked. I haven't seen anyone yet reaching out to noc@wikimedia.org but we no longer have that amount of excessive ISBN citation usage. I don't know if we are under quota again though and if everything is working as expected. I did a quick test and it seems to work. @Mvolz, care to have a look?

This paints a nice picture. https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&from=1635185829182&to=1635189452614 something is eating up our isbn requests for about a half hour, they keep going for a half our after that with all the requests getting 404ed, and then the requests stop.

In T294010#7466225, @Dzahn wrote:

Can we ask for an exemption for Wikipedia?

According to WorldCat we're probably exceeding a usage cap of 50,000 requests a day.

In T294010#7462059, @Joe wrote:

If the problem was solved and has come back, I would imagine we get banned from their systems for some reason?

Thank you to @Dzahn in ops for rotating the keys! It looks all fixed now.

I have checked and can confirm locally that the key we use for worldcat in production is no longer working, whereas our other keys still work. However, according to Worldcat dashboard the key has not exceeded usage and is not blocked. I can try rotating the key and see if that works, otherwise will need to get in touch with support. It might have just been dropped from their db or something and reissuing might do the trick.

In terms of a get endpoint, would swagger docs suffice? I started to do something like that but never got around to finishing it :/ https://github.com/zotero/translation-server/issues/76

That would work fine. Many thanks for working on that!

In T291707#7383868, @akosiaris wrote:

In T291707#7383407, @Mvolz wrote:

The PDF connection might be a red herring because although that's what happened in the past, attempting to translate those was disabled here: https://github.com/zotero/translation-server/issues/68

https://github.com/zotero/translation-server/commit/5cf698e20acf8a2fe40219a464f5cbdf7154d24f

https://github.com/zotero/translation-server/commit/9f2c36664fe5f0c3e98ffeb369194f7c242120c3

Locally I've checked and this is still working as expected. So the rejections happen because they get requested a lot, not because it's necessarily related to the crash.

Per those commits, the limit seems to be 50MB. @dstillman is probably right in his comment that it is maybe too high. Can we lower this more/is it configurable now? Like 10MB ? Do we have any use case requiring that we are able to cite objects > 10MB ?

The PDF connection might be a red herring because although that's what happened in the past, attempting to translate those was disabled here: https://github.com/zotero/translation-server/issues/68

In T290717#7357773, @Zabe wrote:

@Mvolz is this still the case?

Could Legal do it?

A slightly edge-casey library is https://github.com/wikimedia/html-metadata which is not on gerrit and is also an npm package, so I'll need to point npm to the new location (Come to think of it I might be the only one with the rights to do that since mobrovac left? need to fix..). Happy to do that earlier rather than later but not be helpful as a template for other repos.

I think at this stage it makes sense to just move to gitlab instead so change the task to that ^-^

In T288811#7282816, @DannyS712 wrote:

Can this be made public if the credentials have been cycled and a new key created?

In T288811#7281764, @Mvolz wrote:

In T288811#7281761, @Mvolz wrote:

In T288811#7281744, @sbassett wrote:

Tagging Release-Engineering-Team as the ostensible owners of gerrit. It looks like the problem config file appears in PS2 and PS3 - not sure if we can remove those patch sets (or at least the config file/diff) from this change set's history?

@Mvolz - has the exposed key been deactivated or rotated? It likely shouldn't be used again, especially not within any kind of production envrionment.

It's not the production key fortunately but a currently unused one intended for beta but never configured. I can get it revoked upstream and ask for a new one.

I guess an easier solution would just be to revoke it actually!

In T288811#7281761, @Mvolz wrote:

In T288811#7281744, @sbassett wrote:

Tagging Release-Engineering-Team as the ostensible owners of gerrit. It looks like the problem config file appears in PS2 and PS3 - not sure if we can remove those patch sets (or at least the config file/diff) from this change set's history?

@Mvolz - has the exposed key been deactivated or rotated? It likely shouldn't be used again, especially not within any kind of production envrionment.

It's not the production key fortunately but a currently unused one intended for beta but never configured. I can get it revoked upstream and ask for a new one.

In T288811#7281744, @sbassett wrote:

Tagging Release-Engineering-Team as the ostensible owners of gerrit. It looks like the problem config file appears in PS2 and PS3 - not sure if we can remove those patch sets (or at least the config file/diff) from this change set's history?

@Mvolz - has the exposed key been deactivated or rotated? It likely shouldn't be used again, especially not within any kind of production envrionment.

I can temporarily silence this if you want, by commenting out the offending test and redeploying, at least for the time being? Doesn't seem to actually affect operation, at least. We used to actually probe www.example.com but that would cause errors like this too, and switched to our own stuff so uptime would be concurrent... is it a temporary DNS issue do you think?

connectivity hiccup probably

In T284345#7137909, @WMDE-Fisch wrote:

While playing with this locally I realized that, npm v7 uses a new lockfile version and whenever I run npm to install the packages it also updates the v1 lockfile to v2 adding a lot of diff overhead. - So I wonder if it might be useful ( when ever we are sure to stick with this version ) to run Libraryupgrader and mass adjust the repos.

Not sure if this a temporary hiccup / just things timing out, and if I should just try to deploy again or something more serious so filing :)

Not all citation templates have a DOI field, and only have a URL field. As long as we have this information, we should provide it, because some consumers will prefer to use the URL. If en wiki prefers to remove these with bots that's a preferred option imo. We can't delete fields that are being used on other wikis in the back-end just because of policy in en wiki.

@zeljkofilipin Thalia's handle is Tchanders but for some reason when I try to actually tag her it breaks Phabricator.

Came across this as well - triggered in reftoolbar (https://en.wikipedia.org/wiki/MediaWiki:RefToolbar.js) found this archived topic where it's suggested it's due to Ace being outdated and it's since been fixed* https://www.mediawiki.org/wiki/Topic:Tat8gxmbmjtpn5dl