Page MenuHomePhabricator
Create Task
Maniphest T435179

ISBN requests hitting Library of Congress rate limits
Open, MediumPublic
Actions

Description

We use Library of Congress in Zotero to look up isbns. Temporary IP blocks are made if we exceed 10 requests in one minute.

They return a 200 without a machine readable cool-down for IP blocks, and the message is displayed in html.

https://lx2.loc.gov/sru/lcdb?operation=searchRetrieve&version=1.1&query=bath.ISBN=^9781912454785&maximumRecords=1

Compounding the issue is that translation-server doesn't handle the block correctly and chokes.

Short term:

Long-term

  • Rate limit ISBN requests and/or bot requests
  • Try to get a contract with WorldCat again
  • Disallow bot requests for isbns entirely

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
zotero: update to latestMvolzoperations/deployment-chartsmaster+1 -1
zotero: update versionMvolzoperations/deployment-chartsmaster+1 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Update translators submodulerepos/mediawiki/services/zotero!26mvolzwork/mvolz/updatemain
Lower priority and improve open library translatorrepos/mediawiki/services/zotero-translators!16mvolzupdateolisbnmain
Add Open Library ISBN search translatorrepos/mediawiki/services/zotero-translators!15mvolzopenlibraryisbnmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

Mvolz created this task.Tue, Aug 18, 10:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Aug 18, 10:04 AM
Mvolz renamed this task from Library of Congress is likely rate limiting Zotero to Library of Congress is likely rate limiting us.Tue, Aug 18, 10:24 AM
Mvolz updated the task description. (Show Details)
Tobi_WMDE_SW subscribed.Tue, Aug 18, 11:27 AM
Johannes_Richter_WMDE subscribed.Tue, Aug 18, 4:09 PM
NightWolf1223 subscribed.Tue, Aug 18, 8:06 PM
Mvolz renamed this task from Library of Congress is likely rate limiting us to 100% failure for ISBN requests.Wed, Aug 19, 1:02 PM
Mvolz updated the task description. (Show Details)
Mvolz updated the task description. (Show Details)
Mvolz updated the task description. (Show Details)
Mvolz added subscribers: sjvipin, Samwalton9-WMF.
CodeReviewBot added a project: Patch-For-Review.Wed, Aug 19, 2:08 PM

mvolz opened https://gitlab.wikimedia.org/repos/mediawiki/services/zotero/-/merge_requests/21

Fix IP blocks from LoC causing all ISBN requests to fail

Mvolz triaged this task as High priority.Thu, Aug 20, 9:24 AM
Mvolz updated the task description. (Show Details)
Mvolz added a project: Editing-team (FY2026-27 Q1 sprint 5).Fri, Aug 21, 11:31 AM
Mvolz moved this task from Incoming to Code review on the Editing-team (FY2026-27 Q1 sprint 5) board.
Mvolz edited projects, added: Editing-team (FY2026-27 Q1 sprint 4); removed: Editing-team (FY2026-27 Q1 sprint 5).Fri, Aug 21, 12:28 PM
Mvolz moved this task from Incoming to Code review on the Editing-team (FY2026-27 Q1 sprint 4) board.
Lina_Farid_WMDE subscribed.Mon, Aug 24, 11:43 AM
Mvolz updated the task description. (Show Details)Mon, Aug 24, 1:26 PM
ppelberg assigned this task to dchan.Mon, Aug 24, 5:20 PM
CodeReviewBot added a comment.Mon, Aug 24, 7:25 PM

jforrester merged https://gitlab.wikimedia.org/repos/mediawiki/services/zotero/-/merge_requests/21

Fix IP blocks from LoC causing all ISBN requests to fail

Maintenance_bot removed a project: Patch-For-Review.Mon, Aug 24, 7:31 PM
gerritbot added a comment.Tue, Aug 25, 9:50 AM

Change #1329240 had a related patch set uploaded (by Mvolz; author: Mvolz):

[operations/deployment-charts@master] zotero: update version

https://gerrit.wikimedia.org/r/1329240

gerritbot added a project: Patch-For-Review.Tue, Aug 25, 9:51 AM
Mvolz updated the task description. (Show Details)Tue, Aug 25, 10:11 AM
CodeReviewBot added a comment.Tue, Aug 25, 2:58 PM

mvolz opened https://gitlab.wikimedia.org/repos/mediawiki/services/zotero-translators/-/merge_requests/15

Add Open Library ISBN search translator

Quiddity added a project: User-notice.Tue, Aug 25, 7:58 PM
Quiddity subscribed.

Tech News draft-entry, which might need to be revised, as devs are still working on very-short-term potential solutions:

The ISBN lookup tools for generating citations are currently not working because of external service problems. Developers are working on solutions. [1]

Quiddity moved this task from To Triage to Announce in next Tech/News on the User-notice board.Tue, Aug 25, 7:58 PM
CodeReviewBot added a comment.Wed, Aug 26, 9:51 AM

mvolz merged https://gitlab.wikimedia.org/repos/mediawiki/services/zotero-translators/-/merge_requests/15

Add Open Library ISBN search translator

Mvolz added a subtask: T435916: Consider adding a php interface over the citoid api in the Citoid extension.Wed, Aug 26, 10:03 AM
gerritbot added a comment.Wed, Aug 26, 11:02 AM

Change #1329240 merged by jenkins-bot:

[operations/deployment-charts@master] zotero: update version

https://gerrit.wikimedia.org/r/1329240

Mvolz added a comment.Wed, Aug 26, 3:24 PM

Screenshot From 2026-08-26 16-22-14.png (516×411 px, 36 KB)

Since Zotero was deployed with the fixes, there's been a big improvement in isbn success ratio. Some degree of failures is expected because these open databases unfortunately aren't as comprehensive as some of the closed ones.

For the future: it's not super easy to tell if LoC is blocking us because logs in Zotero are not great.

Mvolz closed this task as Resolved.Wed, Aug 26, 3:25 PM
Mvolz claimed this task.
Mvolz updated the task description. (Show Details)
Mvolz added a subscriber: dchan.

Resolving because it's not 100% anymore, but feel free to re-open and/or rename.

STei-WMF moved this task from Announce in next Tech/News to In current Tech/News draft on the User-notice board.Thu, Aug 27, 1:45 PM
Nux reopened this task as Open.Mon, Aug 31, 9:38 PM
Nux subscribed.

The priority for the Open Library should be made so it is less important then most if not all national libraries.

National libraries and priorities (the lower value the more important it is):

Of which I am certain that Polish national library (PL-NL) works, and also serves data only for specific ISBNs (see function detectSearch(item)), and has better data then the Open Library.

I think OL should have a priority set to 101 to work as a fallback or at least change it to 99 to go after more specialised PL-NL.

See also:

Mvolz added a comment.EditedTue, Sep 1, 7:51 AM

@Nux curious whether you've hit any bad metadata accordingly?

LoC has priority 97 upstream: https://github.com/zotero/translators/blob/master/Library%20of%20Congress%20ISBN.js

I lowered it to 100 in these changes because we were hitting their rate limits because of this.

The rationale here is that OL has imputed the Library of Congress dumps that were released in 2016 and so in theory all of that metadata should be pretty good. https://loc.gov/cds/products/MDSConnect-books_all.html

So, in theory, by hitting OL first we're just getting the older LoC data, which leaves us some spare requests to get an ISBN from 2016 onward. However I guess if OL contains 2016 onward metadata that's messy, this is potentially worse.

We can potentially lower OL below the other libraries and lower LoC even further than 100, but I think this would require some more research about rate limits for these libraries. OL is safely within range within our typical usage, these other libraries may not be, so I think that'd have to be investigated before raising priority.

Mvolz renamed this task from 100% failure for ISBN requests to ISBN requests hitting Library of Congress rate limits.Tue, Sep 1, 7:52 AM
Mvolz lowered the priority of this task from High to Medium.
Nux added a comment.EditedTue, Sep 1, 7:57 AM

LoC has priority 97 upstream: https://github.com/zotero/translators/blob/master/Library%20of%20Congress%20ISBN.js

Oh, right, I missed that. Even so, not hitting LoC for a Polish ISBN would've hit the National Library of Poland. Now it'll hit OL, as OL has bibliographic descriptions of books in various languages, including Polish.

MLechvien-WMF mentioned this in T435916: Consider adding a php interface over the citoid api in the Citoid extension.Tue, Sep 1, 8:22 AM
Mvolz moved this task from Next & Doing to Zotero on the Citoid board.Tue, Sep 1, 12:09 PM
tfmorris subscribed.Tue, Sep 1, 7:30 PM

However I guess if OL contains 2016 onward metadata that's messy, this is potentially worse.

OpenLibrary contains metadata from Amazon resellers (uneven, often poor, quality) and Better World Books (terrible quality), as well as user contributed metadata. Booksellers have fundamentally different goals and quality standards than library catalogers.

DLynch edited projects, added: Editing-team (FY2026-27 Q1 sprint 5); removed: Editing-team (FY2026-27 Q1 sprint 4).Wed, Sep 2, 4:54 PM
STei-WMF moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Thu, Sep 3, 11:28 AM
CodeReviewBot added a comment.Fri, Sep 11, 9:16 AM

mvolz opened https://gitlab.wikimedia.org/repos/mediawiki/services/zotero-translators/-/merge_requests/16

Lower priority and improve open library translator

CodeReviewBot added a comment.Fri, Sep 11, 2:46 PM

jforrester merged https://gitlab.wikimedia.org/repos/mediawiki/services/zotero-translators/-/merge_requests/16

Lower priority and improve open library translator

Nux added a comment.Sat, Sep 12, 11:00 AM

Thank you. I appreciate the change :-). As long as WebArchive doesn't get defunded this is a good change :-)

Mvolz updated the task description. (Show Details)Sat, Sep 12, 1:47 PM
CodeReviewBot added a comment.Mon, Sep 14, 8:59 AM

mvolz opened https://gitlab.wikimedia.org/repos/mediawiki/services/zotero/-/merge_requests/26

Update translators submodule

CodeReviewBot added a comment.Tue, Sep 15, 11:47 AM

jforrester merged https://gitlab.wikimedia.org/repos/mediawiki/services/zotero/-/merge_requests/26

Update translators submodule

gerritbot added a comment.Tue, Sep 15, 12:35 PM

Change #1341867 had a related patch set uploaded (by Mvolz; author: Mvolz):

[operations/deployment-charts@master] zotero: update to latest

https://gerrit.wikimedia.org/r/1341867

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits