Page MenuHomePhabricator

Improve the way how Citoid extracts DOIs from URLs
Open, Needs TriagePublic

Description

Context

Citoid can not read PDFs but can extract DOIs from URLs. This is already working in a lot of cases. Let's figure out how that can be improved so that users who try to make a citation out of an "unsupported" PDF type can still get a meaningful result back from Citoid based on the DOI lookup.

Tasks
  • Look into how DOIs are extracted from URLS
  • See if that mechanism can be improved
  • Figure out how to log the successful attempts (would be good to have something other than HTTP status code 200, Log level INFO to be able to distinguish the PDF URLs that went through a DOI lookup vs those that were working because the server responds with HTML)
  • Make sure existing workflows are still working the same
Testing

PDF URLs with DOI for testing:

URLIssue
https://www.eucass.eu/doi/EUCASS2025-590.pdfIncomplete DOI in URLnot solvable
https://www.3af-spacepropulsion.com/images/DOI/SPC2026/DocFinal-527-2526-(1).pdfNo DOI in URLnot solvable
https://academic.oup.com/smr/advance-article-pdf/doi/10.1093/sxmrev/qeaf057/64711993/qeaf057.pdfUnclear where DOI endsmight be solvable by checking multiple DOIs
https://www.frontiersin.org/articles/10.3389/fpsyg.2010.00001/pdfUnclear where DOI endscould be solvable by cutting of /pdf or .pdf from the URL
  • More examples of PDF URLs that fail in Citoid and have a DOI: see here (internal document)

Event Timeline

lilients_WMDE renamed this task from Allow PDF URLs that contain DOIs to Enable Citoid to parse DOIs from PDF URLs.Jun 19 2026, 9:29 AM

Change #1306712 had a related patch set uploaded (by Mareike Heuer; author: Mareike Heuer):

[mediawiki/services/citoid@master] Enable Citoid to parse DOIs from PDF URLs

https://gerrit.wikimedia.org/r/1306712

Change #1310073 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/citoid@master] Extract regex matching code into own module

https://gerrit.wikimedia.org/r/1310073

Change #1310125 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/citoid@master] Improve DOI extraction from URLs

https://gerrit.wikimedia.org/r/1310125

WMDE-Fisch renamed this task from Enable Citoid to parse DOIs from PDF URLs to Look into how Citoid parses DOIs from URLs.Jul 14 2026, 7:39 AM
WMDE-Fisch updated the task description. (Show Details)

Change #1306712 abandoned by WMDE-Fisch:

[mediawiki/services/citoid@master] Allow reading DOIs from PDF URLs

Reason:

We're not following this approach anymore. PDF links are already parsed for DOIs and covered by alternative requests.

https://gerrit.wikimedia.org/r/1306712

Change #1310073 merged by jenkins-bot:

[mediawiki/services/citoid@master] Extract regex matching code into own module

https://gerrit.wikimedia.org/r/1310073

Change #1318714 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/citoid@master] Add contentType to response for logging

https://gerrit.wikimedia.org/r/1318714

Change #1318714 merged by jenkins-bot:

[mediawiki/services/citoid@master] Add contentType to response for logging

https://gerrit.wikimedia.org/r/1318714

WMDE-Fisch renamed this task from Look into how Citoid parses DOIs from URLs to Improve the way how Citoid extracts DOIs from URLs.Mon, Aug 3, 9:36 AM
WMDE-Fisch updated the task description. (Show Details)