Page MenuHomePhabricator

1️⃣ Enable Citoid to read PDF Metadata (MVP version)
Open, Needs TriagePublic

Description

Context

Currently Citoid can't handle pure PDF sources where there's no option to get any metadata from the link alone. There might be valuable information on the source in the PDF's metadata or content thought, that could be used to map to templates and create a more useful reference for users

Idea

We could try to add PDF support to Citoid by adding JS code that loads and analyses PDFs when no other options seems to be working. For now let's work on a POC to see what's possible and how this could be implemented.

Task
  • Look into JS libraries that can analyze PDFs
  • Try to use one of these to get metadata from a PDF
  • Forward that metadata so that the Citoid service can return some useful information to Citoid extension
  • Discuss the idea and next steps
  • Brush PoC into a MVP version
  • Make sure we're not downloading data above a certain limit. Current threshold, 10 MB
  • Add feature flag
Outcome

See some notes https://docs.google.com/document/d/1KRbcDP8Q52lUbvsqwakjAFZk_AyDZg8tdENTgv7F_5o/edit?tab=t.2m8wc9i7fz09#heading=h.19o3kfvemfs0

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Link citoid-service dir to the hostrepos/wmde/docker-dev!15wmde-fischcitoid-devmain
Customize query in GitLab

Event Timeline

WMDE-Fisch renamed this task from Spike: POC to add PDF support for Zotero translation-server to Spike: POC to add PDF support for Citoid.Jun 1 2026, 8:22 AM
WMDE-Fisch updated the task description. (Show Details)

@WMDE-Fisch, @SEgt-WMF shared with me this library that could be useful for this POC:
https://pymupdf.readthedocs.io/en/latest/

Thanks for the link. For now I would concentrate on a JS library though :-).

Currently that's https://github.com/mehmet-kozan/pdf-parse

Change #1301347 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/citoid@master] [POC] Add support for scraping pdf metadata

https://gerrit.wikimedia.org/r/1301347

WMDE-Fisch renamed this task from Spike: POC to add PDF support for Citoid to Enable Citoid to read PDF Metadata (MVP version).Jun 18 2026, 8:19 AM
WMDE-Fisch updated the task description. (Show Details)

Change #1305164 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/extensions/Citoid@master] Add feature flag for PDF support in the citoid service

https://gerrit.wikimedia.org/r/1305164

Change #1305893 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/citoid@master] Add size limit for fetching responses

https://gerrit.wikimedia.org/r/1305893

Change #1305958 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/services/citoid@master] Streamline issueRequest() code a bit

https://gerrit.wikimedia.org/r/1305958

Change #1305958 merged by jenkins-bot:

[mediawiki/services/citoid@master] Streamline issueRequest() code a bit

https://gerrit.wikimedia.org/r/1305958

MareikeHeuerWMDE renamed this task from Enable Citoid to read PDF Metadata (MVP version) to Enable Citoid to read PDF Metadata (MVP version) 1️⃣.Mon, Oct 26, 8:34 AM
lilients_WMDE renamed this task from Enable Citoid to read PDF Metadata (MVP version) 1️⃣ to 1️⃣ Enable Citoid to read PDF Metadata (MVP version).Tue, Oct 27, 8:59 AM