Context
Currently Citoid can't handle pure PDF sources where there's no option to get any metadata from the link alone. There might be valuable information on the source in the PDF's metadata or content thought, that could be used to map to templates and create a more useful reference for users
Idea
We could try to add PDF support to Citoid by adding JS code that loads and analyses PDFs when no other options seems to be working. For now let's work on a POC to see what's possible and how this could be implemented.
Task
- Look into JS libraries that can analyze PDFs
- Try to use one of these to get metadata from a PDF
- Forward that metadata so that the Citoid service can return some useful information to Citoid extension
- Discuss the idea and next steps
- Brush PoC into a MVP version
- Make sure we're not downloading data above a certain limit. Current threshold, 10 MB
- Add feature flag
- Talk to SRE/Operations about any concerns from their side --> T430305: RfC: Citoid service performance assessment for PDF parsing feature
- Adapt code to follow the recommendations by SRE/Operations: T439362: 1️⃣ Introduce size limit for downloading URL sources
Outcome
- I used pdf-parse https://www.npmjs.com/package/pdf-parse
- Patch https://gerrit.wikimedia.org/r/c/mediawiki/services/citoid/+/1301347
See some notes https://docs.google.com/document/d/1KRbcDP8Q52lUbvsqwakjAFZk_AyDZg8tdENTgv7F_5o/edit?tab=t.2m8wc9i7fz09#heading=h.19o3kfvemfs0