Context
To allow better support for user that use the Citoid service we need to better understand how the service works and especially how it prioritizes the different methods to lookup metadata or return the information to the Citoid extension.
What to look into
- Which methods does Citoid know to analyze the source and create metadata
( e.g. ISBN/DOI/plain URL handling )
- In which order does Citoid evaluate the input?
- What changes could be made to possibly improve that process?
- Is content type negotiation something that's involved?
Ideal outcome
Would be great to have a visualized decision tree.
Example
https://www.pnas.org/doi/pdf/10.1073/pnas.2108146119
Notes
Outcome
Summary
The native Citoid scraper first extracts structured metadata from commonly used metadata standards, including Highwire Press, bePress, Open Graph, and Dublin Core. Extracted identifiers such as DOIs are used to enrich metadata through Crossref where possible. The collected metadata is translated into a unified citation format. If structured metadata is incomplete, additional fallback extraction methods are applied directly to the HTML document.
| Input | Workflow |
|---|---|
| URL | Zotero Web → Native Scraper → DOI-Fallback |
| DOI | Zotero Search → URL Workflow + Crossref |
| PMID/PMCID | Zotero Search → PubMed DOI Lookup → DOI Workflow |
| ISBN | Zotero Search |
| Free text | Crossref Search |
Input type check order
Citoid checks the user input first for different input types in this order:
| # | input type | example | service | |
|---|---|---|---|---|
| 1 | PMC URL | https://pmc.ncbi.nlm.nih.gov/articles/PMC6751825/ | requestFromPM() | |
| 2 | PubMed URL | https://pubmed.ncbi.nlm.nih.gov/33305422/ | requestFromPM() | |
| 3 | HTTP/WWW URL | https://example.com/article | requestFromURL() comment: this would also match for above two types, so it make sense that they happen before, unclear if an overwrite happens often if this is faster | |
| 4 | DOI | 10.1038/nature12373 | requestFromDOI() | |
| 5 | QID | Q42 | requestFromURL() | |
| 6 | PMCID | PMC6751825 | requestFromPM() | |
| 7 | ISBN | 978-3-12-732320-7 | requestToZotISBN() | |
| 8 | PMID | 31533695 | requestFromPM() this is only checked if the input type is not an ISBN | |
| 9 | Domain-like URL | wikipedia.de | requestFromURL() | |
| 10 | Open Search | Book by Douglas Adam  | requestFromSearch() comment: this does not seem to always work | |
| 11 | Embedded URL extraction (additional lookup) | This is an interesting article https://example.com/article | requestFromURL() | |
This information has been extracted from the code (lib/CitoidService.js).
The first match ends the check. This does not necessarily need to be the first one. If the first checks takes longer, a later one could be faster.
This means that results can vary over time.
Metadata retrieval
Next one of the following function is called depending on the input type:
- PMC URL / PubMed URL / PMCID / PMID --> requestFromPM(): Resolve PubMed identifier and retrieve metadata with Zotero. Fallback: get DOI via PubMed & requestFromDOI()
- HTTP/WWW URL / QID / Domain-like URL / Embedded URL extraction --> requestFromURL(): here happens the magic! fetchPubMedIDs, content type check (here are the forbidden pdfs), archive.org check (stored in its own field), if not: Zotero, Fallback: HTML scraper, DOI check
- DOI --> requestFromDOI(): Get metadata of the DOI from Zotero. Fallback: unshorten DOI & requestFromURL() with a timer(!) in parallel: retrieve metadata via CrossRef & translate into Zotero format. When URL request takes too long, take crossref data.
- ISBN --> requestToZotISBN(): Gets the metadata via Zotero.
- Open Search --> requestFromSearch(): Free text search in CrossRef