DjVu image files may contain a machine-readable form of the text
contained in the represented document. It could be useful to
extract this text and include it in the search index for the file's
description page.
It's probably possible to extract text using the DjVuLibre library
or tools.
Version: unspecified
Severity: enhancement
URL: http://djvulibre.djvuzone.org/
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=13370
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T43037 [DO NOT USE] PDF related bugs and enhancements (tracking) | |||
| Resolved | None | T8421 Extract embedded text from DjVu and PDF documents for search | |||
| Resolved | None | T8422 Extract embedded text from PDF documents for search | |||
| Resolved | • Deskana | T23061 Add uploaded file text and metadata from files to fulltext search set | |||
| Resolved | None | T23062 Interface to add more data/text fields for Lucene search engine (eg uploaded file text and metadata) | |||
| Resolved | Ladsgroup | T32906 Store DjVu, PDF extracted text in a structured table instead of img_metadata |
sakthi92 wrote:
Do you mean that the file's description page should be on the search results of the 'extracted text' of the DjVu file ?
I'm pretty sure he means that search results should include hits from within the text layer of the file scans proper. So if the scans include text "Foobar" a search for "bar" would return the text from the scan's text layer as a result even though the text layer had not been extracted and placed on a wiki page. There's no reason that this should be restricted to DjVu's (though there was at the time the bug was filed)
dr.trigon wrote:
As I can see the bug here is quite old and additionally marked "low" in prority. Is this bug up to be fixed at all? In my opinion to solve this bug here is a *must have*.
DrTrigonBot [1] does file content based categorization in commons. Due to this embedded text from PDF (later DJVU too) is extracted and processed. We are currently debating [2] about whether to store this text data to a page - in order to enable the mediawiki search engine to index and find those contents - or not.
Now the question is: When is this bug scheduled to become fixed? Will it be fixed at all? IF NOT; As mentioned DrTrigonBot could dump the files text content to a dedicated page in order to enable the mediawiki search engine to handle them. This should be considered as a work-a-round only and would not be needed at all,
if and when this bug here is solved.
[1] http://commons.wikimedia.org/wiki/User:DrTrigonBot
[2] http://commons.wikimedia.org/wiki/User_talk:DrTrigonBot/JavaScript#PDF_content_extraction
I doubt this will be solved any time soon, there's nobody working on this or related issues and search is a monster nobody really wants to touch AFAIK, so I'd suggest you to implement whatever workaround you think it's worth about 5 more years of usage.
The pages you create should probably be as hidden as possible to users, in particular they shouldn't be indexed by external search engines or they would e.g. "compete" with Wikisource (or even archive.org) which doesn't make any sense.
dr.trigon wrote:
Is it possible to include metadata into the search (indexing) in mediawiki software? As I was informed the text layer gets extracted by Pdf Handler [1] and is stored in the images (PDFs) metadata [2] (name="0").
[1] https://www.mediawiki.org/wiki/Extension:PdfHandler
[2] http://commons.wikimedia.org/w/api.php?action=query&iilimit=500&iiprop=metadata|timestamp&prop=imageinfo&titles=File:Resume-.pdf
Change 101252 had a related patch set uploaded by Brian Wolff:
Begin indexing file text from pdf/djvu files
(In reply to comment #6)
Change 101252 had a related patch set uploaded by Brian Wolff:
Begin indexing file text from pdf/djvu files
Ah, wonderful. That's in CirrusSearch and the core part was already done in https://gerrit.wikimedia.org/r/#/c/99715/ , but there's nothing more specific than the DjVu component so I'm not moving this bug.
Merged. It won't take effect until a full reindex of everything in the file namespace. That'll take a few days after the deployment. Results will start showing up when the document is indexed.
Also, the file text results are with .8 of a page text result from a scoring standpoint.
Finally, this'll work with any files from which mediawiki is able to extract text. If a new file type is plugged in at a later date those files will have to be reindexed for the text to be searchable.
dr.trigon wrote:
Nice! Good job - thanks!
What about including metadata into the search (indexing) as well??
(In reply to comment #10)
Nice! Good job - thanks!
What about including metadata into the search (indexing) as well??
Not a bad idea. We have a bug filed for it somewhere?
dr.trigon wrote:
(In reply to comment #11)
Not a bad idea. We have a bug filed for it somewhere?
What about bug 21061, may be bug 13370 as well?! Or shall I create a new one?
(In reply to comment #12)
(In reply to comment #11)
Not a bad idea. We have a bug filed for it somewhere?
What about bug 21061, may be bug 13370 as well?! Or shall I create a new one?
Those will do great :)
dr.trigon wrote:
Good! I linked them.
Am I wrong or should we now be able to close bug 6422 as well?