Extract embedded text from DjVu and PDF documents for search
Closed, ResolvedPublic

Description

DjVu image files may contain a machine-readable form of the text
contained in the represented document. It could be useful to
extract this text and include it in the search index for the file's
description page.

It's probably possible to extract text using the DjVuLibre library
or tools.


Version: unspecified
Severity: enhancement
URL: http://djvulibre.djvuzone.org/
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=13370

bzimport added a project: MediaWiki-DjVu.Via ConduitNov 21 2014, 9:16 PM
bzimport added a subscriber: Unknown Object (MLST).
bzimport set Reference to bz6421.
brion created this task.Via LegacyJun 24 2006, 8:49 AM
bzimport added a comment.Via ConduitMar 10 2012, 8:06 PM

sakthi92 wrote:

Do you mean that the file's description page should be on the search results of the 'extracted text' of the DjVu file ?

Doug added a comment.Via ConduitApr 12 2012, 10:16 PM

I'm pretty sure he means that search results should include hits from within the text layer of the file scans proper. So if the scans include text "Foobar" a search for "bar" would return the text from the scan's text layer as a result even though the text layer had not been extracted and placed on a wiki page. There's no reason that this should be restricted to DjVu's (though there was at the time the bug was filed)

bzimport added a comment.Via ConduitSep 4 2012, 3:32 PM

dr.trigon wrote:

As I can see the bug here is quite old and additionally marked "low" in prority. Is this bug up to be fixed at all? In my opinion to solve this bug here is a *must have*.

DrTrigonBot [1] does file content based categorization in commons. Due to this embedded text from PDF (later DJVU too) is extracted and processed. We are currently debating [2] about whether to store this text data to a page - in order to enable the mediawiki search engine to index and find those contents - or not.

Now the question is: When is this bug scheduled to become fixed? Will it be fixed at all? IF NOT; As mentioned DrTrigonBot could dump the files text content to a dedicated page in order to enable the mediawiki search engine to handle them. This should be considered as a work-a-round only and would not be needed at all,
if and when this bug here is solved.

[1] http://commons.wikimedia.org/wiki/User:DrTrigonBot
[2] http://commons.wikimedia.org/wiki/User_talk:DrTrigonBot/JavaScript#PDF_content_extraction

Nemo_bis added a comment.Via ConduitSep 4 2012, 3:53 PM

I doubt this will be solved any time soon, there's nobody working on this or related issues and search is a monster nobody really wants to touch AFAIK, so I'd suggest you to implement whatever workaround you think it's worth about 5 more years of usage.
The pages you create should probably be as hidden as possible to users, in particular they shouldn't be indexed by external search engines or they would e.g. "compete" with Wikisource (or even archive.org) which doesn't make any sense.

bzimport added a comment.Via ConduitSep 10 2012, 12:08 AM

dr.trigon wrote:

Is it possible to include metadata into the search (indexing) in mediawiki software? As I was informed the text layer gets extracted by Pdf Handler [1] and is stored in the images (PDFs) metadata [2] (name="0").

[1] https://www.mediawiki.org/wiki/Extension:PdfHandler
[2] http://commons.wikimedia.org/w/api.php?action=query&iilimit=500&iiprop=metadata|timestamp&prop=imageinfo&titles=File:Resume-.pdf

gerritbot added a comment.Via ConduitDec 14 2013, 3:54 AM

Change 101252 had a related patch set uploaded by Brian Wolff:
Begin indexing file text from pdf/djvu files

https://gerrit.wikimedia.org/r/101252

Nemo_bis added a comment.Via ConduitDec 14 2013, 7:14 AM

(In reply to comment #6)

Change 101252 had a related patch set uploaded by Brian Wolff:
Begin indexing file text from pdf/djvu files

https://gerrit.wikimedia.org/r/101252

Ah, wonderful. That's in CirrusSearch and the core part was already done in https://gerrit.wikimedia.org/r/#/c/99715/ , but there's nothing more specific than the DjVu component so I'm not moving this bug.

gerritbot added a comment.Via ConduitDec 26 2013, 9:41 PM

Change 101252 merged by jenkins-bot:
Index and search file text from pdf/djvu files

https://gerrit.wikimedia.org/r/101252

Manybubbles added a comment.Via ConduitDec 26 2013, 9:44 PM

Merged. It won't take effect until a full reindex of everything in the file namespace. That'll take a few days after the deployment. Results will start showing up when the document is indexed.

Also, the file text results are with .8 of a page text result from a scoring standpoint.

Finally, this'll work with any files from which mediawiki is able to extract text. If a new file type is plugged in at a later date those files will have to be reindexed for the text to be searchable.

bzimport added a comment.Via ConduitDec 28 2013, 10:42 AM

dr.trigon wrote:

Nice! Good job - thanks!

What about including metadata into the search (indexing) as well??

demon added a comment.Via ConduitDec 28 2013, 7:47 PM

(In reply to comment #10)

Nice! Good job - thanks!

What about including metadata into the search (indexing) as well??

Not a bad idea. We have a bug filed for it somewhere?

bzimport added a comment.Via ConduitDec 29 2013, 9:44 AM

dr.trigon wrote:

(In reply to comment #11)

Not a bad idea. We have a bug filed for it somewhere?

What about bug 21061, may be bug 13370 as well?! Or shall I create a new one?

demon added a comment.Via ConduitDec 29 2013, 7:16 PM

(In reply to comment #12)

(In reply to comment #11)
> Not a bad idea. We have a bug filed for it somewhere?

What about bug 21061, may be bug 13370 as well?! Or shall I create a new one?

Those will do great :)

bzimport added a comment.Via ConduitDec 29 2013, 10:08 PM

dr.trigon wrote:

Good! I linked them.

Am I wrong or should we now be able to close bug 6422 as well?

Aklapper added a project: Wikisource.Via WebMar 10 2015, 4:16 PM

Add Comment