Page MenuHomePhabricator
Create Task
Maniphest T34871

Search indexes limited to first 100k words (MAX_FIELD_LENGTH)
Closed, DeclinedPublic
Actions

Description

Author: simon

Description:
There's a suggestion currently at http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Web_scraping_tool_for_article_research_.28list_expansion.29 that the search indexes only the first 100k words in a page.

This means that important stuff at the bottom of a very long page is not included in the index, which is a bad thing.

Is there any possibility this restriction - if it exists - could be lifted such that all of the text is indexed?

Version: unspecified
Severity: normal

Details

Reference
bz32871

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 22 2014, 12:05 AM
• bzimport added a project: Wikimedia-lucene-search-2.
• bzimport set Reference to bz32871.
• bzimport added a subscriber: Unknown Object (MLST).
• bzimport created this task.Dec 8 2011, 10:19 AM
hashar added a comment.Dec 8 2011, 10:23 AM

Section was renamed on village pump. Permalink:

http://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&oldid=464743644#Web_scraping_tool_for_article_research_.28list_expansion.29

• bzimport added a comment.Dec 8 2011, 10:23 AM

rainman wrote:

http://svn.wikimedia.org/viewvc/mediawiki/trunk/lucene-search-2/src/org/wikimedia/lsearch/index/WikiIndexModifier.java?revision=63824&view=markup

static public final int MAX_FIELD_LENGTH = 100000;

It could be increased, however, I don't remember offhand what were the issues with increasing this number.

• bzimport added a comment.Dec 8 2011, 7:23 PM

simon wrote:

I got the link to the VP wrong - try http://en.wikipedia.org/w/index.php?title=Wikipedia%3AVillage_pump_%28technical%29&action=historysubmit&diff=464743644&oldid=464742465#Archive_search_bug.3F for the original complaint.

Aklapper added a comment.Mar 26 2013, 11:20 AM

[Merging "MediaWiki extensions/Lucene Search" into "Wikimedia/lucene-search2", see bug 46542. You can filter bugmail for: search-component-merge-20130326 ]

• demon added a comment.Oct 29 2013, 2:12 AM

lsearchd has reached end of life and will not be improved further. Marking this WONTFIX as a result.

We don't have this limit in CirrusSearch.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits