Page MenuHomePhabricator
Create Task
Maniphest T284185

Reindex German, Dutch, and Portugese Wikis to Enabled Unpacked Versions
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Once T281379 is deployed in MediaWiki_1.37/wmf.8, we can reindex the relevant wikis, to activate ICU normalization, ICU folding, and homoglyph normalization.

Current counts are: German—9 wikis; Dutch—9 wikis, Portuguese—9 wikis (excluding 1 pt-br wiki).

Acceptance Criteria

  • All wikis in the relevant languages are reindexed
  • A before-and-after analysis for each language's Wikipedia is provided

Related Objects
Search...

Event Timeline

TJones created this task.Jun 2 2021, 7:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 2 2021, 7:59 PM
TJones mentioned this in T147505: [tracking] CirrusSearch: what is updated during re-indexing.Jun 2 2021, 8:01 PM
TJones mentioned this in T104814: Appropriately ignore diacritics for German-language wikis.Jun 2 2021, 8:03 PM
TJones mentioned this in T226812: de.wikipedia: search for "Bedusz" does not find "Będusz".
TJones added a parent task: T281379: Unpack German, Portuguese, and Dutch Elasticsearch Analyzers.Jun 8 2021, 3:44 PM
Gehel added a subtask: T226812: de.wikipedia: search for "Bedusz" does not find "Będusz".Jun 9 2021, 3:09 PM
Gehel added a subtask: T104814: Appropriately ignore diacritics for German-language wikis.
MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board.Jun 9 2021, 3:14 PM
MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search.
MPhamWMF set the point value for this task to 3.Jun 9 2021, 3:43 PM
MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
TJones updated the task description. (Show Details)Jun 9 2021, 6:58 PM
TJones updated the task description. (Show Details)Jun 9 2021, 7:21 PM
TJones claimed this task.Jun 21 2021, 6:42 PM
TJones moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
Stashbot added a comment.Jun 22 2021, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T13:29:00Z] <Trey314159> reindexing German wikis on elastic@eqiad, elastic@codfw, and cloudelastic complete (T284185)

Stashbot added a comment.Jun 22 2021, 4:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T16:41:19Z] <Trey314159> reindexing Dutch wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T284185)

Stashbot added a comment.Jun 22 2021, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T20:12:37Z] <Trey314159> reindexing Dutch wikis on elastic@eqiad, elastic@codfw, and cloudelastic complete (T284185)

Stashbot added a comment.Jun 22 2021, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-22T20:12:55Z] <Trey314159> reindexing Portuguese wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T284185)

Stashbot added a comment.Jun 23 2021, 12:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-23T00:02:32Z] <Trey314159> reindexing Portuguese wikis on elastic@eqiad, elastic@codfw, and cloudelastic complete (T284185)

TJones renamed this task from Reindex German, Dutch, and Portugese Wikis to Reindex German, Dutch, and Portugese Wikis to Enabled Unpacked Versions.Jun 23 2021, 10:03 PM
TJones added a comment.Jul 2 2021, 10:32 PM

More details on Mediawiki: DE/NL/PT Reindexing Impacts.

Summary:

  • All
    • ICU Folding is the biggest driver of zero-results improvements and increased number of results, usually because of missing diacritics.
    • Reindexing seems to decrease discrepancies between shards, which lowers the incidence rate of queries with a different top result
    • Improved filtering, because in German 6 consonants in a row is perfectly normal; also improved URL and email detection, and filtered very long queries with zero results (another form of junk).
  • German (3K sample)
    • I also disabled the folding of ß to ss in the plain field, as per T87136.
    • Zero results rate dropped by -0.3% absolute change; -1.4% relative change
    • 11.5% of queries got more results in general
    • The query was heisst s.w.a.t. ("what does S.W.A.T. do?") dropped from 3369 down to 67 results because ß and ss no longeer match in the plain field, and s.w.a.t. is not broken up in the text field (word_break_helper is not enabled)
  • Dutch (3K sample)
    • Zero results rate dropped by -0.1% absolute change; -0.4% relative change
    • 8.0% of queries got more results in general
  • Portuguese (3K sample)
    • Zero results rate dropped by -0.4% absolute change; -2.1% relative change
    • 15.3% of queries got more results in general
    • Missing tildes (a instead of ã, or o instead of õ) are the biggest sources of changes
    • The query 1926~ got a lot more hits, but I don't know why. Weird.
TJones moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Jul 2 2021, 10:32 PM
TJones closed subtask T104814: Appropriately ignore diacritics for German-language wikis as Resolved.Jul 19 2021, 2:27 PM
MPhamWMF moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Jul 19 2021, 3:35 PM
Gehel closed subtask T226812: de.wikipedia: search for "Bedusz" does not find "Będusz" as Resolved.Jul 26 2021, 12:18 PM
Gehel closed this task as Resolved.Jul 26 2021, 12:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL