Page MenuHomePhabricator

Activate BM25 on all but wikis with spaceless languages
Closed, ResolvedPublic

Description

The new query builder we plan to use as a replacement for QueryString seems to expose bad tokenization behaviors directly to the users. It does not seem wise to go ahead on these languages without investigating better analysis options for such languages (see T151743 for more details).
In order to move forward and continue to activate BM25 on more wikis we decided to go ahead on languages that use spaces.
We will use the new feature offered by https://gerrit.wikimedia.org/r/#/c/319253/ which allows us to use a language tag.

This task is for tacking the reindex process:

  1. activate BM25 index time config for wikis that use spaces
  2. reindex codfw
  3. activate query time options and switch traffic to codfw (except completion)
  4. reindex eqiad
  5. switch traffic back to eqiad

Event Timeline

dcausse created this task.Dec 1 2016, 12:54 PM
Restricted Application added a project: Discovery. · View Herald TranscriptDec 1 2016, 12:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 324738 had a related patch set uploaded (by DCausse):
[cirrus] enable BM25 on all but wikis with spaecless languages

https://gerrit.wikimedia.org/r/324738

Change 324752 had a related patch set uploaded (by DCausse):
[cirrus] enable BM25 on all but wikis with spaceless languages [step 2/3]

https://gerrit.wikimedia.org/r/324752

Change 324753 had a related patch set uploaded (by DCausse):
[cirrus] enable BM25 on all but wikis with spaceless languages [step 3/3]

https://gerrit.wikimedia.org/r/324753

Change 324738 merged by jenkins-bot:
[cirrus] enable BM25 on all but wikis with spaceless languages [step 1/3]

https://gerrit.wikimedia.org/r/324738

Mentioned in SAL (#wikimedia-operations) [2016-12-06T14:49:04Z] <zfilipin@tin> Synchronized tests/cirrusTest.php: SWAT: [[gerrit:324738|[cirrus] enable BM25 on all but wikis with spaceless languages [step 1/3] (T152092)]] (duration: 00m 43s)

Mentioned in SAL (#wikimedia-operations) [2016-12-06T14:49:59Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:324738|[cirrus] enable BM25 on all but wikis with spaceless languages [step 1/3] (T152092)]] (duration: 00m 44s)

Deskana triaged this task as Medium priority.Dec 8 2016, 6:10 PM
Deskana moved this task from Uncategorised to Scoring and result ordering on the CirrusSearch board.
dcausse updated the task description. (Show Details)Dec 12 2016, 9:39 AM

Change 324752 merged by jenkins-bot:
[cirrus] enable BM25 on all but wikis with spaceless languages [step 2/3]

https://gerrit.wikimedia.org/r/324752

dcausse updated the task description. (Show Details)Dec 12 2016, 2:47 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-12T14:59:33Z] <dcausse> elastic@eqiad: T152092 - reindexing all wikis (except spaceless language wikis) from terbium, logs in ~dcausse/bm25_reindex/cirrus_log

dcausse updated the task description. (Show Details)Dec 15 2016, 10:00 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-15T10:02:10Z] <dcausse> elastic@eqiad: T152092 - reindex done (took 2.5 days)

Change 324753 merged by jenkins-bot:
[cirrus] enable BM25 on all but wikis with spaceless languages [step 3/3]

https://gerrit.wikimedia.org/r/324753

Mentioned in SAL (#wikimedia-operations) [2016-12-15T14:51:12Z] <dcausse@tin> Synchronized tests/cirrusTest.php: 1/2 T152092: [cirrus] enable BM25 on all but wikis with spaceless languages (duration: 00m 40s)

Mentioned in SAL (#wikimedia-operations) [2016-12-15T14:52:22Z] <dcausse@tin> Synchronized wmf-config/InitialiseSettings.php: 2/2 T152092: [cirrus] enable BM25 on all but wikis with spaceless languages (duration: 00m 39s)

dcausse closed this task as Resolved.Dec 15 2016, 3:09 PM
dcausse updated the task description. (Show Details)

done, all traffic's back to eqiad with BM25 enabled.