Page MenuHomePhabricator
Create Task
Maniphest T266027

Test perfield_builder on spaceless languages
Closed, ResolvedPublic
Actions

Description

As a maintainer of CirrusSearch I would like to reduce the usage of the query_string query so that I can reduce technical debt and verify that the fixes made for T262845 are valid.

The use of the default query builder and classic rescore window are left overs of the switch to BM25 for which the tests were inconclusive (https://wikimedia-research.github.io/Discovery-Search-2ndTest-BM25_jazhth/).
I think it would make sense to re-assess this by running another A/B test because many components related to these languages have changed since then:

  • auto_generate_phrase_queries is no longer available and was probably the cause of the low recall on such languages: T219267
  • there were no dedicated analyzers for chinese, japanese and thai (T158203, T166731, T151743)

I suggest testing:

  • wgCirrusSearchFullTextQueryBuilderProfile: perfield_builder
  • wgCirrusSearchRescoreProfile: wsum_inclinks

on all wikipedias using spaceless languages:

  • bowiki, dzwiki, ganwiki, jawiki, kmwiki, lowiki, mywiki, thwiki, wuuwiki, zhwiki, zh_classicalwiki, yuewiki, zh_yuewiki, bugwiki, cdowiki, crwiki, hakwiki, jvwiki, nanwiki, zh_min_nanwiki

AC:

  • run an A/B test on these wikis
  • provide some data to ensure that the fixes for T262845 are valid

Details

SubjectRepoBranchLines +/-
[cirrus] Swith to perfield builder for spaceless languagesoperations/mediawiki-configmaster+4 -89
[cirrus] setup perfield builder A/B test on spaceless languagesmediawiki/extensions/WikimediaEventswmf/1.36.0-wmf.21+26 -1
[cirrus] setup perfield builder A/B test on spaceless languagesmediawiki/extensions/WikimediaEventswmf/1.36.0-wmf.20+26 -1
[cirrus] setup perfield builder A/B test on spaceless languagesmediawiki/extensions/WikimediaEventsmaster+26 -1
[cirrus] A/B test perfield build on spaceless languagesoperations/mediawiki-configmaster+17 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

dcausse created this task.Oct 20 2020, 2:50 PM
Restricted Application added subscribers: Stang, Aklapper. · View Herald TranscriptOct 20 2020, 2:50 PM
dcausse updated the task description. (Show Details)Oct 20 2020, 2:52 PM
CBogen updated the task description. (Show Details)Oct 20 2020, 2:58 PM
gerritbot added a comment.Oct 20 2020, 3:14 PM

Change 635311 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikimediaEvents@master] [cirrus] setup perfield builder A/B test on spaceless languages

https://gerrit.wikimedia.org/r/635311

gerritbot added a project: Patch-For-Review.Oct 20 2020, 3:14 PM

Change 635313 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] A/B test perfield build on spaceless languages

https://gerrit.wikimedia.org/r/635313

dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Oct 20 2020, 3:28 PM
Shizhao added a project: Chinese-Sites.Oct 21 2020, 1:38 AM
Shizhao moved this task from Backlog to Research on the Chinese-Sites board.
Gehel added a parent task: T262845: Investigate SearchSatisfaction mismatched test buckets.Oct 30 2020, 2:03 PM
gerritbot added a comment.Dec 7 2020, 12:25 PM

Change 635313 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] A/B test perfield build on spaceless languages

https://gerrit.wikimedia.org/r/635313

Stashbot added a comment.Dec 7 2020, 12:28 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-07T12:28:15Z] <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T266027: [cirrus] A/B test perfield build on spaceless languages (duration: 01m 07s)

gerritbot added a comment.Dec 8 2020, 10:14 AM

Change 646777 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.20] [cirrus] setup perfield builder A/B test on spaceless languages

https://gerrit.wikimedia.org/r/646777

gerritbot added a comment.Dec 8 2020, 10:16 AM

Change 635311 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] [cirrus] setup perfield builder A/B test on spaceless languages

https://gerrit.wikimedia.org/r/635311

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.22; 2020-12-15).Dec 8 2020, 11:00 AM
gerritbot added a comment.Dec 8 2020, 12:06 PM

Change 646777 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.20] [cirrus] setup perfield builder A/B test on spaceless languages

https://gerrit.wikimedia.org/r/646777

Maintenance_bot removed a project: Patch-For-Review.Dec 8 2020, 12:10 PM
Stashbot added a comment.Dec 8 2020, 12:28 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-08T12:28:03Z] <dcausse@deploy1001> Synchronized php-1.36.0-wmf.20/extensions/WikimediaEvents/: T266027: [cirrus] setup perfield builder A/B test on spaceless languages (duration: 01m 00s)

gerritbot added a comment.Dec 8 2020, 12:30 PM

Change 646779 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.21] [cirrus] setup perfield builder A/B test on spaceless languages

https://gerrit.wikimedia.org/r/646779

gerritbot added a project: Patch-For-Review.Dec 8 2020, 12:30 PM
gerritbot added a comment.Dec 8 2020, 12:41 PM

Change 646779 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.21] [cirrus] setup perfield builder A/B test on spaceless languages

https://gerrit.wikimedia.org/r/646779

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.21; 2020-12-08); removed MW-1.36-notes (1.36.0-wmf.22; 2020-12-15).Dec 8 2020, 1:00 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 8 2020, 1:10 PM
dcausse moved this task from Needs review to Waiting on the Discovery-Search (Current work) board.Dec 8 2020, 3:26 PM
dcausse added a comment.Dec 8 2020, 5:38 PM

The A/B test is running and events are flowing in. It will be de-activated on Tuesday December 15.

Stashbot added a comment.Dec 16 2020, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-16T19:18:12Z] <dcausse@deploy1001> Synchronized php-1.36.0-wmf.21/extensions/WikimediaEvents/: T266027: Revert [cirrus] setup perfield builder A/B test on spaceless languages (duration: 01m 03s)

Stashbot added a comment.Dec 16 2020, 7:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-16T19:21:33Z] <dcausse@deploy1001> Synchronized php-1.36.0-wmf.22/extensions/WikimediaEvents/: T266027: Revert [cirrus] setup perfield builder A/B test on spaceless languages (duration: 01m 03s)

dcausse moved this task from Waiting to In Progress on the Discovery-Search (Current work) board.Jan 5 2021, 9:11 AM

Moving back to in progress for writing a small report based on the data collected.

dcausse moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Jan 19 2021, 2:34 PM
dcausse added subscribers: EBernhardson, TJones.

Notebook uploaded to https://people.wikimedia.org/~dcausse/T266027%20perfield%20builder%20test%20spaceless%20languages.html

A total of 6,046,684 events have been collected between 2012-12-09T00:00:00Z and 2012-12-16T00:00:00Z, that originates not surprisingly for the most part from chinese, japanese and thai wikipedias.

There are still a non negligible number of sessions where a problem have been detected (e.g. mismatch between frontend and backend), generally below 3% of the collected sessions have to be dropped. 1.25% for chinese, 0.81% for japanese and 2.70% for thai. It does not seem enough to justify investing more work on fixing the search A/B test instrumentation code, reasons might be because of the way the search session starts and how/when the user is selected in a particular group, there are conditions where these are in conflict. As discussed with @EBernhardson it should be the responsibility of the backend to assign a user to a group not the frontend but doing such work is probably out of the scope of the search team.
As to decide if the fixes attempted in T262845 are valid, the answer is no but they allow at least to clean-up the events by detecting suspicious sessions where a mismatch was detected.

On the effect of the perfield builder on these wikis the goal of the A/B test was to make sure that it does not cause any dramatic degradation in the search experience.
Using a very naive satisfaction metric:

  • successful session: at least one fulltext search and at least one click
  • unsuccessful session: at least one fulltext search and no click

We see that there is a slight preference for the perfield builder, control is still prefered on:

  • wuuwiki: decreased from 22.45% to 22.00% of successful sessions
  • zh_yuewiki: decreased from 27.56% to 25.07%
  • bowiki: decreased from 45.45% to 22.22% but is very unlikely to bear any meaning given the number of session collected here (11 for control, 9 for perfield)

On the ZRR side there does not seem to be any noticeable variation and this is what we expected.

As a conclusion I think this is safe to enable the perfield builder on these languages without degrading massively the user experience.

On the analysis itself, I could not reuse the automated A/B report generator as it is currently broken (still depends on mysql), no confidence intervals have been calculated as the goal was not to prove that A is better over B but only verifies that perfield does not degrade the experience in noticeable manner. I'd be happy to update the notebook with some confidence intervals if someone shows me how to that.

TJones added a comment.Jan 19 2021, 9:04 PM

A total of 6,046,684 events have been collected between 2012-12-09T00:00:00Z and 2012-12-16T00:00:00Z,

I think you meant 2020-12, not 2012-12.

Other than that, this looks good to me. Nothing is obviously broken, so let's go for it.

(I do wish our A/B testing infrastructure had not degraded, but that is a task for another ticket.)

dcausse moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Jan 25 2021, 4:38 PM
dcausse moved this task from Needs Reporting to To Be Deployed on the Discovery-Search (Current work) board.
gerritbot added a comment.Jan 27 2021, 10:03 AM

Change 658921 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] Swith to perfield builder for spaceless languages

https://gerrit.wikimedia.org/r/658921

gerritbot added a project: Patch-For-Review.Jan 27 2021, 10:03 AM
gerritbot added a comment.Jan 28 2021, 12:02 PM

Change 658921 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Swith to perfield builder for spaceless languages

https://gerrit.wikimedia.org/r/658921

Stashbot added a comment.Jan 28 2021, 12:07 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T12:07:10Z] <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T266027: [cirrus] Swith to perfield builder for spaceless languages (duration: 01m 06s)

dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Jan 28 2021, 1:42 PM
Gehel closed this task as Resolved.Feb 1 2021, 1:46 PM
Shizhao moved this task from Research to Closed on the Chinese-Sites board.Feb 2 2021, 3:40 AM
Stang unsubscribed.Nov 14 2021, 12:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL