Perform A/B test for search location change
Closed, DuplicatePublic
Actions

Assigned To

Authored By

	ovasileva
	Aug 31 2020, 2:44 PM

Description

We would like to do two A/B test on the changes planned to the search widget as a part of the desktop improvements project:

Moving the search bar to the header of the page (target 2.5% overall increase in search sessions initiated, monitor and report on search sessions completed)
Switching the current search widget to the new search widget (target 2.5% overall increase in search sessions initiated, monitor and report on search sessions completed)
Individual test will be performed per wiki for logged-in user only (nice to have: all users)

Additional background in T256100, T249366, T251740, and T257698.

This tasks covers the a/b test for changing the location of the search widget

Acceptance criteria

Enable A/B test to all test wikis
Run test for 2 weeks
Disable A/B test

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T49145 Formally deprecate jQuery UI after we've stopped using jQuery UI in extensions and core
Open		None	T100270 Replace use of jQuery UI and MW UI with OOUI across all Wikimedia-deployed extensions and core
Open		None	T85394 Use OOUI suggestions/autocompletion components only (instead of jquery.suggestions, jquery.ui.autocomplete)
Open		None	T125725 [epic] Update autocomplete search box with metadata and remove and delete the old searchSuggest system
Open		None	T177251 Dead keys prevent autocomplete in search box
Resolved		ovasileva	T244392 [GOAL] Deploy the new Vue.js search experience
Resolved		ovasileva	T261636 [EPIC] Determine changes in search usage during desktop improvements project
Resolved		ovasileva	T267011 Analyze results of A/B test for new search location
Resolved		ovasileva	T263032 Deploy the new location of the search bar to new vector and begin A/B test on test wikis
Resolved		ovasileva	T275200 Analyze results of A/B test for new search widget
Resolved		ovasileva	T249297 Deploy the new Vue.js search experience
Resolved		Edtadros	T259798 Deploy the new Vue.js search experience to the Beta-Cluster and Test Wikipedia
Declined		None	T257865 Merge feat/search to master for Vue.js search deployment
Resolved		Jdlrobson	T265333 End A/B test for search location
Duplicate		ovasileva	T261648 Perform A/B test for search location change
Resolved		MNeisler	T259250 A/B test setup for search changes
Resolved		• jlinehan	T255585 [EPIC] Extend client-side error logging coverage to include English Wikipedia
Resolved		Tgr	T248884 Documentation of client side error logging capabilities on mediawiki
Resolved		None	T226986 Client side error logging production launch
Resolved		• jlinehan	T229442 Create client side error schema
Resolved		• jlinehan	T235189 Develop small client-side error logger for MediaWiki
Resolved		Ottomata	T236386 Set up eventgate-logging-external in production
Resolved		Ottomata	T246030 Enable client side error logging in prod for small wiki
Resolved	Spike	Jdlrobson	T255204 [Spike, 1hr]: Analyse Catalan Wikipedia error logs for iOS error
Resolved		• jlinehan	T258073 Enable client error logging on ca.wikipedia.org (Catalan)
Open		None	T259711 Determine requirements for logging native app client errors to Logstash
Resolved		• jlinehan	T258099 Enable client error logging for Desktop Improvements Project and Vue.js search
Resolved		• jlinehan	T259371 Limit the number of mw-client-errors from a single client (suggested: maximum 5 errors should be recorded for a single IP on a single page)
Resolved		• jlinehan	T259383 Filter out non-wikimedia domains and extensions from web-client-errors
Resolved		• jlinehan	T259369 "Script error." Scripts loaded from other domains with empty file_uri and no stack trace should not be included
Resolved		Jdlrobson	T259385 Work with community to fix faulty gadgets triggering JS errors
Resolved		• jlinehan	T258406 Allow tagging of client error events on the client
Resolved		• jlinehan	T262627 Error: vs Uncaught Error: in client error messages
Resolved		Tgr	T256173 Allow filtering of errors from logged in users
Resolved		colewhite	T265938 Create a separate logstash ElasticSearch index for schemaed events
Declined		None	T263898 Filter out errors with file_url "undefined"
Resolved		None	T266054 If stack_trace field is empty log as null value to support filtering
Resolved		• jlinehan	T262626 Remove http.client_ip from EventGate default schema (again)
Resolved		Tgr	T265096 Tag errors where a banner is being displayed
Resolved	BUG REPORT	Tgr	T316992 ClientError fieldname 'tags' is conflicting
Open		None	T267602 Client-side error logging should use Elastic Common Schema (ECS) fields when possible
Open		None	T262566 Enable DWIM support for Vue.js search
Resolved		phuedx	T273090 Deploy modern Vector and new Vue.js search experience to new pilot wikis
Resolved		• alexhollender_WMF	T273085 Deploy more logos for new DIP pilot wikis
Resolved		ovasileva	T273477 Update Wikivoyage logos (Vector modern & mobile)
Duplicate		None	T264299 German Wikivoyage: select modern Vector skin as default
Resolved		• alexhollender_WMF	T252701 Spike: what to do with Wikivoyage icons in new design?
Resolved		MNeisler	T274869 Instrumentation QA for new search widget

Event Timeline

ovasileva created this task.Aug 31 2020, 2:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 31 2020, 2:45 PM

ovasileva triaged this task as Medium priority.Aug 31 2020, 2:45 PM

ovasileva added a subtask: T259250: A/B test setup for search changes.

ovasileva added a project: Design-Systems-team-20200324-20220422.

ovasileva added a parent task: T261636: [EPIC] Determine changes in search usage during desktop improvements project.

Analysis is blocked on T259250 but this looks relatively easy to estimate once that's complete. I'll move this to "upcoming" when the subtasks are completed.

Jdlrobson mentioned this in T261647: Set up A/B test for new search widget.Sep 1 2020, 10:24 PM

Jdlrobson mentioned this in T258116: Post deploy: Move search styles into layout.less (and other places) and remove temporary feature flags.Sep 1 2020, 10:39 PM

Blocked on QA for new fields and A/B test and roll out to office wiki T262207. Earliest date we can deploy is 14th September. Assigning to Olga to decide when this can become unblocked

ovasileva added a parent task: T263032: Deploy the new location of the search bar to new vector and begin A/B test on test wikis.Sep 16 2020, 3:11 PM

Jdlrobson closed this task as a duplicate of T263032: Deploy the new location of the search bar to new vector and begin A/B test on test wikis.Sep 17 2020, 1:14 AM

MNeisler closed subtask T259250: A/B test setup for search changes as Resolved.Oct 26 2020, 6:50 PM

ovasileva added a parent task: T265333: End A/B test for search location.Nov 2 2020, 12:16 PM

ovasileva mentioned this in T267011: Analyze results of A/B test for new search location.Nov 2 2020, 12:22 PM