Page MenuHomePhabricator
Create Task
Maniphest T390849

[Epic] SDS 2.4.16 Establish Automated Analytics for Experiments Conducted With Experimentation Lab
Closed, ResolvedPublic
Actions

Description

From SDS2.4.16 Automated Analytics Hyp PRD:

If we create a Superset dashboard to report experiment results based on interaction (non-fundraising) metrics from the measurement plan (SDS 2.4.14), the product team will have fast and ready access to initial insights as soon as data becomes available in our Data Lake – and full insights shortly after the A/B test has concluded – without depending on a data specialist (product analyst).

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedmpopovT390849 [Epic] SDS 2.4.16 Establish Automated Analytics for Experiments Conducted With Experimentation Lab
 ResolvedmpopovT390851 Sketch of architecture for automated experiment analytics MVP
 DuplicatecjmingT391092 xLab experiments API should include recently concluded (<24h) experiments
 ResolvedcjmingT391957 xLab API: generate enrollment and bucketing for Varnish authority
 ResolvedmpopovT391267 [Automated Analytics] Analysis modules
 ResolvedmpopovT391286 [Automated Analytics] Skeleton Python job repo and Airflow DAG
 ResolvedmpopovT391287 [Automated Analytics] Mechanisms for defining metrics and registering experiments for analysis
 DeclinedmpopovT391683 [Automated Analytics][Stretch Goal] Visualize uncertainty in experiment results reporting
 Resolved• StevemuneneT391692 [Spike] Investigate option for Superset to access internally-hosted images
 ResolvedmpopovT392220 Create database `wmf_experiments`
 ResolvedmpopovT392301 [Automated Analytics] Implement experiment analysis in Python job repo
 ResolvedmpopovT392312 [Automated Analytics] User authentication status segmentation in analysis/results
 ResolvedMilimetricT392315 [Epic] SDS 2.4.8 Eventgate updates
 ResolvedMilimetricT392314 [Epic] SDS 2.4.9 Synthetic Beacon Endpoints
 ResolvedKOforiT391411 SDS 2.4.4 Edge Uniques Production Cookie Deployment
 ResolvedBUG REPORTBBlackT391633 Replace ableist usage of `sane` and `insane` in libvmod-wmfuniq codebase
ResolvedBCornwallT392059 Provide debian packages for libvmod-wmfuniq and wmfuniq-keygen
 ResolvedBCornwallT392533 Update libvmod-netmapper to 1.10
 ResolvedBTullisT394437 Include all CDN SANs on eventgate-analytics-external.discovery.wmnet:4692 TLS certificate
 ResolvedBBlackT394862 libvmod_wmfuniq: add stats counter for cookie values of incorrect length
 ResolvedVgutierrezT395001 Fetch experiments configuration programmatically from CDN servers
 ResolvedphuedxT391988 FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: Route experiment-oriented MediaWiki JavaScript-based events conditionally
 ResolvedtchinT391959 FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2
 DeclinedNoneT392529 Cannot save query with Unicode characters as virtual dataset in Superset
 ResolvedmpopovT393262 [Automated Analytics] Fix problem with MERGE INTO query
 ResolvedmpopovT393338 [Automated Analytics] Gracefully handle experiments without data during analysis
 ResolvedmpopovT393776 [Automated Analytics] Gracefully handle no experiments to analyze
 ResolvedmpopovT393777 [Automated Analytics] Airflow DAG for configuration loading
 ResolvedmpopovT393778 [Automated Analytics] Airflow DAG for analyzing experiments
 ResolvedmpopovT393906 [Automated Analytics] Switch experiments config from mockup to xLab API
 ResolvedmpopovT396024 [Automated Analytics] Clean up example/demo experiments and metrics

Event Timeline

mpopov created this task.Apr 2 2025, 12:05 PM
mpopov triaged this task as Medium priority.
mpopov moved this task from Ready for Development to Sprint Goals & Active Epics on the Test Kitchen (Experiment Platform Sprint 4) board.
mpopov moved this task from Triage to Epics on the Product-Analytics board.
dr0ptp4kt subscribed.Apr 2 2025, 7:39 PM
Milimetric edited projects, added Test Kitchen (Experiment Platform Sprint 5); removed Test Kitchen (Experiment Platform Sprint 4).Apr 14 2025, 1:29 PM
Milimetric moved this task from Ready for Development to Sprint Goals & Active Epics on the Test Kitchen (Experiment Platform Sprint 5) board.
• VirginiaPoundstone closed subtask T391267: [Automated Analytics] Analysis modules as Resolved.Apr 14 2025, 7:33 PM
• VirginiaPoundstone closed subtask T390851: Sketch of architecture for automated experiment analytics MVP as Resolved.Apr 14 2025, 8:22 PM
Milimetric renamed this task from [Epic] Establish Automated Analytics for Experiments Conducted With Experimentation Lab to [Epic] SDS 2.4.16 Establish Automated Analytics for Experiments Conducted With Experimentation Lab.Apr 18 2025, 8:22 PM
Milimetric added a parent task: T392313: [Epic] SDS 2.4.11 Run a Synthetic A/A Experiment.
Milimetric removed a parent task: T392313: [Epic] SDS 2.4.11 Run a Synthetic A/A Experiment.
Milimetric added a subtask: T392315: [Epic] SDS 2.4.8 Eventgate updates.Apr 18 2025, 8:34 PM
Milimetric closed subtask T392315: [Epic] SDS 2.4.8 Eventgate updates as Resolved.Apr 22 2025, 2:31 PM
mpopov added a subtask: T392529: Cannot save query with Unicode characters as virtual dataset in Superset.Apr 23 2025, 7:44 PM
Milimetric edited projects, added Test Kitchen (Experiment Platform Sprint 6); removed Test Kitchen (Experiment Platform Sprint 5).May 1 2025, 3:56 PM
Milimetric moved this task from Ready for Development to Sprint Goals & Active Epics on the Test Kitchen (Experiment Platform Sprint 6) board.
mpopov added a comment.May 3 2025, 1:02 PM

Nice-to-have emerged: T393265: [Automated Analytics] Enable overriding default Bayesian prior on lift

• VirginiaPoundstone closed subtask T392301: [Automated Analytics] Implement experiment analysis in Python job repo as Resolved.May 4 2025, 1:55 PM
• VirginiaPoundstone closed subtask T392220: Create database `wmf_experiments` as Resolved.
• VirginiaPoundstone closed subtask T391286: [Automated Analytics] Skeleton Python job repo and Airflow DAG as Resolved.
Milimetric closed subtask T392312: [Automated Analytics] User authentication status segmentation in analysis/results as Resolved.May 5 2025, 6:23 PM
Milimetric closed subtask T391287: [Automated Analytics] Mechanisms for defining metrics and registering experiments for analysis as Resolved.
Milimetric closed subtask T393262: [Automated Analytics] Fix problem with MERGE INTO query as Resolved.May 7 2025, 3:05 PM
mpopov closed subtask T391683: [Automated Analytics][Stretch Goal] Visualize uncertainty in experiment results reporting as Declined.May 8 2025, 11:00 PM
mpopov closed subtask T392529: Cannot save query with Unicode characters as virtual dataset in Superset as Declined.May 8 2025, 11:04 PM
Milimetric closed subtask T393338: [Automated Analytics] Gracefully handle experiments without data during analysis as Resolved.May 12 2025, 3:06 PM
Milimetric closed subtask T393776: [Automated Analytics] Gracefully handle no experiments to analyze as Resolved.
Milimetric closed subtask T393778: [Automated Analytics] Airflow DAG for analyzing experiments as Resolved.May 15 2025, 3:10 PM
Milimetric closed subtask T393777: [Automated Analytics] Airflow DAG for configuration loading as Resolved.
mpopov closed subtask T393906: [Automated Analytics] Switch experiments config from mockup to xLab API as Resolved.May 19 2025, 3:08 PM
Milimetric edited projects, added Test Kitchen (Experiment Platform Sprint 7); removed Test Kitchen (Experiment Platform Sprint 6).May 27 2025, 12:09 PM
Milimetric moved this task from Ready for Development to Sprint Goals & Active Epics on the Test Kitchen (Experiment Platform Sprint 7) board.
JVanderhoop-WMF mentioned this in T386453: Update documentation to include how to create superset dashboard/jupyter notebook to work with Hive data.May 28 2025, 9:01 PM
mpopov added a subtask: T396024: [Automated Analytics] Clean up example/demo experiments and metrics.Jun 4 2025, 1:40 PM
mpopov closed this task as Resolved.Jun 5 2025, 8:27 PM
mpopov moved this task from Sprint Goals & Active Epics to Done on the Test Kitchen (Experiment Platform Sprint 7) board.
mpopov closed subtask T396024: [Automated Analytics] Clean up example/demo experiments and metrics as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits