Page MenuHomePhabricator

Stop RelatedArticles A/B test and clean up config
Closed, ResolvedPublic1 Story Points

Description

On or after the 19th July we should stop the RelatedArticles A/B test

Pre-signoff

  • Make a task to analyse the data.

Done in T169948#3553064.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 6 2017, 11:42 PM
ovasileva triaged this task as High priority.Jul 7 2017, 1:48 PM
ovasileva moved this task from Incoming to Upcoming on the Readers-Web-Backlog board.
Restricted Application added a subscriber: Dereckson. · View Herald TranscriptJul 7 2017, 1:48 PM
ovasileva set the point value for this task to 1.Jul 11 2017, 4:47 PM

Change 366314 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Stop RelatedArticles A/B test and clean up config

https://gerrit.wikimedia.org/r/366314

@phuedx not sure what happened during my swat window this afternoon but my patch was overlooked. If you or @pmiagza can seat tomorrow I'd be very grateful. A/b test is still running unnecessarily... Otherwise ill try and get the one before the train.

Change 366314 merged by jenkins-bot:
[operations/mediawiki-config@master] Stop RelatedArticles A/B test and clean up config

https://gerrit.wikimedia.org/r/366314

After I left office patch got merged and reverted https://gerrit.wikimedia.org/r/#/c/366490/

So patch needs to be resubmitted.

phuedx claimed this task.Jul 20 2017, 7:05 AM

I can get this deployed during the European Mid-day SWAT deployment.

Change 366546 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[operations/mediawiki-config@master] Revert "Revert "Stop RelatedArticles A/B test and clean up config""

https://gerrit.wikimedia.org/r/366546

phuedx updated the task description. (Show Details)Jul 20 2017, 11:54 AM
phuedx added a subscriber: ovasileva.

Change 366546 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Revert "Stop RelatedArticles A/B test and clean up config""

https://gerrit.wikimedia.org/r/366546

Mentioned in SAL (#wikimedia-operations) [2017-07-20T13:28:39Z] <zfilipin@tin> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:366546|Revert "Revert "Stop RelatedArticles A/B test and clean up config"" (T169948)]] (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-07-20T13:29:52Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:366546|Revert "Revert "Stop RelatedArticles A/B test and clean up config"" (T169948)]] (duration: 00m 46s)

@ovasileva, @Tbayer: The experiment ended between 13:30 and 13:35 PM UTC today – the additional 5 minutes comes from the ResourceLoader startup module cache. I confirmed that RelatedArticles is now configured to be enabled for everyone on all wikis.

phuedx removed phuedx as the assignee of this task.Jul 20 2017, 1:45 PM
phuedx updated the task description. (Show Details)

Thanks Sam! Olga will sign off.

phuedx added a comment.Aug 7 2017, 5:05 PM

@ovasileva, @Tbayer: Is there a task that tracks analyzing the data?

The A/B test was disabled on 20th July. It's now been off for a month. Our data has a short life span and I'm worried we'll lose our A/B test data i we don't collect it soon. @Tbayer @ovasileva can you give reassurance that we're on top of this?

The plan was that @ovasileva is going to take a look at the data first (basically resuming T157307, now that T167236 has been resolved), but I'm happy to lend a hand if needed.

The A/B test was disabled on 20th July. It's now been off for a month. Our data has a short life span and I'm worried we'll lose our A/B test data i we don't collect it soon.

That's not yet a limitation right now - see the schema talk page for the concrete purging settings.

Thanks @Tbayer - feel a lot better now (to quote[[ https://meta.wikimedia.org/wiki/Schema_talk:RelatedArticles | talk page ]] "Auto-purge just eventCapsule PII after 90 days, keep the rest indefinitely" (not sure what PII means but sounds like we don't purge). I'll wait until @ovasileva get's back and sync up with her.

Thanks @Tbayer - feel a lot better now (to quote[[ https://meta.wikimedia.org/wiki/Schema_talk:RelatedArticles | talk page ]] "Auto-purge just eventCapsule PII after 90 days, keep the rest indefinitely" (not sure what PII means but sounds like we don't purge).

PII = personally identifiable information, or in this case information that could potentially become PII in connection with certain other information; in practice this means the user agent field in this case.
Thanks for following up and making sure no balls are dropped ;) Opening a new, separate task for the analysis doesn't sound like a bad idea actually.

ovasileva closed this task as Resolved.Aug 25 2017, 2:11 PM

Confirmed that no events are logged after July 20

Confirmed that no events are logged after July 20

Hm, according to both Grafana and MariaDB (see below), events continue to be logged at an undiminished rate?
CC @JKatzWMF

SELECT DATE(timestamp) AS date, COUNT(*) AS events FROM log.RelatedArticles_16352530 
GROUP BY date ORDER BY date;
+------------+---------+
| date       | events  |
+------------+---------+
| 2017-03-29 |  277850 |
| 2017-03-30 |  807016 |
| 2017-03-31 |  826709 |
| 2017-04-01 |  830810 |
| 2017-04-02 |  860367 |
| 2017-04-03 |  763252 |
| 2017-04-04 |  903474 |
| 2017-04-05 |  817784 |
| 2017-04-06 |  841699 |
| 2017-04-07 |  879389 |
| 2017-04-08 |  932706 |
| 2017-04-09 |  990785 |
| 2017-04-10 |  891978 |
| 2017-04-11 |  903183 |
| 2017-04-12 |  842101 |
| 2017-04-13 |  856601 |
| 2017-04-14 |  979614 |
| 2017-04-15 | 1040926 |
| 2017-04-16 | 1107881 |
| 2017-04-17 |  991841 |
| 2017-04-18 | 1037726 |
| 2017-04-19 | 1066976 |
| 2017-04-20 |  999268 |
| 2017-04-21 |  984038 |
| 2017-04-22 |  969934 |
| 2017-04-23 | 1025040 |
| 2017-04-24 |  926605 |
| 2017-04-25 | 1031564 |
| 2017-04-26 | 1010311 |
| 2017-04-27 |  949630 |
| 2017-04-28 |  926666 |
| 2017-04-29 |  858763 |
| 2017-04-30 |  979675 |
| 2017-05-01 | 1027545 |
| 2017-05-02 |  909854 |
| 2017-05-03 |  947070 |
| 2017-05-04 |  865252 |
| 2017-05-05 |  796734 |
| 2017-05-06 |  798458 |
| 2017-05-07 |  845914 |
| 2017-05-08 |  760959 |
| 2017-05-09 |  773935 |
| 2017-05-10 |  746359 |
| 2017-05-11 |  765589 |
| 2017-05-12 |  832830 |
| 2017-05-13 |  911587 |
| 2017-05-14 |  902963 |
| 2017-05-15 |  924182 |
| 2017-05-16 |  893967 |
| 2017-05-17 |  833927 |
| 2017-05-18 |  809071 |
| 2017-05-19 |  789182 |
| 2017-05-20 |  814451 |
| 2017-05-21 |  852634 |
| 2017-05-22 |  777281 |
| 2017-05-23 |  785300 |
| 2017-05-24 |  733546 |
| 2017-05-25 |  755718 |
| 2017-05-26 |  695911 |
| 2017-05-27 |  733859 |
| 2017-05-28 |  794860 |
| 2017-05-29 |  914041 |
| 2017-05-30 |  908333 |
| 2017-05-31 |  908757 |
| 2017-06-01 |  868755 |
| 2017-06-02 |  843753 |
| 2017-06-03 |  897099 |
| 2017-06-04 |  937367 |
| 2017-06-05 |  874119 |
| 2017-06-06 |  886780 |
| 2017-06-07 |  806523 |
| 2017-06-08 |  794736 |
| 2017-06-09 |  855929 |
| 2017-06-10 |  851453 |
| 2017-06-11 |  848846 |
| 2017-06-12 |  887783 |
| 2017-06-13 |  893476 |
| 2017-06-14 |  826272 |
| 2017-06-15 |  820424 |
| 2017-06-16 |  852298 |
| 2017-06-17 |  859472 |
| 2017-06-18 |  891665 |
| 2017-06-19 |  797843 |
| 2017-06-20 |  767803 |
| 2017-06-21 |  757477 |
| 2017-06-22 |  716011 |
| 2017-06-23 |  716214 |
| 2017-06-24 |  705568 |
| 2017-06-25 |  731509 |
| 2017-06-26 |  783580 |
| 2017-06-27 |  870949 |
| 2017-06-28 |  859120 |
| 2017-06-29 |  833724 |
| 2017-06-30 |  878673 |
| 2017-07-01 |  894863 |
| 2017-07-02 |  851422 |
| 2017-07-03 |  748204 |
| 2017-07-04 |  782965 |
| 2017-07-05 |  756870 |
| 2017-07-06 |  821818 |
| 2017-07-07 |  755865 |
| 2017-07-08 |  801973 |
| 2017-07-09 |  842213 |
| 2017-07-10 |  774710 |
| 2017-07-11 |  843787 |
| 2017-07-12 |  798254 |
| 2017-07-13 |  738847 |
| 2017-07-14 |  711391 |
| 2017-07-15 |  747813 |
| 2017-07-16 |  767395 |
| 2017-07-17 |  713098 |
| 2017-07-18 |  888753 |
| 2017-07-19 |  839894 |
| 2017-07-20 |  829805 |
| 2017-07-21 |  873485 |
| 2017-07-22 |  907193 |
| 2017-07-23 |  910027 |
| 2017-07-24 |  757270 |
| 2017-07-25 |  720870 |
| 2017-07-26 |  744583 |
| 2017-07-27 |  781297 |
| 2017-07-28 |  848245 |
| 2017-07-29 |  811305 |
| 2017-07-30 |  828631 |
| 2017-07-31 |  809280 |
| 2017-08-01 |  806256 |
| 2017-08-02 |  785140 |
| 2017-08-03 |  779198 |
| 2017-08-04 |  843091 |
| 2017-08-05 |  804436 |
| 2017-08-06 |  819165 |
| 2017-08-07 |  722911 |
| 2017-08-08 |  756626 |
| 2017-08-09 |  747333 |
| 2017-08-10 |  763990 |
| 2017-08-11 |  871977 |
| 2017-08-12 |  834217 |
| 2017-08-13 |  856194 |
| 2017-08-14 |  850431 |
| 2017-08-15 |  979318 |
| 2017-08-16 |  882610 |
| 2017-08-17 |  887683 |
| 2017-08-18 |  904212 |
| 2017-08-19 |  937238 |
| 2017-08-20 |  935341 |
| 2017-08-21 |  875662 |
| 2017-08-22 |  907010 |
| 2017-08-23 |  854925 |
| 2017-08-24 |  841916 |
| 2017-08-25 |  880753 |
| 2017-08-26 |  838995 |
| 2017-08-27 |  864356 |
| 2017-08-28 |  907787 |
| 2017-08-29 |  915787 |
| 2017-08-30 |  816086 |
| 2017-08-31 |  629631 |
+------------+---------+
156 rows in set (1 min 27.83 sec)

@Tbayer - sorry for the confusion, I was referring to disabled events. The schema should still be running, now with 100% enabled, so we're not expecting the rate of events to diminish.

SELECT count(*) AS num_Events, event_eventName, 
LEFT(TIMESTAMP, 8) AS yyyymmdd 
FROM `RelatedArticles_16352530`
WHERE event_eventName = 'feature-disabled'
GROUP BY yyyymmdd, event_eventName;

Gives me 1-2 disabled events per day since 7/23

3025	feature-disabled	20170714
3394	feature-disabled	20170715
3426	feature-disabled	20170716
2923	feature-disabled	20170717
3481	feature-disabled	20170718
3324	feature-disabled	20170719
1724	feature-disabled	20170720
14	feature-disabled	20170721
11	feature-disabled	20170722
3	feature-disabled	20170723
5	feature-disabled	20170724
3	feature-disabled	20170725
2	feature-disabled	20170726
1	feature-disabled	20170727
2	feature-disabled	20170728
12	feature-disabled	20170729
2	feature-disabled	20170730
1	feature-disabled	20170731
5	feature-disabled	20170801
1	feature-disabled	20170802
2	feature-disabled	20170809
1	feature-disabled	20170810
2	feature-disabled	20170811
1	feature-disabled	20170818
2	feature-disabled	20170820
1	feature-disabled	20170821
1	feature-disabled	20170822

The schema should still be running, now with 100% enabled, so we're not expecting the rate of events to diminish.

Wait... what?!!? cc @phuedx
I see $wgRelatedArticlesLoggingBucketSize = 0.01;
Shouldn't this be zero now? Are we not done with instrumentation now we have data from an A/B test?
This task is about analysing the A/B test data... not new data.

phuedx added a comment.Sep 4 2017, 2:15 PM

The schema should still be running, now with 100% enabled, so we're not expecting the rate of events to diminish.

Wait... what?!!? cc @phuedx
I see $wgRelatedArticlesLoggingBucketSize = 0.01;
Shouldn't this be zero now? Are we not done with instrumentation now we have data from an A/B test?

I believe we wanted to leave the instrumentation on while the A/B test was wound down as one last test of the instrumentation itself, i.e. that ~0% of events were feature-disabled events.

I've confirmed with @ovasileva that $wgRelatedArticlesLoggingBucketSize should now be set to 0 everywhere.

This task is about analysing the A/B test data... not new data.

This task was about disabling the A/B test. The analysis of the data collected during the test is tracked in T157307: Analyze performance of related pages feature.

phuedx updated the task description. (Show Details)Sep 4 2017, 2:16 PM