On or after the 19th July we should stop the RelatedArticles A/B test
Done in T169948#3553064.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | pmiazga | T174944 Disable RelatedArticles instrumentation on all wikis | |||
| Resolved | ovasileva | T169948 Stop RelatedArticles A/B test and clean up config |
Change 366314 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Stop RelatedArticles A/B test and clean up config
@phuedx not sure what happened during my swat window this afternoon but my patch was overlooked. If you or @pmiagza can seat tomorrow I'd be very grateful. A/b test is still running unnecessarily... Otherwise ill try and get the one before the train.
Change 366314 merged by jenkins-bot:
[operations/mediawiki-config@master] Stop RelatedArticles A/B test and clean up config
After I left office patch got merged and reverted https://gerrit.wikimedia.org/r/#/c/366490/
So patch needs to be resubmitted.
Change 366546 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[operations/mediawiki-config@master] Revert "Revert "Stop RelatedArticles A/B test and clean up config""
I've scheduled rOMWC48ee5039b643: Revert "Revert "Stop RelatedArticles A/B test and clean up config"" for deployment during today's European Mid-day SWAT.
Change 366546 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Revert "Stop RelatedArticles A/B test and clean up config""
Mentioned in SAL (#wikimedia-operations) [2017-07-20T13:28:39Z] <zfilipin@tin> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:366546|Revert "Revert "Stop RelatedArticles A/B test and clean up config"" (T169948)]] (duration: 00m 47s)
Mentioned in SAL (#wikimedia-operations) [2017-07-20T13:29:52Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:366546|Revert "Revert "Stop RelatedArticles A/B test and clean up config"" (T169948)]] (duration: 00m 46s)
@ovasileva, @Tbayer: The experiment ended between 13:30 and 13:35 PM UTC today – the additional 5 minutes comes from the ResourceLoader startup module cache. I confirmed that RelatedArticles is now configured to be enabled for everyone on all wikis.
The A/B test was disabled on 20th July. It's now been off for a month. Our data has a short life span and I'm worried we'll lose our A/B test data i we don't collect it soon. @Tbayer @ovasileva can you give reassurance that we're on top of this?
The plan was that @ovasileva is going to take a look at the data first (basically resuming T157307, now that T167236 has been resolved), but I'm happy to lend a hand if needed.
That's not yet a limitation right now - see the schema talk page for the concrete purging settings.
Thanks @Tbayer - feel a lot better now (to quote[[ https://meta.wikimedia.org/wiki/Schema_talk:RelatedArticles | talk page ]] "Auto-purge just eventCapsule PII after 90 days, keep the rest indefinitely" (not sure what PII means but sounds like we don't purge). I'll wait until @ovasileva get's back and sync up with her.
PII = personally identifiable information, or in this case information that could potentially become PII in connection with certain other information; in practice this means the user agent field in this case.
Thanks for following up and making sure no balls are dropped ;) Opening a new, separate task for the analysis doesn't sound like a bad idea actually.
re-opening T157307: Analyze performance of related pages feature as continuation of analysis
Hm, according to both Grafana and MariaDB (see below), events continue to be logged at an undiminished rate?
CC @JKatzWMF
SELECT DATE(timestamp) AS date, COUNT(*) AS events FROM log.RelatedArticles_16352530 GROUP BY date ORDER BY date; +------------+---------+ | date | events | +------------+---------+ | 2017-03-29 | 277850 | | 2017-03-30 | 807016 | | 2017-03-31 | 826709 | | 2017-04-01 | 830810 | | 2017-04-02 | 860367 | | 2017-04-03 | 763252 | | 2017-04-04 | 903474 | | 2017-04-05 | 817784 | | 2017-04-06 | 841699 | | 2017-04-07 | 879389 | | 2017-04-08 | 932706 | | 2017-04-09 | 990785 | | 2017-04-10 | 891978 | | 2017-04-11 | 903183 | | 2017-04-12 | 842101 | | 2017-04-13 | 856601 | | 2017-04-14 | 979614 | | 2017-04-15 | 1040926 | | 2017-04-16 | 1107881 | | 2017-04-17 | 991841 | | 2017-04-18 | 1037726 | | 2017-04-19 | 1066976 | | 2017-04-20 | 999268 | | 2017-04-21 | 984038 | | 2017-04-22 | 969934 | | 2017-04-23 | 1025040 | | 2017-04-24 | 926605 | | 2017-04-25 | 1031564 | | 2017-04-26 | 1010311 | | 2017-04-27 | 949630 | | 2017-04-28 | 926666 | | 2017-04-29 | 858763 | | 2017-04-30 | 979675 | | 2017-05-01 | 1027545 | | 2017-05-02 | 909854 | | 2017-05-03 | 947070 | | 2017-05-04 | 865252 | | 2017-05-05 | 796734 | | 2017-05-06 | 798458 | | 2017-05-07 | 845914 | | 2017-05-08 | 760959 | | 2017-05-09 | 773935 | | 2017-05-10 | 746359 | | 2017-05-11 | 765589 | | 2017-05-12 | 832830 | | 2017-05-13 | 911587 | | 2017-05-14 | 902963 | | 2017-05-15 | 924182 | | 2017-05-16 | 893967 | | 2017-05-17 | 833927 | | 2017-05-18 | 809071 | | 2017-05-19 | 789182 | | 2017-05-20 | 814451 | | 2017-05-21 | 852634 | | 2017-05-22 | 777281 | | 2017-05-23 | 785300 | | 2017-05-24 | 733546 | | 2017-05-25 | 755718 | | 2017-05-26 | 695911 | | 2017-05-27 | 733859 | | 2017-05-28 | 794860 | | 2017-05-29 | 914041 | | 2017-05-30 | 908333 | | 2017-05-31 | 908757 | | 2017-06-01 | 868755 | | 2017-06-02 | 843753 | | 2017-06-03 | 897099 | | 2017-06-04 | 937367 | | 2017-06-05 | 874119 | | 2017-06-06 | 886780 | | 2017-06-07 | 806523 | | 2017-06-08 | 794736 | | 2017-06-09 | 855929 | | 2017-06-10 | 851453 | | 2017-06-11 | 848846 | | 2017-06-12 | 887783 | | 2017-06-13 | 893476 | | 2017-06-14 | 826272 | | 2017-06-15 | 820424 | | 2017-06-16 | 852298 | | 2017-06-17 | 859472 | | 2017-06-18 | 891665 | | 2017-06-19 | 797843 | | 2017-06-20 | 767803 | | 2017-06-21 | 757477 | | 2017-06-22 | 716011 | | 2017-06-23 | 716214 | | 2017-06-24 | 705568 | | 2017-06-25 | 731509 | | 2017-06-26 | 783580 | | 2017-06-27 | 870949 | | 2017-06-28 | 859120 | | 2017-06-29 | 833724 | | 2017-06-30 | 878673 | | 2017-07-01 | 894863 | | 2017-07-02 | 851422 | | 2017-07-03 | 748204 | | 2017-07-04 | 782965 | | 2017-07-05 | 756870 | | 2017-07-06 | 821818 | | 2017-07-07 | 755865 | | 2017-07-08 | 801973 | | 2017-07-09 | 842213 | | 2017-07-10 | 774710 | | 2017-07-11 | 843787 | | 2017-07-12 | 798254 | | 2017-07-13 | 738847 | | 2017-07-14 | 711391 | | 2017-07-15 | 747813 | | 2017-07-16 | 767395 | | 2017-07-17 | 713098 | | 2017-07-18 | 888753 | | 2017-07-19 | 839894 | | 2017-07-20 | 829805 | | 2017-07-21 | 873485 | | 2017-07-22 | 907193 | | 2017-07-23 | 910027 | | 2017-07-24 | 757270 | | 2017-07-25 | 720870 | | 2017-07-26 | 744583 | | 2017-07-27 | 781297 | | 2017-07-28 | 848245 | | 2017-07-29 | 811305 | | 2017-07-30 | 828631 | | 2017-07-31 | 809280 | | 2017-08-01 | 806256 | | 2017-08-02 | 785140 | | 2017-08-03 | 779198 | | 2017-08-04 | 843091 | | 2017-08-05 | 804436 | | 2017-08-06 | 819165 | | 2017-08-07 | 722911 | | 2017-08-08 | 756626 | | 2017-08-09 | 747333 | | 2017-08-10 | 763990 | | 2017-08-11 | 871977 | | 2017-08-12 | 834217 | | 2017-08-13 | 856194 | | 2017-08-14 | 850431 | | 2017-08-15 | 979318 | | 2017-08-16 | 882610 | | 2017-08-17 | 887683 | | 2017-08-18 | 904212 | | 2017-08-19 | 937238 | | 2017-08-20 | 935341 | | 2017-08-21 | 875662 | | 2017-08-22 | 907010 | | 2017-08-23 | 854925 | | 2017-08-24 | 841916 | | 2017-08-25 | 880753 | | 2017-08-26 | 838995 | | 2017-08-27 | 864356 | | 2017-08-28 | 907787 | | 2017-08-29 | 915787 | | 2017-08-30 | 816086 | | 2017-08-31 | 629631 | +------------+---------+ 156 rows in set (1 min 27.83 sec)
@Tbayer - sorry for the confusion, I was referring to disabled events. The schema should still be running, now with 100% enabled, so we're not expecting the rate of events to diminish.
SELECT count(*) AS num_Events, event_eventName, LEFT(TIMESTAMP, 8) AS yyyymmdd FROM `RelatedArticles_16352530` WHERE event_eventName = 'feature-disabled' GROUP BY yyyymmdd, event_eventName;
Gives me 1-2 disabled events per day since 7/23
3025 feature-disabled 20170714 3394 feature-disabled 20170715 3426 feature-disabled 20170716 2923 feature-disabled 20170717 3481 feature-disabled 20170718 3324 feature-disabled 20170719 1724 feature-disabled 20170720 14 feature-disabled 20170721 11 feature-disabled 20170722 3 feature-disabled 20170723 5 feature-disabled 20170724 3 feature-disabled 20170725 2 feature-disabled 20170726 1 feature-disabled 20170727 2 feature-disabled 20170728 12 feature-disabled 20170729 2 feature-disabled 20170730 1 feature-disabled 20170731 5 feature-disabled 20170801 1 feature-disabled 20170802 2 feature-disabled 20170809 1 feature-disabled 20170810 2 feature-disabled 20170811 1 feature-disabled 20170818 2 feature-disabled 20170820 1 feature-disabled 20170821 1 feature-disabled 20170822
The schema should still be running, now with 100% enabled, so we're not expecting the rate of events to diminish.
Wait... what?!!? cc @phuedx
I see $wgRelatedArticlesLoggingBucketSize = 0.01;
Shouldn't this be zero now? Are we not done with instrumentation now we have data from an A/B test?
This task is about analysing the A/B test data... not new data.
I believe we wanted to leave the instrumentation on while the A/B test was wound down as one last test of the instrumentation itself, i.e. that ~0% of events were feature-disabled events.
I've confirmed with @ovasileva that $wgRelatedArticlesLoggingBucketSize should now be set to 0 everywhere.
This task is about analysing the A/B test data... not new data.
This task was about disabling the A/B test. The analysis of the data collected during the test is tracked in T157307: Analyze performance of related pages feature.