Review remaining growthexperiments-homepage-variant rows at enwiki
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Urbanecm_WMF
	Wed, Nov 6, 11:52 AM

Description

As part of reviewing T375753: Drop unnecessary growthexperiments-homepage-variant entries from user_properties at Wikimedia wikis, I noticed enwiki has an unusual number of remaining growthexperiments-homepage-variant rows in user_properties:

mysql:research@dbstore1008.eqiad.wmnet [enwiki]> select count(*), up_value from user_properties where up_property='growthexperiments-homepage-variant' group by up_value;
+----------+-----------+
| count(*) | up_value  |
+----------+-----------+
|  1226972 | control   |
|        1 | oldimpact |
+----------+-----------+
2 rows in set (14.031 sec)

mysql:research@dbstore1008.eqiad.wmnet [enwiki]>

This doesn't seem correct, we'd expect few dozens (up to few hundreds) of remaining rows max. We should review what happened and possibly run userOptions.php to drop the remaining million of rows as well.

Related Objects

Mentioned In: T341553: Allow running one-off scripts manually
T377631: Add a link (Structured task): Release to a subset of newcomers on English Wikipedia
T375753: Drop unnecessary growthexperiments-homepage-variant entries from user_properties at Wikimedia wikis
Mentioned Here: P71213 Masterwork From Distant Lands
P71214 (An Untitled Masterwork)
T377631: Add a link (Structured task): Release to a subset of newcomers on English Wikipedia
T374544: Use "control" as the static default for growthexperiments-homepage-variant
T375753: Drop unnecessary growthexperiments-homepage-variant entries from user_properties at Wikimedia wikis

Event Timeline

Urbanecm_WMF created this task.Wed, Nov 6, 11:52 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Nov 6, 11:52 AM

Follow-up to recently prioritized work => moving to Up Next.

KStoller-WMF triaged this task as Medium priority.Wed, Nov 6, 5:16 PM

Sgs subscribed.Mon, Nov 11, 12:14 PM

I checked enwiki and a couple more (eswiki, dewiki). The results were the expected for eswiki and dewiki where there are some hundreds of rows caused by the incorrect backport of 1084181 to wmf/1.44.0-wmf.1 and enabling the experiment config in 1081104. That created "control" rows for 50% of new created accounts within October 30 and October 31.

For the enwiki rows, I checked and the "control" variant is present in older accounts up to August 23 2023. Also the control set as default change (1072548 ) reached enwiki on Oct 3 in wmf/1.43.0-wmf.25 and I verified there are no users with the variant after that date (aside from the Oct 30-31 issue). I cannot explain why these records would have survived the userOptions.php run from T374544#10242055.

Unless there are any objections, I'd run foreachwiki userOptions.php --delete-defaults growthexperiments-homepage-variant again and see what's the outcome. If we still see records, we should look closer at what's userOptions getting as default for those.

Pulling this in, as it caused a bug in T377631.

Before touching any rows, I checked the current status:

[urbanecm@stat1008 ~]$ analytics-mysql enwiki
[...]

mysql:research@dbstore1008.eqiad.wmnet [enwiki]> select count(*), up_value from user_properties where up_property='growthexperiments-homepage-variant' group by up_value;
+----------+-----------+
| count(*) | up_value  |
+----------+-----------+
|  1190038 | control   |
|        1 | oldimpact |
+----------+-----------+
2 rows in set (14.295 sec)

mysql:research@dbstore1008.eqiad.wmnet [enwiki]> ^DBye
[urbanecm@stat1008 ~]$ echo "select * from user_properties where up_property='growthexperiments-homepage-variant'" | analytics-mysql enwiki | gzip > backup-user-properties-enwiki-T379146.txt.gz
[urbanecm@stat1008 ~]$

backup-user-properties-enwiki-T379146.txt.gz is available as F57749998 (visibility restricted to WMF-NDA).

Mentioned in SAL (#wikimedia-operations) [2024-11-26T16:40:51Z] <urbanecm> mwscript-k8s -f userOptions.php -- --wiki=enwiki --old=control --delete 'growthexperiments-homepage-variant' # T379146, T377631

Stashbot mentioned this in T377631: Add a link (Structured task): Release to a subset of newcomers on English Wikipedia.Tue, Nov 26, 4:40 PM

Okay, enwiki now has no rows:

mysql:research@dbstore1008.eqiad.wmnet [enwiki]> select count(*), up_value from user_properties where up_property='growthexperiments-homepage-variant' group by up_value;
Empty set (0.002 sec)

mysql:research@dbstore1008.eqiad.wmnet [enwiki]>

Mentioned in SAL (#wikimedia-operations) [2024-11-26T19:26:44Z] <urbanecm> mwscript-k8s -f userOptions.php -- --wiki=enwiki --old=oldimpact --delete 'growthexperiments-homepage-variant' # T379146

Mentioned in SAL (#wikimedia-operations) [2024-11-26T19:27:49Z] <urbanecm> [urbanecm@mwmaint2002 ~]$ foreachwiki userOptions.php --delete-defaults growthexperiments-homepage-variant # T379146, logging to /home/urbanecm/T379146.log

In T379146#10311986, @Sgs wrote:

Unless there are any objections, I'd run foreachwiki userOptions.php --delete-defaults growthexperiments-homepage-variant again and see what's the outcome. If we still see records, we should look closer at what's userOptions getting as default for those.

Since there were no objections here, I started running this script.

Script finished: transcript is at P71213.

Remaining rows (across all wikis):

wiki	up_value	COUNT(*)
arwiki	community-updates-module	3
arwiki	control	132
cswiki	community-updates-module	4
cswiki	control	15
elwiki	newimpact	1
eswiki	community-updates-module	2
eswiki	control	112
frwiki	community-updates-module	6
frwiki	control	108
frwiktionary	B	1
test2wiki	community-updates-module	2
testwiki	community-updates-module	3
testwiki	control	3
testwiki	imagerecommendation	1

(generated via P71214 at stat1008)

I see the following group of variants defined:

Ongoing experiment (community-updates-module): this is OK, and in low enough volumes to be considered testing/QA
Old variant names (B, imagerecommendation): this is likely OK, deletion in T375753 was manual and we might've missed variants: the variants are low
Invalid variant names (newimpact): The newimpact variant never existed, this is probably "what will happen if I do X" kind of testing (and it's a single row anyway)
control: This is surprising, shouldn't have survived the first round of --delete-defaults. However, those rows are only present on wikis where community-updates-module is ongoing, so I presume this is just the equivalent of T377631#10356043 for community-updates-module.

We can easily drop the invalid/old variant names. That would leave us with control rows. We can probably safely delete them as well, as that would just result in enabling a feature the users are supposed to have (rather than taking a feature away). Not having those rows will also help us to avoid running into similar issues for future deployments.

Deleted:

[urbanecm@deploy2002 ~]$ for WIKI in arwiki cswiki eswiki frwiki; do mwscript-k8s -- userOptions.php --wiki=$WIKI --delete --old=control growthexperiments-homepage-variant; done
⏳ Starting userOptions.php on Kubernetes as job mw-script.codfw.4mbi5rkn ...
🚀 Job is running. For streaming logs, run:
K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.4mbi5rkn mediawiki-4mbi5rkn-app
⏳ Starting userOptions.php on Kubernetes as job mw-script.codfw.89twb9q6 ...
🚀 Job is running. For streaming logs, run:
K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.89twb9q6 mediawiki-89twb9q6-app
⏳ Starting userOptions.php on Kubernetes as job mw-script.codfw.c1p1p0b0 ...
🚀 Job is running. For streaming logs, run:
K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.c1p1p0b0 mediawiki-c1p1p0b0-app
⏳ Starting userOptions.php on Kubernetes as job mw-script.codfw.25l24055 ...
🚀 Job is running. For streaming logs, run:
K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.25l24055 mediawiki-25l24055-app
[urbanecm@deploy2002 ~]$ K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.4mbi5rkn mediawiki-4mbi5rkn-app
The script is about to delete 'growthexperiments-homepage-variant' option for ALL USERS from user_properties table.
This action is IRREVERSIBLE.

Abort with control-c in the next five seconds....0
Done! Deleted 132 rows.
[urbanecm@deploy2002 ~]$ K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.89twb9q6 mediawiki-89twb9q6-app
The script is about to delete 'growthexperiments-homepage-variant' option for ALL USERS from user_properties table.
This action is IRREVERSIBLE.

Abort with control-c in the next five seconds....0
Done! Deleted 15 rows.
[urbanecm@deploy2002 ~]$ K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.c1p1p0b0 mediawiki-c1p1p0b0-app
The script is about to delete 'growthexperiments-homepage-variant' option for ALL USERS from user_properties table.
This action is IRREVERSIBLE.

Abort with control-c in the next five seconds....0
Done! Deleted 112 rows.
[urbanecm@deploy2002 ~]$ K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-codfw.config kubectl logs -f job/mw-script.codfw.25l24055 mediawiki-25l24055-app
The script is about to delete 'growthexperiments-homepage-variant' option for ALL USERS from user_properties table.
This action is IRREVERSIBLE.

Abort with control-c in the next five seconds....0
Done! Deleted 108 rows.
[urbanecm@deploy2002 ~]$ mwscript-k8s -f -- userOptions.php --wiki=elwiki --delete --old=newimpact growthexperiments-homepage-variant
⏳ Starting userOptions.php on Kubernetes as job mw-script.codfw.0gisvqdt ...
⏳ Waiting for the container to start...
🚀 Job is running.
📜 Streaming logs:
The script is about to delete 'growthexperiments-homepage-variant' option for ALL USERS from user_properties table.
This action is IRREVERSIBLE.

Abort with control-c in the next five seconds....0
Done! Deleted 1 rows.
[urbanecm@deploy2002 ~]$ mwscript-k8s -f -- userOptions.php --wiki=frwiktionary --delete --old=B growthexperiments-homepage-variant
⏳ Starting userOptions.php on Kubernetes as job mw-script.codfw.m6cy2k6v ...
⏳ Waiting for the container to start...
🚀 Job is running.
📜 Streaming logs:
The script is about to delete 'growthexperiments-homepage-variant' option for ALL USERS from user_properties table.
This action is IRREVERSIBLE.

Abort with control-c in the next five seconds....0
Done! Deleted 1 rows.
[urbanecm@deploy2002 ~]$

After the deletions:

wiki	up_value	COUNT(*)
arwiki	community-updates-module	3
cswiki	community-updates-module	4
enwiki	control	1
eswiki	community-updates-module	2
frwiki	community-updates-module	6
test2wiki	community-updates-module	2
testwiki	community-updates-module	3
testwiki	control	3

All done it seems.

Urbanecm_WMF closed this task as Resolved.Wed, Nov 27, 4:37 PM

	Restricted File
	Tue, Nov 26, 4:20 PM

Review remaining growthexperiments-homepage-variant rows at enwikiClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Review remaining growthexperiments-homepage-variant rows at enwiki
Closed, ResolvedPublic
Actions