Changing the priority of this task to medium, I don't think this is an UBN task since we're probably working on this slowly over this quarter and next.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Today
Mon, Apr 15
The report from the analysis is now on-wiki: https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Structured_tasks/Add_an_image/Experiment_analysis,_March_2024
Fri, Mar 29
In T360829#9662018, @mpopov wrote:Morten, can you please confirm if you're probably going to use that definition or if there's a good chance you change the time frame based on your experience over the past several years? (No need to commit to anything here, just a simple vibes check.)
Mar 5 2024
Feb 27 2024
I agree with what's being said here. The performer values would reflect what the current user is, and if that user successfully logs in it'll reflect their logged in status. If the login attempt fails, it'll reflect their either anon or temp user status.
Feb 5 2024
Jan 30 2024
In T346327#9455949, @Sgs wrote:In T346327#9444225, @nettrom_WMF wrote:
- I'm definitely in favour of not having duplicate events. Just so I understand the event generation here, we're instrumenting both account creation and login, but the auto-login doesn't trigger a login event through this instrumentation? It's instead that it creates a second account creation "success" event?
See also my comments above T346327#9446991, a way of solving this could be to have a different dedicated schema for login attempts (and impressions?). Would that be appropriate @nettrom_WMF ?
Jan 23 2024
A written summary of our findings have been published on the Positive reinforcement project page in the section "Personalized praise experiment results". The notebooks associated with this analysis have been uploaded to this Gitlab repository and carry the name of this phab task. Closing as resolved.
A written summary of our findings have been published on the Positive reinforcement project page in the section "Personalized praise experiment results". The notebooks associated with this analysis have been uploaded to this Gitlab repository and carry the name of this phab task. Closing as resolved.
Jan 22 2024
The notebooks for the previously mentioned preliminary analysis of the effects of mentorship have been added to this Gitlab repository using a naming convention that reflects that they're associated with this phab task. Resolving this parent task now, and I'll close the child tasks once their notebooks are published as well.
Out of curiosity, and in hoping to make this easier to fix, I grabbed daily aggregates for October 2023. Looks like something was deployed the week of Oct 23 as that's when the proportion increases from the expected 6–7%:
Jan 9 2024
In T346327#9421209, @Sgs wrote:@nettrom_WMF would you mind taking a second look to the analytics schema change 962569, I've added some questions. Mainly, are we instrumenting all account creations or just from temporary users? And how should the auto-login after account creation be logged? Currently it would produce a "success"-Special:CreateAccount event which duplicates the account creation success. cc @Cyndymediawiksim
Jan 8 2024
Closing this task as declined for now. If the need for additional data analysis comes up, feel free to file another request.
Jan 4 2024
I've collected data and calculated the statistics for both of the campaigns using the date/time ranges listed in the task description.
Jan 3 2024
I've tested creating new environments on two stat machines with different OSes, and both with and without adding the conda-meta/pinned file. From what I can tell, the conda-forge R 4.2.3 package no longer results in a broken environment because it works just fine without the pin-file. Looking at the file listing on conda-forge, there have been several new versions of the package released since this bug was opened. Based on documentation I've found from googling it should be possible to install a specific build version, but it looks like the Mamba solver doesn't support it properly yet. It doesn't make sense to try this with the classic solver because that wasn't able to resolve the dependencies when I tried that back in August.
Dec 12 2023
Dec 6 2023
I've collected data to understand the effects of this campaign after about one week. During the data gathering for this update, I discovered a data quality issue in one of the underlying datasets that rendered us unable to correctly identify reverts (this has been filed as T352899). I was able to work around this by modifying the query that collects editing data.
Nov 29 2023
I've run the numbers on the currently available data^1 to get a sense of where we're at. Overall, we get the following counts and proportions of page views, "unique visitors" (based on combinations of IPs and user-agents for page views of the account creation page), registrations, and activations.
Nov 28 2023
In T350745#9361752, @spatton wrote:@KStoller-WMF and/or @nettrom_WMF, could you please let us know if you'd be able to provide an update on account signups some time after the first 24 hour mark, and then occasionally in December? Thanks!
Nov 15 2023
In T350745#9331846, @Sgs wrote:Do we have any kind of instrumentation for this in place? Or should we add it to the requirements and prepare some experiment so we can analyze the impact of the TY pages cc @nettrom_WMF
Nov 14 2023
Nov 10 2023
Nov 7 2023
Oct 24 2023
Oct 10 2023
Oct 2 2023
Sep 25 2023
Sep 19 2023
Sep 18 2023
This is no longer relevant, closing as invalid.
Sep 8 2023
I pulled data from Growth's KPI Grafana board from 2023-07-31 to 2023-08-28 (available here) for Section-Level and Article-Level suggestions. This timeframe was chosen because it should not be as much affected by the June/July slump in activity that we often see on the wikis. The end date is limited by the team shutting off image suggestions in late August (see T345188 for more information). This data range covers four whole weeks of data. While this dataset does not allow us to separate it by platform (desktop and mobile web), nor does it allow us more fine-grained user filtering, it was easily available and provides us with a reasonably good picture that's sufficient for this kind of analysis at this time.
Aug 29 2023
Aug 28 2023
Moving this out of the various doing/in progress columns as I'll instead be focusing on Levelling Up for a bit.
Aug 24 2023
Reducing the priority on this to Medium since we've completed the time-sensitive part of this.
Aug 23 2023
I've now completed testing using the EditAttemptStep data. There's a summary of the approaches, run times, and some observations and notes in this notebook on Gitlab.
Aug 18 2023
@KStoller-WMF : I made some changes to the summary of the findings. There's now a graph for the key finding, and I elaborated on the work we've done to investigate and mitigate the issue. I'll close this task as resolved, but you should feel free to edit the results section or reach out to me with proposed changes as you see fit.
Aug 14 2023
I've completed a preliminary analysis of mentorship and made a writeup of the findings on mw:Growth/Positive reinforcement/Mentorship preliminary analysis, August 2023. The goal of this analysis was to provide some initial insights into mentorship in support of the upcoming Wikimania discussion. We therefore focus on high-level questions such as what proportion of newcomers ask questions, how quickly they do, and to understand if mentorship has an impact on the Growth KPIs. The writeup is therefore also fairly high-level and without specifics for now, but does have some supporting visualizations.
Aug 8 2023
Aug 7 2023
@BTullis : Thank you for the update! While I no longer maintain the chart where this was an issue, I think switching the chart type to an Echarts type is a perfectly reasonable solution. I'd be happy to share this fix more widely on Slack as well so that others know about it, because I wasn't aware that there were deprecated chart types. Let me know if there are other chart types too that we should be aware of, I wasn't able to easily find a list of deprecated types.
I'll start digging into query performance this week. In the meantime, I want to make sure that previous work is documented. The experiments and notes around working with these flattened tables can be found in this notebook on gitlab.
Jul 27 2023
Jul 24 2023
Jul 21 2023
Added the Product Analytics tag, and myself as secondary assignee on this task. I've started looking at this and will continue that next week.
Jul 19 2023
From what I can tell we've deployed a patch to have the new impact module only switch out of deactivated state through article edits (Main namespace) per T338640, but that there was also a bug that affected it that got fixed in T341994. I'm going to restart the experiment clock in the notebook that aggregates statistics so that we can monitor whether there's a change after this patch. The aggregated leading indicators from the Levelling Up experiment will continue to exist in our database for comparison.
Jul 17 2023
Adding this to the Product Analytics kanban board. I'll pick this up and grab statistics on “How many newcomers/editors have at least one topic defined?” later this week.
Jul 7 2023
I've added this to Product Analytics tag and reset the priority so the PA team can triage it there. Given the timeliness of this work in preparation for Wikimania, I think the High priority should be re-added at triage. I'll then also move it to the PA kanban board to pick it up next week.
Jun 27 2023
Jun 26 2023
In T338640#8953804, @KStoller-WMF wrote:That way we are removing a potential confounding variable from the experiment. Any concerns with that approach? @nettrom_WMF / @RHo / @JFernandez-WMF
Jun 21 2023
Jun 16 2023
I introduced console-based event QA to @KStoller-WMF in our 1:1. We went through most of the workflow and checked the various events. From what we found the instrumentation is working correctly, there's nothing there that blocks deployment.
Jun 15 2023
Jun 9 2023
In T337320#8911657, @Tgr wrote:Yeah, at some point we stopped filtering by namespace. I'd assume that there are so few new users with a non-mainspace first edit that they can't cause the difference in activation... @nettrom_WMF is that something you can confirm or refute?
Jun 6 2023
@Urbanecm_WMF : From what I can tell, the notifications are being sent out by MediaWiki but not captured as events in event.mediawiki_mentor_dashboard_personalized_praise. The first query below is the Hive/Spark query of the event data, which returns a count of zero. The second I've run for each of the four pilot wiki MariaDB replicas (e.g. analytics-mysql eswiki --use-x1 on a stat* machine), and they all return a non-zero result.
Jun 5 2023
I've taken a first look at this by investigating two proportions. First, the proportion of newcomers with constructive article activation. Second, the proportion of those activating through Suggested Edits. While that doesn't specifically answer if users are able to find Suggested Edits and/or activate it, it should give us an indication of whether we are running into issues.
I'm reopening this after we discussed this in light of the recent deployment of the New Impact Module to additional wikis in T336203. There's a difference in how the Suggested Edits module behaves on desktop and mobile that we're interested in understanding. On desktop, users have to initialize the module before getting any suggestions, whereas on mobile the module is already initialized. Timeboxed to one hour to grab fresh data and investigate this four the pilot wikis, with an additional hour if we want to expand it to the nine other Wikipedia wikis we're currently deployed to.
Jun 1 2023
May 31 2023
I added a QA step to the acceptance criteria since we'll be wanting to verify that this doesn't alter the logged data in any way. Happy to help out with that as needed, let me know! Moving this to "Tracking" on the Product Analytics board for now.