Page MenuHomePhabricator

Inconsistent metrics results for 90 day rolling active editors calculation
Closed, DeclinedPublic

Description

Hello. This is the first time I am reporting a "suspected" bug, apologies if the description is rather unorthodox.

I have been calculating metrics for the project Let's Fill the Gender Gap that took place in Geneva in 2015 and 2016 and I have received several strange results on a variety of attempts at running reports with different cohorts in Wikimetrics.

For instance, here are the results I came across when calculating the 90 day retention rate for active rolling editors for the following cohorts ( 5 edits) :

Letsfillthegendergap_session1 ( As from 15th December 2015)
->I obtained the figure of 22 out of 27 participants - very high
Letsfillthegendergap_session2 ( As from the 12th of April 2016)
->I obtained the figure of 0 when I know that a minimum of 3 active contributors should appear in the results - very low
Letsfillthegendergap_sessionHESSO ( As from the 24th May 2016)
I obtained the figure of 12 out of 13 - very high
Letsfillthegendergap_group4_V2 ( As of the 28th June 2016)
I obtained the figure of 19 out of 21 - very high

When changing the number of monthly edits to try to figure out a trend the exact same figures come out from 5 to 15 to 25 monthly edits which indicates that there is probably an underlying issue.

I use google chrome as a web browser - I have no idea if this might have an incidence.

I had to calculate many metrics for those workshops and I am know quite unsure of the validity of the results I obtained. I would much appreciate any help to check if the 4 enquiries above have experienced any issues and to have any advice on how to be sure my previously collected results are correct. Many thanks for your help.

Kind regards,
Gabrielle

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 3 2016, 10:02 AM

Hi @gabrielle_marie_wmch, thanks for taking the time to report this!
How exactly are your metrics collected and calculated? What are steps that would allow someone else to reproduce the problem?
Which project is this task about?
Thanks! :)

Hi @Aklapper, thanks for your quick response.
This task is related to calculating the metrics for the 4 workshops related to this project:
I will try to illustrate the steps I would carry for each workshop in Wikimetrics:

Workshop 1

Step 1- Select my cohort Letsfillthegendergap_session1 , which is basically the list of usernames on this page.

Step 2 - Select the metric "active rolling editors"

Step 3 - Leave the by default figure 5

Step 4- Input 90 days instead of the 30 days suggested by default

Step 5 - Input the date of the 15th December 2015 ( the date on which the workshop ended).

Step 6 - Validate the configuration

Step 7 - Run the report

Workshop 2

Step 1- Select my cohort Letsfillthegendergap_session2 , which is basically the list of usernames on this page.

Step 2 - Select the metric "active rolling editors"

Step 3 - Leave the by default figure 5

Step 4- Input 90 days instead of the 30 days suggested by default

Step 5 - Input the date of the 12th April 2016 ( the date on which the workshop ended).

Step 6 - Validate the configuration

Step 7 - Run the report

Workshop 3

Step 1- Select my cohort Letsfillthegendergap_sessionHESSO , which is basically the list of usernames on this page.

Step 2 - Select the metric "active rolling editors"

Step 3 - Leave the by default figure 5

Step 4- Input 90 days instead of the 30 days suggested by default

Step 5 - Input the date of the 24th May 2016 ( the date on which the workshop ended).

Step 6 - Validate the configuration

Step 7 - Run the report

Workshop 4

Step 1- Select my cohort Letsfillthegendergap_group4_V2 , which is basically the list of usernames on this page.

Step 2 - Select the metric "active rolling editors"

Step 3 - Leave the by default figure 5

Step 4- Input 90 days instead of the 30 days suggested by default

Step 5 - Input the date of the 28th June 2016 ( the date on which the workshop ended).

Step 6 - Validate the configuration

Step 7 - Run the report

Many thanks for your assistance.

Kind regards,
Gabrielle

@gabrielle_marie_wmch: That sounds pretty awesome (as far as I understand French) but I'm still trying to understand why a task was created here. :)
Is there a technical problem you would like to see fixed by someone else? If so, where exactly to see the problem?
Or is this task used for planning your work for yourself, and you plan to work on this task? If it is the latter, you would need a project here. :)

Hello @Aklapper , apologies if I got it completely wrong. My understanding was that this was the place to find support when noticing an issue with Wikimetrics, by creating a task.
My difficulty is that I know for a fact that the figures provided when I run those particular reports ( above) are incorrect as I know some of the active contributors participating to the project. For instance, the second cohort calculation provides a the figure of 0 active rolling editors when there are at least 3 in the list of usernames. I am so sorry if I am in the wrong place and end up wasting your time. Would much appreciate any guidance or redirection please.
Many thanks for your patience :)

Ah! Thanks a lot! All makes sense now, and sorry for my confusion! So this task is about Community-Wikimetrics and Analytics-Wikimetrics, I see. Adding those projects to this task so the Analytics team can take a look here.

Ah, thank you so much. i am sorry i got the notion of the project wrong, I was thinking in terms of Wikimedia project. Thank you so much for your patience.

Abit assigned this task to Milimetric.Oct 28 2016, 12:44 PM
Abit added a subscriber: Abit.

Dan, any ideas about this one? The pages Gabrielle linked include the cohort and start and end dates for each report. It sounds like it couldn't be the same problem as Michael was having?

Abit added a comment.Oct 28 2016, 12:47 PM

@gabrielle_marie_wmch could you please attach the reports to the phab task as well?

@Abit: I'm sorry but these investigations are too time consuming and so far have resulted in "it works as designed but it's complicated and perhaps not designed properly". My team isn't well suited to this and my free time is being pulled in a million directions these days, so I can't help here. I will, however, support any attempt to get organizational support for this work, including advocating for more resources for your team. In my opinion, there are two possibilities:

  1. wikimetrics is considered a valuable tool. In this case we should either get top-down support for prioritizing it over other responsibilities of the Analytics team or we should hire more people to support it. Not doing so is repeating an anti-pattern at WMF: letting good tools go to waste
  1. wikimetrics is considered outdated and to be replaced by something else, like PAWS. In this case we have the same choice as above. Either we decide that this is important for the Analytics team or we hire more people.

In my opinion, any other course of action is basically implying that this kind of work should be done manually, and that hundreds of hours of labor are to be wasted on it.

mforns closed this task as Declined.Mar 26 2019, 2:25 PM
mforns added a subscriber: mforns.

Declining because Wikimetrics is being discontinued. See: T211835.

Restricted Application added a project: Analytics. · View Herald TranscriptMar 26 2019, 2:25 PM