In today's team meeting we spoke about approach, creating a prototype table with the corrections and lowering the scope to phase it out for each metric area, starting with pageviews or unique devices.

We are leaning towards switching off the Prefetch feature since the fraction of requests with no-cookies is 17.5% . numbers in the notebook (last cell)

I think classifying the issues by data domain will definitely be useful. As a start, we can use workboard columns to do this, unless we decide it would be more useful to use columns for status (e.g. backlog, investigation, fixing...).

Yes! lets start with that.
@nshahquinn-wmf , do you have permission to create the yellow project tags? or can we request to have that created in this task ?

Ty @nshahquinn-wmf for opening this task. Sharing my initial thoughts below:

• Agree that we should tag all the tasks that reports data anomalies with this tag, for now. Once we use this method and find that the proportion of bot issues surpasses software bugs, or vice versa, we can think about splitting it out.
• I would like us to use this tag for data issues reported Community as well. Not sure if you already intended that but in the examples I saw tasks opened by WMF staff so wanted to make it explicit here
• I'm wondering if instead of one data-problem tag, we should instead have functional area tags like data-problem-traffic, data-problem-readership, data-problem-contributors, data-problem-content to keep track of the areas where we have data problems. But this would mean not having tags for tables that fall outside of functional areas (if we have any). And the quintessential problem of too many tags!

Similar to T307883, Im inclined towards declining this task since -

1. they've been open for a while
2. we are re-thinking our strategy for Contributors metric since it is now a core annual plan metric. @OSefu-WMF is there a task I can link here for the work you are doing?

Im inclined towards declining this task since -

1. they've been open for a while
2. we are re-thinking our strategy for Contributors metric since it is now a core annual plan metric. @OSefu-WMF is there a task I can link here for the work you are doing?

Observed results:
Querying the daily sum of unique devices across all domains using druid.unique_devices_per_domain_daily or all project families using druid.unique_devices_per_project_family_daily shows no daily data logged for January 2024 and February 2024. There is only one data point logged for both datasets 2024-01-01 and 2024-02-01. It appears this impacts all project families and domains. Daily data returns to expected values as of March 2024.

I got the below error when upgraded to Pandas 2.2.2

Can I add more scope to this?

@nshahquinn-wmf can you share the timeline you have planned for removing them from the repo?

• Dig into our data on user traffic and how we’re segmenting traffic

Hi @VirginiaPoundstone this work supports the Essential Work we have to perform of providing movement metrics each month to inform internal staff as well as share externally.
As a part of the monthly movement metrics, we need to analyze trends and investigate anomalies, to inform executive leadership as well as engage various teams to resolve potential issues. This view of the editor data will help us do that.

Summary
We have decided to close out the hypothesis.

Hi @lbowmaker , Since the Contributors metric in SDS1.1.1 was rejected and the hypothesis was disproven, we didn’t go ahead with implementing data stewardship even though data stewards were selected.

thanks @Hghani ! once this is corrected or confirmed as a caveat/known issue we should document it on Datahub.

sounds good ! thanks !!

great, thanks @lbowmaker for reminding me about that task! will review it.
yes, its ok to not implement something this FY, but having a good idea about what is possible and what we should have, is a good starting point for the remainder of this FY.

Dependency: need Essential metric criteria (T342472)

Completed and Closed out - https://app.asana.com/0/1206624865116405/1206625207405713

@lbowmaker / @Ahoelzl : I made this task as a placeholder for upcoming work we are going to be doing for Relevance metric data governance. I'll schedule time for us to discuss further in the coming weeks.

Wanted to note here that the smaller project families that were seeing increases during December every year did not see an increase in 2023-12 after this issue was fixed. see chart

We will be closing out Hypothesis SDS1.1.1 soon and creating new hypothesis to work on Relevance, Content and Programmatic Expense ratio.

Jaime and I have completed the close out report .
We have also drafted the new hypotheses for SDS1.1 in our project charter (see Relevance and Content metrics)
Both are pending review and approval.

thank you !! we can continue the conversations on Slack.

@mpopov , curious to know if some of us in Movement Insights can be added to this mailing list to get the alerts instead of creating and maintaining a separate mailing list. what are your thoughts about why this is a bad idea?

thanks @Ladsgroup. yes I'm able to query the mariadb table for a few wikis and can see '0' in the user_is_temp column. Ive opened T356701 and requested the Data Platform Engineering team to add this field to the tables required for data analysis.

T356701 has been opened to request data platform engineering to add the user_is_temp field in the relevant downstream data tables. This field will help us identify temporary accounts from registered and IP accounts.
(reference : T333223)

@Ladsgroup Is this added to the user tables in All the wikis? Do you know exactly when this was completed? Is Jan 22, 2024 the correct date based on the merge comment?
IIUC we will not be seeing any data in the user table for this field as the program is not yet rolled out.

@Hghani can you post update on this task?
Data-QA for the Q2 quarterly metrics is completed (T356229).

Everything looks good.
Methodology is documented here. Will look into moving this to a wiki page.

Currently in QA.

Another issue discovered recently T355608 which could benefit from improving automated bot detection.

Thanks @leila ! :)
pls keep us posted on your review.

@CMyrick-WMF Since you authored this task I'm assuming you had use cases in mind for using the cultural model. Would it be possible for you to add some usecases in the description ? thanks a lot!! <3

I wanted to note here that per the refactored SDS OKRs, the new SDS1.1 KR, "SDS1.1 For three out of the four core metric areas, provide at least 1 metric with documented adherence to essential metric criteria" , we now have a dependency on the approval of the Essential Metrics framework to continue work on SDS1.1

whats the priority on this?

Hi DPE team, Can you pls let me know the status of this request? I was not able to get any results for Sec-Purpose: Prefetch; anonymous-client-ip when querying the x_analytics field.
And a follow up question, can we get this added as a flag or new column to the readers tables derived from webrequest - pageviews_hourly and pageview_daily so its easier for us to query and visualize? pls let me know if this requires a new phab task.