Page MenuHomePhabricator

SDS1.3 Hypothesis 1: Working Group to create draft essential metrics selection framework
Closed, ResolvedPublic

Description

This task tracks efforts to establish a working group that will produce a draft essential metrics selection framework by the end of Q1 FY 2023-2024.

Objective/Hypothesis:
Establishing an expertise-based working group with a clear scope, mandate, and timeline will result in a clear and agreed upon process for reviewing, updating, or expanding the Essential Metrics.

How does this objective/hypothesis relate to organizational goals?

  • OKRs: This hypothesis directly relates to the 2023 fiscal year annual plan for the Product and Technology department to deliver on SDS objective 1 KR 3, which states:
    • SDS 1 Objective: Each metric and dimension in our essential metric data set is scientifically or empirically supported, standardized, productionized, and shared across the Foundation.
      • Key Result 3: Establish and implement a process to ensure that our essential metrics continually evolve to support data-informed decision making.

Notes / Resources:

Deliverables

  • Draft Essential Metrics selection framework
  • Draft governance model for the framework

Event Timeline

YLiou_WMF created this task.

Update:
Yu-Ming has begun to draft a project plan covering Q1-Q2 including the following:

  • Timeline and deliverables
  • Principles for essential metric definition and revision
  • Working group membership, meeting cadence, and scope of work

In addition, I met with @Mayakp.wiki (SDS1.1.1) and @JAnstee_WMF (SDS1.1.2) to clarify relationship between our hypotheses and working groups and to level up questions to Leila, Kate, and Desiree

Update:

Continue to update the draft project plan and worked with Leila to produce a working group structure modeled on the ERC core / extended team structure.

  • identified a tentative core working team list as well as a contact list for extended team nominations.
  • sketched out an initial working group agenda (defining hypothesis success/failure with an eye towards a proposed set of metrics selection principles)

In addition, since it has emerged that there remains some uncertainty about the specific scope of work tackled by SDS 2.4 and SDS 1.1 respectively, we have scheduled a sync with the SDS 1.1 team (Jaime, Maya, Kate) for next Wednesday (August 3).

Update:

Continue to update the draft project plan and worked with Leila to produce a working group structure modeled on the ERC core / extended team structure.

  • identified a tentative core working team list as well as a contact list for extended team nominations.
  • sketched out an initial working group agenda (defining hypothesis success/failure with an eye towards a proposed set of metrics selection principles)

In addition, since it has emerged that there remains some uncertainty about the specific scope of work tackled by SDS 2.4 and SDS 1.1 respectively, we have scheduled a sync with the SDS 1.1 team (Jaime, Maya, Kate) for next Wednesday (August 3).

YLiou_WMF renamed this task from SDS2.4 Hypothesis 1: Working Group to create draft essential metrics selection framework to SDS1.3 Hypothesis 1: Working Group to create draft essential metrics selection framework.Aug 4 2023, 4:00 PM
YLiou_WMF updated the task description. (Show Details)

Update:

  • As reflected on the name of this, this work has been moved from Objective 2 (Empirical Decision-Making) to Objective 1 (Defining Essential Metrics).
  • Yu-Ming and Leila met with Kate, Jaime, and Maya (along with Mariya) to discuss coordination between SDS 1.1 and (now) SDS 1.3 work.
  • Invitations for the core working group went out yesterday. As of today, we have received confirmations from Maya, Luke, and Miriam. Tentative group membership is listed here
  • A potential dependency relationship has been identified by the SDS 2.1 team and Luke will be joining the SDS 1.3 Working Group core team so we anticipate we will be able to maintain alignment.

Update:

  • Core team membership and meetings have been tentatively established/scheduled.
  • Extended team invitations (nomination requests, mostly) have gone out.
  • A potential dependency has been identified with SDS 3.4 and we are coordinating with Virginia Poundstone (KR owner) on how to avoid blocking them.

Update:

  • Core team membership has been finalized
  • Met with Mariya Shilova to discuss facilitation techniques for first core team meeting next Wednesday
  • Extended team still awaiting a handful of nominees

Update:

  • Initial core team meeting held September 30. Notes here (private doc).
  • Agenda slide deck available here
  • Takeaways / Next steps for next meeting:
    • Core team members will generate candidate essential metrics lists
    • Will build discussion of principles for metrics definition based on induction from existing metrics viewed as "essential"

Update:

  • Met with Leila to discuss takeaways and next steps based on session 1
  • Shared agenda for session 2 with core team
  • Importantly: at end of last week, Kate Z posted a definition of essential metrics here
  • next core team meeting to take place Wednesday September 13

Update:

  • Core team Meeting 2 was widely regarded by participants as extremely productive and successful!
  • Notes available here
  • We were able to achieve alignment on what is and is not in scope for the core team
  • Also began brainstorming principles / dimensions for determining whether something is or is not an essential metric
  • Between today and the next meeting (Septmber 27), core team will continue work asynchronously to:
    • clarify current proposals on the following principles / dimensions:
      1. Accessibility / Explanability
      2. Scientific Rigor
      3. Commitment
    • Generate additional principles / dimensions

Update:

  • Preparing for Core team Meeting 3
  • Tentatively planning for deep dive on the following dimensions during meeting on September 27:
    1. Accessibility / Explainability
    2. Commitment

Update:

  • Logistical updates:
    • Core team meeting cadence has been increased to weekly to ensure that we can complete our work in Q2 while accounting for time for feedback from the extended group
  • Core team meeting was held on September 27 and covered two key decision-making dimensions:
    • Metrics Accessibility (tentative proposal compiled)
    • Metrics Relevance (discussion is continuing asynchronously with an eye towards compiling a tentative proposal by our next meeting on October 4)

Update:

  • Core team met to tentatively define the Commitment and Rigor / Validity dimensions
  • Draft proposal for essential metrics governance under development

Update:

  • Core team met to discuss draft Governance Proposal
  • Final overall proposal (inclusive of dimensions / criteria and governance proposal) drafting has begun
  • Rigor / Validity dimension to be discussed next week along with possible new Privacy / Security dimension

Update:

  • Core team met to discuss Scientific Rigor dimension and potential new Security/Privacy dimension
  • Overall proposal drafting continues
  • Dot voting on dimension criteria to take place next week

Update:

  • Core team met to engage in dot-voting to produce a preliminary set of proposals along five dimensions:
    1. Security / Privacy
    2. Relevance
    3. Commitment
    4. Trust / Credibility
    5. Accessibility
  • Draft proposal is being prepared for review and comment by core team on November 8

Update:

  • Core team met to co-refine and -revise draft proposal to share with extended group

Update:

  • Extended group feedback session rescheduled from this week (November 16) to after Thanksgiving (November 30)
  • Core team met to surface and resolve (where possible) outstanding questions in advance of draft shareout with extended group

Update:

  • Draft decision-making framework shared with extended group for feedback (November 22)
  • Extended group feedback will be discussed on November 30

Update:

  • Held Q&A session with extended group for feedback on draft decision making framework
    • Overall takeaways:
      • Broad alignment with draft framework, but desire for more specifics / concrete requirements for Essential metrics
        • For example: desire for pre-commitment by Essential metrics owners and end users (decision makers) to actions taken given an observed change or level in that metric
      • Concern that "Essential"-ness is in tension with annual planning cycle, push to separate Essential metrics from core annual plan metrics (specifically, the notion that Essential metrics are better understood as aligning with longer-term goals).
        • Related: concern that there not be an excessive number of Essential metrics in the interest of both continuity and limiting overall staff time devoted to Essential metrics proposals
  • Next steps:
    • Incorporating extended group feedback into draft proposal during next core group meeting (December 6)
    • Revised draft proposal shared with extended group and Leila (December 8)

Update:

  • Working meeting with core group to incorporate revisions from extended group session on November 30.
  • Revised draft proposal shared with full core and extended group and Leila on December 8.
  • Next steps: incorporating feedback from Leila and retro session with core group on December 13

Update:

  • Working meeting with core group to incorporate revisions from extended group session on November 30.
  • Revised draft proposal shared with full core and extended group and Leila on December 8.

clarifying point: My understanding is that the document in front of me to review now is your recommendation to me for making a decision based on. I don't expect to give feedback to you and the committee at this point that may result in more iterations. Instead, I'll work with your recommendation to make a decision. Let me know if I should hold off as you may need to update the text to make it your final recommendation to me.

Sorry for the misunderstanding! I do not have any updates in mind so please
go ahead and review with an eye towards decision making!

  • Revised draft proposal shared with full core and extended group and Leila on December 8.

@YLiou_WMF , Can you share the draft proposal document here, or on Asana, or the Slack channel? Sorry, I am not sure I understood where it was shared?

Update:

  • Proposal shared with KR owner (@leila)
  • Retrospective conducted with core working group
  • Decision by Leila on proposal expected no later than January 19, 2024

I wanted to note here that per the refactored SDS OKRs, the new SDS1.1 KR, "SDS1.1 For three out of the four core metric areas, provide at least 1 metric with documented adherence to essential metric criteria" , we now have a dependency on the approval of the Essential Metrics framework to continue work on SDS1.1

acknowledged. (fwiw: Omari also flagged this privately to me, but it's good to have it clearly here as well.) working on it, and also will work with Omari to make sure you all have everything you need to be able to proceed with the other hypothesis.

Thanks @leila ! :)
pls keep us posted on your review.

@YLiou_WMF we have pre-emptively started documenting adherence of Relevance metric and after we have approval, as next steps, we would like to get your review on this document which could then be templatized for other essential metrics.

Weekly update:

Weekly update:

  • Provided recommendations for @leila's proposal for essential metrics definition and governance
  • Aligned on next steps for documentation and support for Movement Insights' SDS 1.1 work

Weekly update:

  • Shared high-level draft with Marcel and Virginia for input on what it means for metric to be in production

Weekly update:

  • shared recommended revisions to language on metric in production with Leila
  • Leila has approved the hypothesis and will share a formal writeup of criteria with @OSefu-WMF, @Mayakp.wiki , and @JAnstee_WMF

Weekly update:

  • Hypothesis has been formally accepted
  • Leila plans to post high-level accepted hypothesis language on meta
  • Yu-Ming to follow with documentation

Weekly updateL

  • Leila has posted an updated (high-level) definition of essential metrics based on the work of the working group on Meta-Wiki
  • I will continue to update with more detailed documentation and will plan to close the hypothesis on Asana next week (March 29)

Weekly update:

  • I've closed the project on Asana
  • Will wait for official go-ahead from @leila to close on Phab