Page MenuHomePhabricator

Measure user impact of using a central login / signup page
Closed, ResolvedPublic

Description

User story & summary:

As the Growth team, we want to ensure that the SUL3 (Single Unified Login) release does not negatively affect login or account creation, as these are critical steps in the editor funnel and user workflow.

Any disruption in account creation or login could hinder new user activation and editor retention, which are key metrics for Growth team objectives.

Task scope:

This task focuses on defining the analytics approach to measure the impact of changes to the login and signup flows resulting from the SUL3 release T348388: SUL3: Use a dedicated domain for login and account creation .

Specifically, it involves evaluating existing instrumentation, reviewing rollout plans, and recommending further tracking or engineering enhancements to accurately monitor user impact.

Note: The impact on users may depend on UX choices made in T364941: Decide on the login flow in MediaWiki under SUL3.

Acceptance Criteria:
  • Review the current instrumentation related to login and account creation flows (e.g., success rates, error tracking).
  • Summarize findings and insights in this task, highlighting any potential gaps.
  • Evaluate the rollout plan for SUL3; is there an opportunity for A/B testing?
  • Recommend additional instrumentation or engineering efforts (if needed) to ensure comprehensive tracking of user impact post-release.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr renamed this task from SUL3 metrics to Measure user impact of using a central login / signup page.May 22 2024, 12:50 PM
Tgr updated the task description. (Show Details)

@nettrom_WMF - was this deprioritized, or is this on the Product Analytics roadmap?

@KStoller-WMF: This was deprioritized and is no longer on the roadmap (tab "Q3"). I never received any formal request for support with SUL3 and it's not clear what OKR work depends on this, if any. My takeaway from the Slack thread where I asked about this and checking in with @nettrom_WMF in Q2 is that there's nothing for us to do.

Some of the challenges with SUL3 project is there isn't a product decision hinging here. This is going to happen and they have infrastructure in place in a way that meets the technical requirements of the browsers. The concern is "if something breaks, will we know? And will we know how it's broken?"
From the meetings and the conversations about the work, there's quite a bit of instrumentation and reporting already. There's an account registration dashboard in Grafana that tracks failed registrations.

If that is incorrect, please let me know.

KStoller-WMF added a subscriber: JTweed-WMF.

OK, thanks for the update, @mpopov!

I'm only peripherally involved in SUL3 work, so I'll leave it to @JTweed-WMF to decide if more should be done to monitor impact.

Tgr claimed this task.

This partially didn't happen, partially happened in the subtasks. In any case, no point in keeping the task open.