Measure the difference between the intended and actual execution of delayed notification jobs
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	Urbanecm_WMF
	May 26 2025, 2:32 PM

Description

Background

GrowthExperiments currently sends a Getting Started notification 48 hours after an user account is registered, assuming the user did not make enough(*) edits yet. This delay is implemented using a delayed job: notificationGettingStartedJob. The same use-case is present for notificationKeepGoingJob as well.

Problem

As the Growth-Team, we have limited visibility into the actual execution time of those jobs. We do not know if they are getting executed on time, too late or never.

Checklist

track the time delay between the job being intended to run and it actually running
create a Grafana panel that shows that data

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	feat(LevelingUp): Measure the delay between actual and intended notification timestamp	mediawiki/extensions/GrowthExperiments	master	+73 -8
	feat(LevelingUp): Measure the delay between actual and intended notification timestamp	mediawiki/extensions/GrowthExperiments	wmf/1.45.0-wmf.5	+73 -8

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	KStoller-WMF	T368187 [EPIC] Constructive activation experimentation (Growth work related to Wiki Experiences 1.2)
Resolved	KStoller-WMF	T392256 [EPIC] "Get Started" notifications
Resolved	Urbanecm_WMF	T395260 Measure the difference between the intended and actual execution of delayed notification jobs

Event Timeline

Urbanecm_WMF created this task.May 26 2025, 2:32 PM

Restricted Application added a project: Growth-Team. · View Herald TranscriptMay 26 2025, 2:32 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Urbanecm_WMF mentioned this in T394958: Shorten the NotificationGetStartedJob delay on pilot wikis as an experiment.May 26 2025, 2:32 PM

Michael subscribed.May 26 2025, 5:45 PM

Urbanecm_WMF triaged this task as Medium priority.May 28 2025, 1:27 PM

Urbanecm_WMF moved this task from Inbox to Up Next (estimated tasks) on the Growth-Team board.

KStoller-WMF added a parent task: T393722: Get Started Notification Experiment - finalize copy.Jun 3 2025, 5:17 PM

KStoller-WMF set the point value for this task to 3.Jun 9 2025, 4:07 PM

KStoller-WMF moved this task from Up Next (estimated tasks) to Current Sprint on the Growth-Team board.Jun 10 2025, 4:49 PM

KStoller-WMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.

Urbanecm_WMF removed a parent task: T393722: Get Started Notification Experiment - finalize copy.Jun 14 2025, 4:38 PM

Urbanecm_WMF added a parent task: T392256: [EPIC] "Get Started" notifications.

Urbanecm_WMF claimed this task.Jun 15 2025, 10:45 AM

Change #1158487 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] feat(LevelingUp): Measure the delay between actual and intended notification timestamp

https://gerrit.wikimedia.org/r/1158487

gerritbot added a project: Patch-For-Review.Jun 15 2025, 12:22 PM

Once the metric data starts flowing in, we will need to create the panel itself.

Change #1158487 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] feat(LevelingUp): Measure the delay between actual and intended notification timestamp

https://gerrit.wikimedia.org/r/1158487

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2025, 8:32 PM

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.6; 2025-06-17).Jun 16 2025, 9:00 PM

Change #1159927 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.45.0-wmf.5] feat(LevelingUp): Measure the delay between actual and intended notification timestamp

https://gerrit.wikimedia.org/r/1159927

gerritbot added a project: Patch-For-Review.Jun 17 2025, 7:02 AM

Change #1159927 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.45.0-wmf.5] feat(LevelingUp): Measure the delay between actual and intended notification timestamp

https://gerrit.wikimedia.org/r/1159927

Mentioned in SAL (#wikimedia-operations) [2025-06-17T07:54:30Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1159927|feat(LevelingUp): Measure the delay between actual and intended notification timestamp (T395260)]]

Mentioned in SAL (#wikimedia-operations) [2025-06-17T07:56:54Z] <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1159927|feat(LevelingUp): Measure the delay between actual and intended notification timestamp (T395260)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.5; 2025-06-10); removed MW-1.45-notes (1.45.0-wmf.6; 2025-06-17).Jun 17 2025, 8:00 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-17T08:04:53Z] <urbanecm@deploy1003> Finished scap sync-world: Backport for [[gerrit:1159927|feat(LevelingUp): Measure the delay between actual and intended notification timestamp (T395260)]] (duration: 10m 22s)

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2025, 8:31 AM

Urbanecm_WMF mentioned this in T397155: CI should prevent attempts to call GaugeMetric::observe().Jun 17 2025, 10:52 AM

The deployment in this task caused a production error: T397135: Error: Call to undefined method Wikimedia\Stats\Metrics\GaugeMetric::observe(). There is also a follow-up task to avoid running into similar issues in the future: T397155: CI should prevent attempts to call GaugeMetric::observe().

Created an initial Grafana dashboard with visualisation of this new data: https://grafana.wikimedia.org/goto/Wz56zGPNR?orgId=1. With this, I think this can go to QA. I also asked @Michael for feedback on the dashboard itself, if he has any.

So far, I do not see any issues on the associated Grafana board.

Urbanecm_WMF closed this task as Resolved.Jul 14 2025, 2:38 PM

Measure the difference between the intended and actual execution of delayed notification jobsClosed, ResolvedPublic3 Estimated Story PointsActions