EmailNotification.php on line 222
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	hashar
	Apr 30 2025, 8:26 AM

Description

Error

service.version: 1.44.0-wmf.25
trace.id: c3f7c975-e97c-478f-9a4c-8e0bcd38a4e6
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   TypeError: MediaWiki\Mail\RecentChangeMailComposer::__construct(): Argument #6 ($timestamp) must be of type string, null given, called in /srv/mediawiki/php-1.44.0-wmf.25/includes/mail/EmailNotification.php on line 222

Frame	Location	Call
from	/srv/mediawiki/php-1.44.0-wmf.25/includes/mail/RecentChangeMailComposer.php(96)
#0	/srv/mediawiki/php-1.44.0-wmf.25/includes/mail/EmailNotification.php(222)	MediaWiki\Mail\RecentChangeMailComposer->__construct(MediaWiki\User\User, MediaWiki\Title\Title, null, null, null, null, string)
#1	/srv/mediawiki/php-1.44.0-wmf.25/includes/mail/EnotifNotifyJob.php(57)	EmailNotification->actuallyNotifyOnPageChange(MediaWiki\User\User, MediaWiki\Title\Title, null, null, null, null, array, string)
#2	/srv/mediawiki/php-1.44.0-wmf.25/extensions/EventBus/includes/JobExecutor.php(88)	EnotifNotifyJob->run()
#3	/srv/mediawiki/rpc/RunSingleJob.php(60)	MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#4	{main}

Impact

My understanding is MediaWiki does not send recent changes emails notifications anymore. I have thus rolled back the train (T386222).

Notes

There were some recent changes mades to includes/mail such as https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1136760.

Details

MediaWiki Version: 1.44.0-wmf.27
Request URL: https://mw-jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	EnotifNotifyJob: Forward-compat for wmf.27 jobs	mediawiki/core	wmf/1.44.0-wmf.25	+10 -3

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	dduvall	T386222 1.44.0-wmf.27 deployment blockers
		Resolved	PRODUCTION ERROR	matmarex	T392988 TypeError: MediaWiki\Mail\RecentChangeMailComposer::__construct(): Argument #6 ($timestamp) must be of type string, null given, called in /srv/mediawiki/php-1.44.0-wmf.25/includes/mail/EmailNotification.php on line 222

Event Timeline

hashar created this task.Apr 30 2025, 8:26 AM

hashar triaged this task as Unbreak Now! priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 30 2025, 8:26 AM

hashar added a parent task: T386222: 1.44.0-wmf.27 deployment blockers.Apr 30 2025, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-30T08:29:57Z] <hashar> Rolled back MediaWiki train from group 1 to group 0 due to T392988 # T386222

Stashbot mentioned this in T386222: 1.44.0-wmf.27 deployment blockers.Apr 30 2025, 8:29 AM

The stack traces are for wmf.25 but the errors surged as I was pushing wmf.27 which is why I have rolled the train. But maybe the two events are unrelated.

A_smart_kitten added a project: MediaWiki-Email.Apr 30 2025, 8:43 AM

rMW4dfbf67f4d63: Pass RecentChange back to EmailNotif changed EnotifNotifyJob (which eventually passes some of its parameters to RecentChangeMailComposer) to not have a timestamp. This is not a cross-wiki job; it would be understandable if existing jobs started to fail (because the job parameters were stored with the wmf.25 code but processed with the wmf.27 code) but this error seems to be the other way around - it's the old code that passed an explicit timestamp and the new one that doesn't (and the errors are also thrown on the old branch). So maybe just a deploy artifact during the time when the servers had mixed code, so wmf.27 created jobs that were processed by wmf.25? The error spike is not that narrow, but then wmf.27 jobs would keep erroring after the rollback so that's to be expected.

Ladsgroup subscribed.Apr 30 2025, 11:03 AM

So maybe just a deploy artifact during the time when the servers had mixed code, so wmf.27 created jobs that were processed by wmf.25?

Sounds likely – canary servers create jobs, ???, normal servers process them?

We don't have a way to "pin" a job to a MW version, do we? That'd be nice.

We can either backport some fix to wmf.25 so that it can accept jobs in the new format too, or revert the wmf.27 changes. I will have a look and find out what is easier to do.

Change #1140184 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/core@wmf/1.44.0-wmf.25] EnotifNotifyJob: Forward-compat for wmf.27 jobs

https://gerrit.wikimedia.org/r/1140184

gerritbot added a project: Patch-For-Review.Apr 30 2025, 1:57 PM

That patch is only for wmf.25 and should unblock the wmf.27 deployment.

Test plan (on a local test wiki):

Make sure jobs don't run automatically
Switch to wmf.27
Make some edits, causing some jobs to be generated
Switch to wmf.25
Run jobs – This should cause errors like in the task
Apply the patch
Run jobs again – This time they should succeed, and they should send correct email notifications

It's a bit tedious and time-consuming, I'm going through the steps right now.

Looks good to me.

Matmarex has verified the patch works for him. I will look at deploying https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1140184 .

Change #1140184 merged by jenkins-bot:

[mediawiki/core@wmf/1.44.0-wmf.25] EnotifNotifyJob: Forward-compat for wmf.27 jobs

https://gerrit.wikimedia.org/r/1140184

Maintenance_bot removed a project: Patch-For-Review.Apr 30 2025, 3:30 PM

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.25; 2025-04-15).Apr 30 2025, 4:00 PM

There was a Gerrit maintenance that started immediately after the patch got merged. It is now being deployed by Timo as part of another deployment!

I will proceed with the train.

The patch has been deployed but the train is blocked on an issue we have with Gerrit ( T393034 ).

matmarex closed this task as Resolved.May 1 2025, 11:25 AM

matmarex claimed this task.

TypeError: MediaWiki\Mail\RecentChangeMailComposer::__construct(): Argument #6 ($timestamp) must be of type string, null given, called in /srv/mediawiki/php-1.44.0-wmf.25/includes/mail/EmailNotification.php on line 222Closed, ResolvedPublicPRODUCTION ERRORActions