eqiad.mediawiki.job.sendCampaignEmail (and possibly others) seemingly aren't enqueued in beta (kafka split-brain)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Daimona
	Aug 1 2025, 1:56 PM

Project Tags

Referenced Files

None

Subscribers

Description

In T400928, we noticed that in beta, organizers are unable to email participants, which is implemented via EmailUsersJob. I did some debugging in T400928#11053888, reusing some of the commands from T387631. What seems to happen is that:

MediaWiki pushes the job to the job queue
The job event is POSTed to eventgate-main, and this is successful (HTTP 201)
However, kafkacat does not show the job being enqueued

On top of that, checking /srv/mw-log/JobExecutor.log, there are only instances of newcomerTasksCacheRefreshJob and no other job, which seems suspicious, and possibly a sign that this affects many more jobs than just the organizer emails.

Related Objects

Mentioned In: T423615: Jobs are not being processed in beta, April 2026 edition
T422842: Kafka-topics broken in beta: "zookeeper is not a recognized option"
T400928: Spike - check if participants are receiving email from organizer and event registration confirmation on beta and prod
Mentioned Here: T387631: Are eqiad.mediawiki.job.CampaignEventsFindPotentialInvitees jobs being processed in beta?
T400928: Spike - check if participants are receiving email from organizer and event registration confirmation on beta and prod

Event Timeline

Daimona created this task.Aug 1 2025, 1:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2025, 1:56 PM

Daimona mentioned this in T400928: Spike - check if participants are receiving email from organizer and event registration confirmation on beta and prod.Aug 1 2025, 1:56 PM

Maintenance_bot added a project: Data-Engineering.Aug 1 2025, 2:30 PM

vaughnwalters subscribed.Aug 1 2025, 2:41 PM

Checking the archives logs (until July 21), I only see instances of newcomerTasksCacheRefreshJob in JobExecutor.log, and nothing else. So presumably many jobs affected?

However, kafkacat does not show the job being enqueued

If eventgate 201s and event is not in Kafka that is bad!

But:

otto@deployment-kafka-main-5:~$ kafkacat -u -C -b localhost:9092 -t eqiad.mediawiki.job.sendCampaignEmail -o beginning | jq .meta.dt
"2025-07-31T21:26:17.721Z"
"2025-07-31T21:26:58.363Z"
"2025-08-01T00:50:00.218Z"
"2025-08-01T13:15:07.300Z"
"2025-08-01T13:30:38.034Z"
"2025-08-01T13:48:23.929Z"

I see 5 of these jobs in that topic since July 31. Offset for this topic is at 106, which means since creation there have been 106 messages produced. Topic retention is 7 days, so anything before the latest 5 either was purged, or there was an actual problem before July 31.

Ah indeed, I SSHed to deployment-kafka-main-5 and confirmed it gets enqueued, but that wasn't the case in deployment-kafka-main-6:

daimona@deployment-kafka-main-5:~$ kafkacat -b localhost:9092 -C -t eqiad.mediawiki.job.sendCampaignEmail -o end
% Reached end of topic eqiad.mediawiki.job.sendCampaignEmail [0] at offset 106
{"$schema":"/mediawiki/job/1.0.0","meta":{"uri":"https://placeholder.invalid/wiki/Special:Badtitle","request_id":"aJD1tASsTBS8YU61U5Mu5QAAAEc","id":"008e080d-d2ff-4139-8d20-6bdc7f6197bf","domain":"en.wikipedia.beta.wmcloud.org","stream":"mediawiki.job.sendCampaignEmail","dt":"2025-08-04T18:02:29.273Z"},"database":"enwiki","type":"sendCampaignEmail","sha1":"773fe6e0097ae8d75a9555d5326263657fe25945"
[...]

Not sure what's going on then.

! Indeed!

otto@deployment-kafka-main-6:~$ kafkacat -u -C -b localhost:9092 -t eqiad.mediawiki.job.sendCampaignEmail -o beginning
% Reached end of topic eqiad.mediawiki.job.sendCampaignEmail [0] at offset 96

That is very strange. kafka-main-6 is one of the 2 brokers in the main-eqiad kafka cluster in beta.

Connecting to either of them should give the same result.

Indeed, only broker 1 (kafka-main-5) is in the ISR:

kafka-topics --zookeeper deployment-zookeeper-3.deployment-prep.eqiad1.wikimedia.cloud/kafka/main-deployment-prep --describe --topic eqiad.mediawiki.job.sendCampaignEmail
Topic:eqiad.mediawiki.job.sendCampaignEmail	PartitionCount:1	ReplicationFactor:2	Configs:
	Topic: eqiad.mediawiki.job.sendCampaignEmail	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1

I think this means kafka topic replication is broken. But still, both hosts show that only replica 1 is in the ISR and is the leader, so attempting connecting to either should result in consuming from replica 1, and give the same results!

...

I restarted kafka on kafka-main-6. kafka-main-5 become the leader and I see:

[2025-08-04 18:53:29,925] INFO [Partition eqiad.changeprop.retry.mediawiki.job.sendCampaignEmail-0 broker=1] Expanding ISR from 1 to 1,2 (kafka.cluster.Partition)

And from kafka-main-6:

otto@deployment-kafka-main-6:~$ kafka topics --describe --topic eqiad.mediawiki.job.sendCampaignEmail
kafka-topics --zookeeper deployment-zookeeper-3.deployment-prep.eqiad1.wikimedia.cloud/kafka/main-deployment-prep --describe --topic eqiad.mediawiki.job.sendCampaignEmail
Topic:eqiad.mediawiki.job.sendCampaignEmail	PartitionCount:1	ReplicationFactor:2	Configs:
	Topic: eqiad.mediawiki.job.sendCampaignEmail	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

otto@deployment-kafka-main-6:~$ kafkacat -u -C -b localhost:9092 -t eqiad.mediawiki.job.sendCampaignEmail -o beginning | jq .meta.dt
...
% Reached end of topic eqiad.mediawiki.job.sendCampaignEmail [0] at offset 107

I don't know what happened, and understanding exactly would take more time. But, I would hope/guess that changeprop-jobqueue is now able to consume messages correctly.

BPirkle moved this task from Incoming (Needs Triage) to Radar (other teams work) on the MediaWiki-API-Platform-Team board.Aug 4 2025, 7:36 PM

Thank you! I retried the steps in T400928#11053888, and this time I received the email, so everything works as expected. Other jobs are being run too:

$ egrep -o '"job":"[^ ]+' JobExecutor.log | sort | uniq -c | sort -r
    466 "job":"parsoidCachePrewarm
    432 "job":"RecordLintJob
    288 "job":"newcomerTasksCacheRefreshJob
    236 "job":"mediaModerationScanFileJob
    150 "job":"EchoNotificationDeleteJob
    138 "job":"EchoPushNotificationRequest
     68 "job":"refreshLinks
     68 "job":"cirrusSearchLinksUpdate
     62 "job":"wikibase-InjectRCRecords
     38 "job":"htmlCacheUpdate
     36 "job":"EntityChangeNotification
     34 "job":"DispatchChanges
     18 "job":"recentChangesUpdate
     16 "job":"CentralAuthCreateLocalAccountJob
      4 "job":"notificationKeepGoingJob
      4 "job":"notificationGetStartedJob
      4 "job":"enotifNotify
      2 "job":"sendCampaignEmail
      2 "job":"activityUpdateJob

I am resolving this task given T401002#11058756; might reopen if it starts happening again.

Is there some monitoring that we should have to check for split brain in kafka? Or a troubleshooting step we can add to https://wikitech.wikimedia.org/wiki/Kafka or https://wikitech.wikimedia.org/wiki/Kafka/Administration?

bd808 moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.Aug 4 2025, 8:26 PM

Monitoring under replicated partitions would have caught this.

We alert in prod on this.

bd808 renamed this task from eqiad.mediawiki.job.sendCampaignEmail (and possibly others) seemingly aren't enqueued in beta to eqiad.mediawiki.job.sendCampaignEmail (and possibly others) seemingly aren't enqueued in beta (kafka split-brain).Aug 4 2025, 8:54 PM

Daimona mentioned this in T422842: Kafka-topics broken in beta: "zookeeper is not a recognized option".Apr 9 2026, 3:12 PM

Daimona mentioned this in T423615: Jobs are not being processed in beta, April 2026 edition.Apr 16 2026, 3:21 PM

eqiad.mediawiki.job.sendCampaignEmail (and possibly others) seemingly aren't enqueued in beta (kafka split-brain)Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

eqiad.mediawiki.job.sendCampaignEmail (and possibly others) seemingly aren't enqueued in beta (kafka split-brain)
Closed, ResolvedPublic
Actions