"JobExecutor not loaded" error for BounceHandlerJob on wikitech.wikimedia.org
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	Krinkle
	Jun 5 2019, 2:04 PM

Project Tags

Referenced Files

	F29350205: Screenshot 2019-06-05 at 15.03.55.png
	Jun 5 2019, 2:04 PM

Subscribers

Description

Error

Request ID: XPfF4QpAMD4AADNSoHcAAAAX

message

Exception from line 68 of /srv/mediawiki/rpc/RunSingleJob.php:

JobExecutor not loaded for job:

{ "database":"labswiki",
  "mediawiki_signature": …,
  "meta":{"domain":"wikitech.wikimedia.org","dt":"2019-06-05T13:38:41+00:00","id": …"request_id":"XPfF4QpAMD4AADNSoHcAAAAX","schema_uri":"mediawiki\/job\/3","topic":"mediawiki.job.BounceHandlerJob","uri":"https:\/\/placeholder.invalid\/wiki\/Special:Badtitle"},
  "page_namespace":-1,"page_title":":","params":{"email":…},
  "type":"BounceHandlerJob"
}

Impact

Some kinds of user actions on Wikitech wiki are presumably not working as intended.

Notes

Breakdown of last 30 days:

Screenshot 2019-06-05 at 15.03.55.png (2,110×870 px, 129 KB)

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Disable BounceHandler on Wikitech	operations/mediawiki-config	master	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: T237773: Move Wikitech onto the production MW cluster
Mentioned Here: T167973: Move database for wikitech (labswiki) to a main cluster section
T237773: Move Wikitech onto the production MW cluster
T208922: PHP Fatal Error: Class undefined: JobExecutor (jobrunners try to run labswiki jobs)
T192361: Transfer wikitech jobs to Kafka queue

Event Timeline

Krinkle created this task.Jun 5 2019, 2:04 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2019, 2:04 PM

Krinkle moved this task from Untriaged to Older on the Wikimedia-production-error board.Jun 5 2019, 2:04 PM

Wikitech should not be using kafka job queue at all per T192361#4139799

In mediawiki-config we set wmgUseClusterJobqueue to false for wikitech, thus it should be using the JobQueueDB. Plus we set the wikitech group to send events of TYPE_NONE - which means no events should be produced, so no jobs should get into kafka even if the queue is incorrect.

Something is clearly not working as expected, will investigate.

I think the root cause is the same as T208922: PHP Fatal Error: Class undefined: JobExecutor (jobrunners try to run labswiki jobs), namely T208922#4766050 (global config problems when spawning inter-wiki jobs). It's just that now we detect the case when JobExecutor is not loaded instead of bluntly crashing.

• Pchelolo edited projects, added: Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Platform Team Workboards (Clinic Duty Team); removed: Platform Team Legacy (Next), Services (next).Jul 17 2019, 3:00 AM

• WDoranWMF triaged this task as Low priority.Jul 17 2019, 8:14 PM

• WDoranWMF moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.

• WDoranWMF moved this task from Later to Inbox on the Platform Team Workboards (Clinic Duty Team) board.Aug 6 2019, 3:21 PM

• WDoranWMF removed a project: Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)).Aug 13 2019, 7:00 PM

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM

Krinkle moved this task from Untriaged to EventBus infra on the WMF-JobQueue board.Sep 9 2019, 9:03 PM

daniel moved this task from Inbox to Backlog on the Platform Team Workboards (Clinic Duty Team) board.Oct 8 2019, 9:57 AM

Krinkle unsubscribed.Oct 8 2019, 5:18 PM

• bd808 moved this task from Backlog to Snowflakes on the wikitech.wikimedia.org board.Nov 11 2019, 12:07 AM

Ok, it has happened again. The problem is essentially that varying the $wgJobTypeConf by wiki is not supported, so cross-wiki job posts for wiki tech don't work.

We should reevaluate why after all we can't go to kafka jobqueue on wiki tech. When we first deployed it, there were some reasons, but I don't remember them. I'll follow up and discuss with SRE.

In T225097#5924197, @Pchelolo wrote:

Ok, it has happened again. The problem is essentially that varying the $wgJobTypeConf by wiki is not supported, so cross-wiki job posts for wiki tech don't work.

We should reevaluate why after all we can't go to kafka jobqueue on wiki tech. When we first deployed it, there were some reasons, but I don't remember them. I'll follow up and discuss with SRE.

T237773: Move Wikitech onto the production MW cluster and related T167973: Move database for wikitech (labswiki) to a main cluster section are what is really needed. Wikitech is a snowflake deployment of MediaWiki. Today it runs on isolated servers with an isolated db. The core cluster job runners can't deal with that.

• AMooney moved this task from Backlog to Teleport on the Platform Team Workboards (Clinic Duty Team) board.Mar 24 2020, 1:47 PM

• WDoranWMF edited projects, added: Platform Engineering (Icebox); removed: Platform Team Workboards (Clinic Duty Team).Mar 24 2020, 9:48 PM

I traced this issue to this configuration in Puppet (Codesearch):

common/mail/mx.yaml

profile::mail::mx::verp_post_connect_server: meta.wikimedia.org
profile::mail::mx::verp_bounce_post_url: api-rw.discovery.wmnet/w/api.php

…
/usr/bin/curl -H 'Host: <%= @verp_post_connect_server %>' <%= @verp_bounce_post_url %> -d "action=bouncehandler" --data-urlencode "email@-" -o /dev/null

This is currently leading to the ChangeProp jobrunner making requests to the mw* machines for jobrunners with a wikitech.wikimedia.org header and initialising part of MediaWiki until it finds that JobExecutor is not installed there.

Given this isn't working right now, shall we disable the BounceHandler extension on Wikitech for now, citing that it is only supported in combination with EventBus/Kafka/JobQueue?

• bd808 mentioned this in T237773: Move Wikitech onto the production MW cluster.Jul 10 2020, 12:16 AM

Change 818286 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] Disable BounceHandler on Wikitech

https://gerrit.wikimedia.org/r/818286

gerritbot added a project: Patch-For-Review.Jul 29 2022, 2:00 AM

Change 818286 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable BounceHandler on Wikitech

https://gerrit.wikimedia.org/r/818286

Krinkle closed this task as Resolved.Aug 2 2022, 12:01 AM

Krinkle claimed this task.

Krinkle edited projects, added: Performance-Team; removed: Platform Engineering (Icebox).

Maintenance_bot removed a project: Patch-For-Review.Aug 2 2022, 12:30 AM