Page MenuHomePhabricator
Create Task
Maniphest T199594

Exception "Job queue is read-only"
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

This exception has started to show little over a month ago (possibly earlier) in Logstash under mediawiki-errors.

Frequency: About 20-40 fatal exceptions each day.

Only for enwiki and only from codfw appservers and api_appservers (mw2177, mw2137, mw2144, ..). Most likely triggered by monitoring requests, but still these exceptions shouldn't happen.

Is it not possible to write to the active jobqueue from codfw? While it's fine not to support database writes (given we don't support active-active yet, so there aren't meant to be write requests there), as far as I know, we do have a small number of jobs that can be inserted from read-only requests - as indicated by these errors.

[W0lDugrAIEIAAL2owt0AAAAL] /wiki/Main_Page   JobQueueReadOnlyError from line 693 of /srv/mediawiki/php-1.32.0-wmf.12/includes/jobqueue/JobQueue.php:

Job queue is read-only: MediaWiki is in read-only mode for maintenance. Please try again in a few minutes.

#0 /srv/mediawiki/php-1.32.0-wmf.12/includes/jobqueue/JobQueue.php(316): JobQueue->assertNotReadOnly()
#1 /srv/mediawiki/php-1.32.0-wmf.12/includes/jobqueue/JobQueue.php(302): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-1.32.0-wmf.12/includes/jobqueue/JobQueueGroup.php(154): JobQueue->push(array)
#3 /srv/mediawiki/php-1.32.0-wmf.12/includes/jobqueue/JobQueueGroup.php(218): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-1.32.0-wmf.12/includes/deferred/MWCallableUpdate.php(34): JobQueueGroup::pushLazyJobs()
#5 /srv/mediawiki/php-1.32.0-wmf.12/includes/deferred/DeferredUpdates.php(268): MWCallableUpdate->doUpdate()
#6 /srv/mediawiki/php-1.32.0-wmf.12/includes/deferred/DeferredUpdates.php(214): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#7 /srv/mediawiki/php-1.32.0-wmf.12/includes/deferred/DeferredUpdates.php(134): DeferredUpdates::execute(array, string, integer)
#8 /srv/mediawiki/php-1.32.0-wmf.12/includes/MediaWiki.php(913): DeferredUpdates::doUpdates(string)
#9 /srv/mediawiki/php-1.32.0-wmf.12/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#10 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#11 {main}

Details

SubjectRepoBranchLines +/-
JobQueue: Signal JobQueueEventBus is never read-onlyoperations/mediawiki-configmaster+1 -1
JobQueueGroup: Allow readOnlyReason to be specified per JQ typemediawiki/coremaster+1 -1
JobQueue: Allow jobs to be enqueued despite $wgReadOnlymediawiki/coremaster+18 -3
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Jul 14 2018, 4:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2018, 4:51 AM
Krinkle updated the task description. (Show Details)Jul 14 2018, 4:53 AM
Krinkle updated the task description. (Show Details)
• Fjalapeno added a project: Services.Jul 17 2018, 1:39 AM
• mobrovac edited projects, added SRE, User-Joe, Services (designing); removed Services.Jul 17 2018, 8:26 AM
• mobrovac edited subscribers, added: • Pchelolo, Joe, • mobrovac; removed: Aklapper.

Thanks for unearthing this, @Krinkle . This is probably the last thing we forgot to change in the JobQueue switch. So, both the JobQueue and the DB layer in MW are controlled by the $wgReadOnly variable, which is set to a truth-y value in codfw. It is read from EtcD.

It seems that the way forward would be to decouple these into multiple settings, since now we have the semantics of "enqueuing a job" and "executing a job" which don't happen in the same place any longer. @Joe, @Pchelolo what do you think?

Joe added a comment.Jul 17 2018, 8:52 AM
In T199594#4429722, @mobrovac wrote:

Thanks for unearthing this, @Krinkle . This is probably the last thing we forgot to change in the JobQueue switch. So, both the JobQueue and the DB layer in MW are controlled by the $wgReadOnly variable, which is set to a truth-y value in codfw. It is read from EtcD.

It seems that the way forward would be to decouple these into multiple settings, since now we have the semantics of "enqueuing a job" and "executing a job" which don't happen in the same place any longer. @Joe, @Pchelolo what do you think?

I guess that we could in fact decouple enqueueing and executing jobs, but I'm not sure why we would want a read-only datacenter to enqueue jobs. Is there a scenario where this makes sense? Isn't this probably some issue in the API code that tries to enqueue jobs even from a read-only dc where there is no traffic?

Do we have more info about the jobs that we failed to submit? that could help us understand which requests triggered it.

• mobrovac added a comment.Jul 17 2018, 9:50 AM
In T199594#4429792, @Joe wrote:

I guess that we could in fact decouple enqueueing and executing jobs, but I'm not sure why we would want a read-only datacenter to enqueue jobs. Is there a scenario where this makes sense? Isn't this probably some issue in the API code that tries to enqueue jobs even from a read-only dc where there is no traffic?

Since now the JobQueue execution path is controlled by other entities (EventBus, ChangeProp, etc), MW should no longer be concerned with where the job is enqueued; wherever a job is pushed into the queue, our set up ensures it will be executed in the correct DC. This would bring us one (small) step closer to having an active-active setup. Moreover, since the JobQueue transport and execution mechanism now works differently, it makes sense conceptually to make the difference between enqueuing a job and executing it.

Do we have more info about the jobs that we failed to submit? that could help us understand which requests triggered it.

These indeed seem to be monitoring requests as their URI is /wiki/Main_Page (for enwiki) coming from einsteinium.

herron triaged this task as High priority.Jul 18 2018, 6:32 PM
• tstarling added subscribers: aaron, • tstarling.Jul 19 2018, 6:25 AM

To serve read traffic correctly, $wgReadOnly needs to be false. $wgReadOnly is mostly a UI-layer concept which shows some informative message to the user, not just on POST, but also on confirmation pages. So it's not really necessary to fix this to implement active-active support, we can just set $wgReadOnly to false. So I don't think this is high priority, it is just logspam.

That said, I don't know why pushing a job should throw an exception when wfReadOnly() is true. It was proposed in T130795 by @aaron, partly in order to have "app level read-only errors instead [of] redis error spam about read-only mode". But that seems to be an argument for configuring a read-only mode on a per-queue basis, rather than using the global read only mode.

Krinkle moved this task from Untriaged to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Jul 19 2018, 7:12 AM
• mobrovac lowered the priority of this task from High to Medium.Jul 19 2018, 8:34 AM
In T199594#4436984, @tstarling wrote:

To serve read traffic correctly, $wgReadOnly needs to be false. $wgReadOnly is mostly a UI-layer concept which shows some informative message to the user, not just on POST, but also on confirmation pages. So it's not really necessary to fix this to implement active-active support, we can just set $wgReadOnly to false. So I don't think this is high priority, it is just logspam.

The problem with $wgReadOnly is that it affects all operations, whereas here we just don't want it to affect the enqueuing of jobs. Hence, it makes sense to keep $wgReadOnly for execution paths that would provoke a change in the system, but remove its impact on execution paths that don't (directly/immediately) do so.

gerritbot subscribed.Jul 19 2018, 9:19 AM

Change 446762 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/core@master] JobQueue: Allow jobs to be enqueued despite $wgReadOnly

https://gerrit.wikimedia.org/r/446762

gerritbot added a project: Patch-For-Review.Jul 19 2018, 9:19 AM
aaron added a comment.Jul 19 2018, 12:52 PM

Normally, it would be odd to let jobs pile up but not execute them, though the multi-DC use case of $wgReadOnly in one of the DCs wasn't considered in T130795. Ideally, jobs enqueued on GET/HEAD wouldn't be a thing...but that's not going away anytime soon.

gerritbot added a comment.Jul 19 2018, 4:18 PM

Change 446866 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/core@master] JobQueueGroup: Allow readOnlyReason to be specified per JQ type

https://gerrit.wikimedia.org/r/446866

gerritbot added a comment.Jul 20 2018, 10:30 AM

Change 446866 merged by jenkins-bot:
[mediawiki/core@master] JobQueueGroup: Allow readOnlyReason to be specified per JQ type

https://gerrit.wikimedia.org/r/446866

gerritbot added a comment.Jul 20 2018, 10:38 AM

Change 446762 abandoned by Mobrovac:
JobQueue: Allow jobs to be enqueued despite $wgReadOnly

Reason:
Superseded by I8f1a57a81ea11c1c587c0057fa8bb3454b0e0b56

https://gerrit.wikimedia.org/r/446762

gerritbot added a comment.Jul 20 2018, 10:43 AM

Change 447055 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/mediawiki-config@master] JobQueue: Signal JobQueueEventBus is never read-only

https://gerrit.wikimedia.org/r/447055

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)).Jul 20 2018, 11:00 AM
• tstarling moved this task from Inbox to Watching on the Core-Platform-Team-Old board.Jul 24 2018, 1:46 AM
gerritbot added a comment.Jul 26 2018, 8:15 AM

Change 447055 merged by jenkins-bot:
[operations/mediawiki-config@master] JobQueue: Signal JobQueueEventBus is never read-only

https://gerrit.wikimedia.org/r/447055

Stashbot subscribed.Jul 26 2018, 8:18 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-26T08:18:25Z] <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings.php: Set readOnlyReason to false everywhere for JobQueueEventBus - T199594 (duration: 00m 55s)

• mobrovac claimed this task.Jul 26 2018, 11:30 AM
• mobrovac edited projects, added Services (done); removed Patch-For-Review, Services (designing).

These errors should disappear shortly once wmf.14 is deployed. Keeping the task open until that happens.

• mobrovac closed this task as Resolved.Jul 30 2018, 8:45 PM

The errors have completely disappeared as of this morning UTC.

CCicalese_WMF moved this task from Watching to Closed on the Core-Platform-Team-Old board.Aug 6 2018, 10:28 PM
Krinkle moved this task from Dec2019/1.35.wmf.10+ to Resolved on the Wikimedia-production-error board.May 29 2019, 4:00 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL