edits seem to be very slow (due to Redis / jobrunners)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Multichill
	May 3 2015, 10:02 AM

Description

I noticed today that the saving of edits takes a very long time, at least much slower than earlier this week. It's most obvious to observe if you update a whole bunch of labels and aliases and you press save. You can see this in https://www.wikidata.org/w/index.php?title=Q2825156&action=history . I did all my edits and pressed save. The first part is saved 11:42, 3 May 2015‎ and the last one 11:44, 3 May 2015‎ so this took over one minute to complete!

I also notice that my bots appear to be editing slower than usual.

Event Timeline

Multichill created this task.May 3 2015, 10:02 AM

Multichill raised the priority of this task from to Needs Triage.

Multichill updated the task description. (Show Details)

Multichill added a project: Wikidata.

Multichill added subscribers: Multichill, Romaine, Sjoerddebruin, Lydia_Pintscher.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2015, 10:02 AM

This doesn't seem to be a Wikidata issue only. Others are complaining about similar issues on other wikis as well.

Editing is very slow on commons too, and other wikis.

(Other users can confirm this / @Jianhui67 , @Sjoerddebruin)

API too:

API request failed (internal_api_error_DBQueryError): [f65e4865] Database query error

Steinsplitter triaged this task as High priority.May 3 2015, 11:20 AM

Steinsplitter added a project: Commons.

Steinsplitter moved this task from Incoming to Backlog on the Commons board.

Well, VFC on Commons was lagging for me just now. I thought I was the only one experiencing this issue until I asked around.

Multichill added a project: Performance Issue.May 3 2015, 11:31 AM

Krenair subscribed.May 3 2015, 12:06 PM

We have identified an issue with connections to Redis, probably due to the fact that the jobrunners are not working since last night. We're working on it now.

Stryn subscribed.May 3 2015, 2:03 PM

After a few back and forth, We're pretty sure the cause of the outage was due to the changes in the jobchron service on the jobrunners that were released on saturday via

https://gerrit.wikimedia.org/r/#/c/208408/

when I correctly restarted the jobchron service (which is not named at all on the jobrunners deploy page on wikitech) after reverting that change the contentions on redis disappeared.

We earlier tracked the problem to redis maxing out 1 CPU, while blocked in the lua interpreter, which is probably single-threaded.

Users report editing is fast again.

Joe closed this task as Resolved.May 3 2015, 3:27 PM

Joe claimed this task.

We'll need to create incident documentation for this.

CCing people involved in fixing it

Still problems with deleting stuff on commons [It takes forever]

In T97930#1255507, @Joe wrote:

Users report editing is fast again.

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150503-JobQueueRedis

Aklapper renamed this task from edits seem to be very slow to edits seem to be very slow (due to Redis / jobrunners).May 19 2015, 1:07 PM

edits seem to be very slow (due to Redis / jobrunners)Closed, ResolvedPublicActions

Description

Event Timeline

edits seem to be very slow (due to Redis / jobrunners)
Closed, ResolvedPublic
Actions