Page MenuHomePhabricator

edits seem to be very slow (due to Redis / jobrunners)
Closed, ResolvedPublic

Description

I noticed today that the saving of edits takes a very long time, at least much slower than earlier this week. It's most obvious to observe if you update a whole bunch of labels and aliases and you press save. You can see this in https://www.wikidata.org/w/index.php?title=Q2825156&action=history . I did all my edits and pressed save. The first part is saved 11:42, 3 May 2015‎ and the last one 11:44, 3 May 2015‎ so this took over one minute to complete!

I also notice that my bots appear to be editing slower than usual.

Event Timeline

Multichill raised the priority of this task from to Needs Triage.
Multichill updated the task description. (Show Details)
Multichill added a project: Wikidata.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2015, 10:02 AM
Lydia_Pintscher renamed this task from Wikidata saving of edits seems to be very slow to edits seem to be very slow.May 3 2015, 11:11 AM
Lydia_Pintscher added a project: acl*sre-team.
Lydia_Pintscher set Security to None.

This doesn't seem to be a Wikidata issue only. Others are complaining about similar issues on other wikis as well.

Editing is very slow on commons too, and other wikis.

(Other users can confirm this / @Jianhui67 , @Sjoerddebruin)

API too:

API request failed (internal_api_error_DBQueryError): [f65e4865] Database query error
Steinsplitter triaged this task as High priority.May 3 2015, 11:20 AM
Steinsplitter added a project: Commons.
Steinsplitter moved this task from Incoming to Backlog on the Commons board.
Jianhui67 added a comment.EditedMay 3 2015, 11:22 AM

Well, VFC on Commons was lagging for me just now. I thought I was the only one experiencing this issue until I asked around.

Joe added a subscriber: Joe.May 3 2015, 12:31 PM

We have identified an issue with connections to Redis, probably due to the fact that the jobrunners are not working since last night. We're working on it now.

Stryn added a subscriber: Stryn.May 3 2015, 2:03 PM
Joe added a comment.May 3 2015, 3:25 PM

After a few back and forth, We're pretty sure the cause of the outage was due to the changes in the jobchron service on the jobrunners that were released on saturday via

https://gerrit.wikimedia.org/r/#/c/208408/

when I correctly restarted the jobchron service (which is not named at all on the jobrunners deploy page on wikitech) after reverting that change the contentions on redis disappeared.

We earlier tracked the problem to redis maxing out 1 CPU, while blocked in the lua interpreter, which is probably single-threaded.

Joe added a comment.May 3 2015, 3:26 PM

Users report editing is fast again.

Joe closed this task as Resolved.May 3 2015, 3:27 PM
Joe claimed this task.
Krenair added a subscriber: greg.May 3 2015, 3:30 PM

We'll need to create incident documentation for this.

Krenair added subscribers: aaron, BBlack.

CCing people involved in fixing it

Still problems with deleting stuff on commons [It takes forever]

In T97930#1255507, @Joe wrote:

Users report editing is fast again.

Aklapper renamed this task from edits seem to be very slow to edits seem to be very slow (due to Redis / jobrunners).May 19 2015, 1:07 PM