Page MenuHomePhabricator
Create Task
Maniphest T141048

cirrusSearchCheckerJob spamming logstash, errors on a few wikis?
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Cirrus jobs are failing on a couple of wikis. Not many. There's a tail, but not a terribly long one. The clustering on the top few make me think something's wrong with these indices (mainly names with underscores in it).

Screen Shot 2016-07-21 at 14.11.32.png (257×338 px, 25 KB)

Was pulled from this kibana query

Details

SubjectRepoBranchLines +/-
Consider retrying a checker job successmediawiki/extensions/CirrusSearchwmf/1.28.0-wmf.11+2 -2
Consider retrying a checker job successmediawiki/extensions/CirrusSearchmaster+2 -2
Customize query in gerrit

Event Timeline

• demon created this task.Jul 21 2016, 9:12 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptJul 21 2016, 9:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot subscribed.Jul 21 2016, 9:22 PM

Change 300424 had a related patch set uploaded (by EBernhardson):
Consider retrying a checker job success

https://gerrit.wikimedia.org/r/300424

gerritbot added a project: Patch-For-Review.Jul 21 2016, 9:22 PM
debt updated the task description. (Show Details)Jul 21 2016, 10:03 PM
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.Jul 21 2016, 10:10 PM
gerritbot added a comment.Jul 21 2016, 10:15 PM

Change 300424 merged by jenkins-bot:
Consider retrying a checker job success

https://gerrit.wikimedia.org/r/300424

gerritbot added a comment.Jul 21 2016, 10:39 PM

Change 300436 had a related patch set uploaded (by EBernhardson):
Consider retrying a checker job success

https://gerrit.wikimedia.org/r/300436

ReleaseTaggerBot added a project: MW-1.28-release (WMF-deploy-2016-07-26_(1.28.0-wmf.12)).Jul 21 2016, 11:00 PM
gerritbot added a comment.Jul 21 2016, 11:11 PM

Change 300436 merged by jenkins-bot:
Consider retrying a checker job success

https://gerrit.wikimedia.org/r/300436

EBernhardson subscribed.Jul 21 2016, 11:43 PM

In the 5 minutes since deploying the fix we've gone from 150/minute to 0. Looks fixed.

EBernhardson moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Jul 21 2016, 11:43 PM
ReleaseTaggerBot added a project: MW-1.28-release (WMF-deploy-2016-07-19_(1.28.0-wmf.11)).Jul 22 2016, 12:00 AM
debt subscribed.Jul 22 2016, 1:15 AM

Woohoo!

• demon closed this task as Resolved.Jul 22 2016, 3:55 PM
• demon claimed this task.
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL