Page MenuHomePhabricator

Cirrus jobs failures: Received job sendData for unwritable cluster 0 0s after insertion
Closed, ResolvedPublic

Description

I'm seeing few failed jobs in CirrusSearch.log :

2015-10-25 22:50:16 mw1006 hiwiki CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-25 22:50:16 mw1014 hiwiki CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-25 23:13:48 mw1013 frwikiversity CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-25 23:13:48 mw1013 frwikiversity CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-25 23:13:48 mw1006 frwikiversity CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 00:59:42 mw1005 frwikiversity CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 00:59:42 mw1007 frwikiversity CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 00:59:42 mw1007 frwikiversity CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 04:29:59 mw1012 frwikibooks CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["content",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0}

There's 55 failures for 20151026 (Sun) and 552 for 20151024 (Fri).

It's not clear to me why we receive jobs with cluster 0, maybe a type conversion from null to 0?

It looks like there always a failure related to Param host does not exist before:

2015-10-26 09:16:23 mw1012 hiwiki CirrusSearch WARNING: Search backend error during counting links to 1 pages after 5: Param host does not exist {"pageCount":1,"took":5,"message":"Param host does not exist"} 
2015-10-26 09:16:23 mw1012 hiwiki CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 09:17:22 mw1014 hiwiki CirrusSearch WARNING: Search backend error during counting links to 1 pages after 5: Param host does not exist {"pageCount":1,"took":5,"message":"Param host does not exist"} 
2015-10-26 09:17:22 mw1014 hiwiki CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 10:01:49 mw1006 hiwiki CirrusSearch WARNING: Search backend error during counting links to 1 pages after 6: Param host does not exist {"pageCount":1,"took":6,"message":"Param host does not exist"} 
2015-10-26 10:01:49 mw1006 hiwiki CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 10:12:20 mw1009 hiwiki CirrusSearch WARNING: Search backend error during counting links to 1 pages after 6: Param host does not exist {"pageCount":1,"took":6,"message":"Param host does not exist"} 
2015-10-26 10:12:20 mw1009 hiwiki CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0} 
2015-10-26 10:14:13 mw1002 hiwiki CirrusSearch WARNING: Search backend error during counting links to 1 pages after 5: Param host does not exist {"pageCount":1,"took":5,"message":"Param host does not exist"} 
2015-10-26 10:14:13 mw1002 hiwiki CirrusSearch WARNING: Received job sendData for unwritable cluster 0 0s after insertion {"method":"sendData","arguments":["general",["[object] (Elastica\\Script: {})"],null],"diff":0,"cluster":0}

This Param host does not exist error seems to be thrown by Elastica (Elastica/vendor/ruflin/elastica/lib/Elastica/Param.php:140) and likely called by ElasticaConnection.php:121

Details

Related Gerrit Patches:
mediawiki/extensions/CirrusSearch : wmf/1.27.0-wmf.4Fix unwritable cluster errors
mediawiki/extensions/CirrusSearch : masterFix unwritable cluster errors

Event Timeline

dcausse created this task.Oct 26 2015, 11:33 AM
dcausse raised the priority of this task from to Needs Triage.
dcausse updated the task description. (Show Details)
dcausse added a subscriber: dcausse.
Restricted Application added a project: Discovery. · View Herald TranscriptOct 26 2015, 11:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 249669 had a related patch set uploaded (by EBernhardson):
Fix unwritable cluster errors

https://gerrit.wikimedia.org/r/249669

EBernhardson set Security to None.

Change 249669 merged by jenkins-bot:
Fix unwritable cluster errors

https://gerrit.wikimedia.org/r/249669

Change 249913 had a related patch set uploaded (by EBernhardson):
Fix unwritable cluster errors

https://gerrit.wikimedia.org/r/249913

Change 249913 merged by jenkins-bot:
Fix unwritable cluster errors

https://gerrit.wikimedia.org/r/249913

Deskana closed this task as Resolved.Nov 20 2015, 5:21 AM
Deskana added a subscriber: Deskana.
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 20 2015, 5:21 AM