Notice: Avro failed to serialize record for CirrusSearchRequestSet : {"payload":{"tookMs":"Expected string, but recieved integer"}} in /srv/mediawiki/php-1.27.0-wmf.10/includes/debug/logger/monolog/AvroFormatter.php on line 97]
Closed, ResolvedPublic

Description

a lot of these in the logs:

Notice: Avro failed to serialize record for CirrusSearchRequestSet : {"payload":{"tookMs":"Expected string, but recieved integer"}} in /srv/mediawiki/php-1.27.0-wmf.10/includes/debug/logger/monolog/AvroFormatter.php on line 97]

aude created this task.Jan 13 2016, 7:21 PM
aude updated the task description. (Show Details)
aude raised the priority of this task from to Needs Triage.
aude added a subscriber: aude.
Restricted Application added a project: Discovery. · View Herald TranscriptJan 13 2016, 7:21 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Deskana moved this task from Uncategorised to Technical on the CirrusSearch board.Jan 14 2016, 7:30 PM

A lot indeed:

krenair@fluorine:/a/mw-log$ tail -fn0 hhvm.log | grep "Avro failed to serialize record for CirrusSearchRequestSet" | pv -lri60 > /dev/null
[ 3.6/s]

Also, keep in mind that some of those lines are "message repeated x times" entries, some having x > 10000

Deskana moved this task from Needs triage to Ops on the Discovery board.Jan 15 2016, 8:56 PM

Quotes from IRC:

15:41:19 ebernhardson: Deskana: that's something we will have to fix next week, it's an easy enough fix
15:41:38 ebernhardson: it means our logging to hadoop is missing things

Change 264989 had a related patch set uploaded (by EBernhardson):
Avro payload can only contain string values

https://gerrit.wikimedia.org/r/264989

Fix for this will go out in ~ 5 hours. This most likely caused the weird dashboard values on searchdata

This is still spamming the logs like there's no tomorrow...

EBernhardson moved this task from Ops to On Sprint Board on the Discovery board.Jan 19 2016, 7:30 PM
EBernhardson claimed this task.

Change 264989 merged by jenkins-bot:
Avro payload can only contain string values

https://gerrit.wikimedia.org/r/264989

Deskana closed this task as Resolved.Jan 23 2016, 4:00 AM

This looks to be SWAT deployed now, so closing this one.

demon moved this task from Backlog to Done on the Wikimedia-log-errors board.Mar 11 2016, 11:58 PM