Page MenuHomePhabricator

Notice: Avro failed to serialize record for CirrusSearchRequestSet : {"payload":{"tookMs":"Expected string, but recieved integer"}} in /srv/mediawiki/php-1.27.0-wmf.10/includes/debug/logger/monolog/AvroFormatter.php on line 97]
Closed, ResolvedPublicPRODUCTION ERROR

Description

a lot of these in the logs:

Notice: Avro failed to serialize record for CirrusSearchRequestSet : {"payload":{"tookMs":"Expected string, but recieved integer"}} in /srv/mediawiki/php-1.27.0-wmf.10/includes/debug/logger/monolog/AvroFormatter.php on line 97]

Event Timeline

aude raised the priority of this task from to Needs Triage.
aude updated the task description. (Show Details)
aude subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

A lot indeed:

krenair@fluorine:/a/mw-log$ tail -fn0 hhvm.log | grep "Avro failed to serialize record for CirrusSearchRequestSet" | pv -lri60 > /dev/null
[ 3.6/s]

Also, keep in mind that some of those lines are "message repeated x times" entries, some having x > 10000

Quotes from IRC:

15:41:19 ebernhardson: Deskana: that's something we will have to fix next week, it's an easy enough fix
15:41:38 ebernhardson: it means our logging to hadoop is missing things

Change 264989 had a related patch set uploaded (by EBernhardson):
Avro payload can only contain string values

https://gerrit.wikimedia.org/r/264989

Fix for this will go out in ~ 5 hours. This most likely caused the weird dashboard values on searchdata

This is still spamming the logs like there's no tomorrow...

Change 264989 merged by jenkins-bot:
Avro payload can only contain string values

https://gerrit.wikimedia.org/r/264989

This looks to be SWAT deployed now, so closing this one.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald Transcript