Error: Couldn't find trailer dictionary
Closed, ResolvedPublic

Description

Spotted just now in fatalmonitor: Syntax Error: Couldn't find trailer dictionary

Might comes from TexLive / file.

https://logstash.wikimedia.org/goto/c89d4201d36e8f6a3177cdf9d82f9a6e

Last serie comes from mw1281.eqiad.wmnet , reported in the hhvm log channel.

Probably related is:Top-level pages object is wrong type

I have traced it to at least two API servers, the messages are spammed together:

Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
hashar created this task.Sep 15 2016, 12:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 15 2016, 12:17 PM
hashar updated the task description. (Show Details)Sep 15 2016, 12:18 PM
hashar updated the task description. (Show Details)Sep 26 2016, 6:21 PM
hashar updated the task description. (Show Details)Sep 26 2016, 6:24 PM
hashar updated the task description. (Show Details)Sep 26 2016, 6:43 PM
hashar triaged this task as Low priority.Oct 6 2016, 3:11 PM
mmodell raised the priority of this task from Low to High.Jan 27 2017, 5:00 PM
mmodell added a subscriber: mmodell.

Log spam like this is not low priority.

greg added a subscriber: greg.Jan 27 2017, 6:05 PM

Any idea of where to assign this task? Which component?

Wild guessing: Maybe texvc triggering, via the Math extension? (Or OCG?)

Is there any way to get binary names or file names involved in Logstash?

hashar added a comment.EditedFeb 23 2017, 4:09 PM

Yeah that might be texvc or OCG who knows really :(

The reason is wfShellExec() by default does not capture standard error. That ends up being relayed by HHVM to its own stderr and thus to logstash flagged as belonging to HHVM. T157646: wfShellExec errors end up in HHVM log talks about it, we should probably shallow stderr by default or rework wfShellExec() so it lets us capture each.

T138036 is a similar behavior coming from EasyTimeline. I have some low priority work in progress patch that would capture all output of wfShellExec AND adds structured logging so it shows up nicely in Logstash: https://gerrit.wikimedia.org/r/#/c/335238/

As already announced in Tech News, OfflineContentGenerator (OCG) will not be used anymore after October 1st, 2017 on Wikimedia sites. OCG will be replaced by Electron. You can read more on mediawiki.org.

Tgr closed this task as Resolved.Thu, Oct 26, 7:05 PM
Tgr claimed this task.
Tgr added a subscriber: Tgr.

Per T157646#3695779, should be fixed. (Well, it will still spam the logs but will use proper structured logging and a dedicated channel now.)