Page MenuHomePhabricator
Create Task
Maniphest T145772

Error: Couldn't find trailer dictionary
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Spotted just now in fatalmonitor: Syntax Error: Couldn't find trailer dictionary

Might comes from TexLive / file.

https://logstash.wikimedia.org/goto/c89d4201d36e8f6a3177cdf9d82f9a6e

Last serie comes from mw1281.eqiad.wmnet , reported in the hhvm log channel.

Probably related is:Top-level pages object is wrong type

I have traced it to at least two API servers, the messages are spammed together:

Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:25 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't find trailer dictionary
Sep 26 10:10:26 mw1191 hhvm[1810]: Syntax Error: Couldn't read xref table

Related Objects
Search...

Event Timeline

hashar created this task.Sep 15 2016, 12:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 15 2016, 12:17 PM
hashar updated the task description. (Show Details)Sep 15 2016, 12:18 PM
hashar updated the task description. (Show Details)Sep 26 2016, 6:21 PM
hashar updated the task description. (Show Details)Sep 26 2016, 6:24 PM
hashar updated the task description. (Show Details)Sep 26 2016, 6:43 PM
hashar triaged this task as Low priority.Oct 6 2016, 3:11 PM
• demon moved this task from Untriaged to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Jan 10 2017, 10:55 PM
• mmodell raised the priority of this task from Low to High.Jan 27 2017, 5:00 PM
• mmodell subscribed.

Log spam like this is not low priority.

• mmodell added parent tasks: T154683: MW-1.29.0-wmf.9 deployment blockers, T155525: MW-1.29.0-wmf.10 deployment blockers.Jan 27 2017, 5:06 PM
greg subscribed.Jan 27 2017, 6:05 PM

Any idea of where to assign this task? Which component?

• mmodell removed a parent task: T154683: MW-1.29.0-wmf.9 deployment blockers.Jan 27 2017, 6:43 PM
TTO mentioned this in T155525: MW-1.29.0-wmf.10 deployment blockers.Jan 30 2017, 2:35 PM
Aklapper added a subscriber: Physikerwelt.Feb 22 2017, 12:50 PM

Wild guessing: Maybe texvc triggering, via the Math extension? (Or OCG?)

Is there any way to get binary names or file names involved in Logstash?

hashar added a parent task: T157646: wfShellExec errors end up in HHVM log.Feb 23 2017, 4:05 PM
hashar added a comment.EditedFeb 23 2017, 4:09 PM

Yeah that might be texvc or OCG who knows really :(

The reason is wfShellExec() by default does not capture standard error. That ends up being relayed by HHVM to its own stderr and thus to logstash flagged as belonging to HHVM. T157646: wfShellExec errors end up in HHVM log talks about it, we should probably shallow stderr by default or rework wfShellExec() so it lets us capture each.

T138036 is a similar behavior coming from EasyTimeline. I have some low priority work in progress patch that would capture all output of wfShellExec AND adds structured logging so it shows up nicely in Logstash: https://gerrit.wikimedia.org/r/#/c/335238/

greg edited projects, added Math, OCG-General; removed Release-Engineering-Team.May 20 2017, 12:20 PM

Per T145772#3046446

Aklapper added a comment.Sep 14 2017, 9:17 AM

As already announced in Tech News, OfflineContentGenerator (OCG) will not be used anymore after October 1st, 2017 on Wikimedia sites. OCG will be replaced by Electron. You can read more on mediawiki.org.

• mmodell awarded a token.Sep 15 2017, 4:39 PM
Tgr closed this task as Resolved.Oct 26 2017, 7:05 PM
Tgr claimed this task.
Tgr subscribed.

Per T157646#3695779, should be fixed. (Well, it will still spam the logs but will use proper structured logging and a dedicated channel now.)

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits