Page MenuHomePhabricator

exception invalidtoken from Util.tokensToString for list starting with <onlyinclude>
Closed, ResolvedPublic

Description

see sample log at wtp1007:/var/lib/parsoid/nohup.out

these entries are 20k lines per exception so I'll just give the start and end of one here:

Util.tokensToString, invalid token: undefined tokens: [ '<onlyinclude>',

{ dataAttribs: { tsr: [Object] } },
undefined,
{ dataAttribs: { tsr: [Object] } },
undefined,
'  ',
undefined,
undefined,
' kbt ',
undefined,
undefined,
' ',
undefined,
'Abadi',
{ name: 'a', attribs: [], dataAttribs: {} },
' ',
undefined,
undefined,
'  ',
undefined,
undefined,
' Levend ',
undefined,
undefined,
' 4300',
undefined,

...

'Zweeds',
{ name: 'a', attribs: [], dataAttribs: {} },
' ',
undefined,
undefined,
'  ',
undefined,
undefined,
'  ',
undefined,
undefined,
undefined,
{ dataAttribs: { tsr: [Object] } },
undefined,
{ dataAttribs: { tsr: [Object] } },
undefined,
undefined,
'</onlyinclude>' ]

Trace

at Object.Util.tokensToString (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.Util.js:346:13)
at stripMetaTags (/srv/deployment/parsoid/Parsoid/js/lib/ext.core.AttributeExpander.js:77:18)
at AttributeExpander._returnAttributes (/srv/deployment/parsoid/Parsoid/js/lib/ext.core.AttributeExpander.js:164:17)
at AttributeTransformManager.process (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.TokenTransformManager.js:1077:8)
at AttributeExpander.onToken (/srv/deployment/parsoid/Parsoid/js/lib/ext.core.AttributeExpander.js:125:6)
at AsyncTokenTransformManager.transformTokens (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.TokenTransformManager.js:611:17)
at AsyncTokenTransformManager.onChunk (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.TokenTransformManager.js:395:17)
at SyncTokenTransformManager.EventEmitter.emit (events.js:88:17)
at SyncTokenTransformManager.onChunk (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.TokenTransformManager.js:955:7)
at PegTokenizer.EventEmitter.emit (events.js:88:17)

The logs were filling with these at about 1 gb in 10 minutes or so, logging is temporarily turned off for now.


Version: unspecified
Severity: normal

Details

Reference
bz57707

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 2:28 AM
bzimport added a project: Parsoid.
bzimport set Reference to bz57707.

Change 99022 had a related patch set uploaded by GWicke:
Bug 57707: Hotfix, don't push undefined onto wikitext array

https://gerrit.wikimedia.org/r/99022

Change 99022 merged by jenkins-bot:
Bug 57707: Hotfix, don't push undefined onto wikitext array

https://gerrit.wikimedia.org/r/99022

We still have to investigate the source of the problem. The hotfix only prevents crashers and should prevents logs from filling up.

This doesn't appear to be a regression. The above trace seems to be coming from,

https://nl.wikipedia.org/wiki/Lijst_van_talen_van_de_wereld

If I checkout a commit from 8 months back, it spits out the same errors. A minimal test case that can be gleamed from that page is,

{|

-<onlyinclude>
-

</onlyinclude>|}

Oooh .. a test case .. just what was needed! How did you find a failing page?? :-)

google search for zweeds abadi levend ;-)

reminder that there are logs in /var/lib/parsoid on wtp1007 which you can look at to see what was being parsed before that error, some of those pages may be candidates.

Ah yes, indeed ... Gabriel and I both missed got distracted by the zillion 'undefined' entries in that log and missed the google keywords there! :-) Gabriel said he looked at the logs and couldn't find the source. In any case, this test case should be sufficient to debug and fix the problem.

Change 99096 had a related patch set uploaded by Subramanya Sastry:
(Bug 57707) Clean up dead code (that was the source of exceptions)

https://gerrit.wikimedia.org/r/99096

Change 99096 merged by jenkins-bot:
(Bug 57707) Clean up dead code (that was the source of exceptions)

https://gerrit.wikimedia.org/r/99096

Change 101279 had a related patch set uploaded by GWicke:
Merge "Bug 57707: Hotfix, don't push undefined onto wikitext array"

https://gerrit.wikimedia.org/r/101279

Change 101282 had a related patch set uploaded by GWicke:
Merge "(Bug 57707) Clean up dead code (that was the source of exceptions)"

https://gerrit.wikimedia.org/r/101282

Change 101331 had a related patch set uploaded by GWicke:
Bug 57707: Hotfix, don't push undefined onto wikitext array

https://gerrit.wikimedia.org/r/101331

Change 101334 had a related patch set uploaded by GWicke:
(Bug 57707) Clean up dead code (that was the source of exceptions)

https://gerrit.wikimedia.org/r/101334

Change 101282 merged by GWicke:
Merge "(Bug 57707) Clean up dead code (that was the source of exceptions)"

https://gerrit.wikimedia.org/r/101282

Change 101334 merged by GWicke:
(Bug 57707) Clean up dead code (that was the source of exceptions)

https://gerrit.wikimedia.org/r/101334

Change 101279 merged by GWicke:
Merge "Bug 57707: Hotfix, don't push undefined onto wikitext array"

https://gerrit.wikimedia.org/r/101279

Change 101331 merged by GWicke:
Bug 57707: Hotfix, don't push undefined onto wikitext array

https://gerrit.wikimedia.org/r/101331