mw.config empty on some pages (and fatal errors emitted) due to Unicode-unaware handling of UTF8 data by Lua
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

Request URL: https://be-tarask.wikipedia.org/wiki/20th_Century_Fox
Request ID: XTb7DApAIC0AAGlGmg0AAACC

message

[XTb7DApAIC0AAGlGmg0AAACC] /wiki/20th_Century_Fox   Exception from line 1522 of /srv/mediawiki/php-1.34.0-wmf.14/includes/resourceloader/ResourceLoader.php: JSON serialization of config data failed. This usually means the config data is not valid UTF-8.

trace

#0 /srv/mediawiki/php-1.34.0-wmf.14/includes/OutputPage.php(3170): ResourceLoader::makeConfigSetScript(array)
#1 /srv/mediawiki/php-1.34.0-wmf.14/includes/skins/Skin.php(683): OutputPage->getBottomScripts()
#2 /srv/mediawiki/php-1.34.0-wmf.14/includes/skins/SkinTemplate.php(457): Skin->bottomScripts()
#3 /srv/mediawiki/php-1.34.0-wmf.14/includes/skins/SkinTemplate.php(217): SkinTemplate->prepareQuickTemplate()
#4 /srv/mediawiki/php-1.34.0-wmf.14/includes/OutputPage.php(2580): SkinTemplate->outputPage()
#5 /srv/mediawiki/php-1.34.0-wmf.14/includes/MediaWiki.php(891): OutputPage->output(boolean)
#6 /srv/mediawiki/php-1.34.0-wmf.14/includes/MediaWiki.php(903): Closure$MediaWiki::main()
#7 /srv/mediawiki/php-1.34.0-wmf.14/includes/MediaWiki.php(515): MediaWiki->main()
#8 /srv/mediawiki/php-1.34.0-wmf.14/index.php(42): MediaWiki->run()
#9 /srv/mediawiki/w/index.php(3): include(string)
#10 {main}

Impact

The page-specific JSON data for client-side JavaScript is consistently unable to be serialised on some articles, due to Scribunto inserting invalid UTF-8 strings into the dataset.

As a result, client-side JavaScript is unable to initialise completely and some features may be broken or (if we're lucky) missing, due to variables that are expected to always be defined, now sometimes not be defined.

Notes

I'm assuming this is a simple config change that can be fixed in SWAT. Thus, I'm not treating it as a train blocker, but the problem should still be fixed, so I'm marking it as UBN.

Only happens on https://be-tarask.wikipedia.org/

Details

	Subject	Repo	Branch	Lines +/-
	Fix invalid UTF-8 in LuaSandbox profiler data	mediawiki/extensions/Scribunto	master	+16 -1

Customize query in gerrit

Related Objects

Mentioned In: rELUAeb01c6ad227c: Fix invalid UTF-8 in LuaSandbox profiler data
T226986: Client side error logging production launch
Mentioned Here: T245573: LuaSandbox should clean up short_src for multibyte-safe truncation
T149552: mediawiki-php-luasandbox produces broken build artifact for HHVM on CentOS

Event Timeline

• LarsWirzenius created this task.Jul 23 2019, 12:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 23 2019, 12:48 PM

• LarsWirzenius triaged this task as Unbreak Now! priority.Jul 23 2019, 12:49 PM

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJul 23 2019, 12:49 PM

• LarsWirzenius removed a parent task: T220740: 1.34.0-wmf.15 deployment blockers.Jul 23 2019, 12:50 PM

hashar updated the task description. (Show Details)Jul 23 2019, 1:01 PM

WDoranWMF added a project: Platform Engineering.Jul 23 2019, 3:24 PM

Not occurring very frequently, so lowering the priority. And not the "simple config change" theorized either.

I couldn't reproduce it with the URL given, but I found that https://be-tarask.wikipedia.org/wiki/Lady_Gaga was reproducing it. It turns out that the cached ParserOutput with key be_x_oldwiki:pcache:idhash:114544-0!canonical and timestamp 20190723103909 and revision id 1836172 included the string init <\320\234\320\276\320\264\321\203\320\273\321\214:\320\222\321\226\320\272\321\226\320\267\321\214\320\262\320\265\321\201\321\202\320\272\321\226/\320\272\320\260\320\275\321\204\321\226\320\263\321\203\321\200\320\260\321\206\321> in $data['scribunto-limitreport-profile'][3][0]. Just before the > is a truncated UTF-8 character. If the page gets reparsed and that module's method doesn't take so much time as to show up in the limit report, the error will go away.

The underlying cause is that Lua 5.1 defines LUA_IDSIZE as 60, so the name gets truncated at 59 bytes by Lua before the data structure is passed to LuaSandbox's luasandbox_timer_profiler_hook(). Unfortunately we probably can't change that without compiling Lua into LuaSandbox (see also T149552), so probably the easiest current fix would be to normalize the strings returned by LuaSandbox's getProfilerFunctionReport() in Scribunto_LuaSandboxEngine::getLimitReportData().

Krinkle renamed this task from Config error in be-tarask wiki (non-UTF8) to mw.config empty on some pages due to non-UTF8 data from LuaSandbox.Jul 23 2019, 3:46 PM

Krinkle moved this task from Untriaged to Jul2019/1.34.wmf.13+ on the Wikimedia-production-error board.

greg removed a subscriber: Platform Engineering.Jul 23 2019, 7:50 PM

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM

• mobrovac moved this task from Inbox to Backlog on the Platform Team Workboards (Clinic Duty Team) board.Sep 3 2019, 2:20 PM

• LarsWirzenius unsubscribed.Sep 3 2019, 2:53 PM

WDoranWMF moved this task from Backlog to Inbox on the Platform Team Workboards (Clinic Duty Team) board.Sep 11 2019, 5:39 PM

Still found in Logstash. Latest url is https://bn.wikipedia.org/w/index.php?printable=yes&mobileaction=toggle_view_desktop&title=%E0%A6%95%E0%A6%BF%E0%A6%AF%E0%A6%BC%E0%A7%87%E0%A6%AD, which I'm able to repro consistently.

Did some ad-hoc logging on mwdebug1001 and got:

> json_last_error_msg()
"Malformed UTF-8 characters, possibly incorrectly encoded"

> array_keys($configuration)
["wgPageParseReport"]

At that point I found this task and realised the issue was known already.

daniel moved this task from Inbox to Backlog on the Platform Team Workboards (Clinic Duty Team) board.Oct 8 2019, 9:57 AM

Anomie merged a task: T245573: LuaSandbox should clean up short_src for multibyte-safe truncation.Feb 19 2020, 2:21 PM

Anomie added a subscriber: Catrope.

In T245573 @Catrope suggested another potential way to fix this: cleaning up the string inside LuaSandbox after it comes from Lua, in luasandbox_timer_profiler_hook() and luasandbox_push_structured_trace().

Jdforrester-WMF renamed this task from mw.config empty on some pages due to non-UTF8 data from LuaSandbox to mw.config empty on some pages (and fatal errors emitted) due to Unicode-unaware handling of UTF8 data by LuaSandbox.Feb 19 2020, 11:55 PM

Anomie renamed this task from mw.config empty on some pages (and fatal errors emitted) due to Unicode-unaware handling of UTF8 data by LuaSandbox to mw.config empty on some pages (and fatal errors emitted) due to Unicode-unaware handling of UTF8 data by Lua.Feb 20 2020, 4:23 PM

In T228746#5897245, @Anomie wrote:

In T245573 @Catrope suggested another potential way to fix this: cleaning up the string inside LuaSandbox after it comes from Lua, in luasandbox_timer_profiler_hook() and luasandbox_push_structured_trace().

I didn't realize Language::normalize() existed; it's probably easier to use that in LuaSandboxEngine, than to try to fix it in C.

Krinkle mentioned this in T226986: Client side error logging production launch.Mar 12 2020, 9:47 PM

daniel moved this task from Backlog to Later on the Platform Team Workboards (Clinic Duty Team) board.Apr 14 2020, 9:00 PM

• AMooney edited projects, added Platform Engineering (Icebox); removed Platform Team Workboards (Clinic Duty Team), LuaSandbox, Scribunto, Wikimedia-production-error.Aug 12 2020, 5:19 PM

@AMooney: Assuming that "Set projects" was accidentally used instead of "Add projects", hence restoring some previous project tags.

• Mholloway moved this task from Jul2019/1.34.wmf.13+ to Older on the Wikimedia-production-error board.Aug 14 2020, 5:12 AM

• Mholloway moved this task from Older to Jul2019/1.34.wmf.13+ on the Wikimedia-production-error board.Aug 18 2020, 5:21 AM

Still seen over the past 7 days. About a hundred per day.

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:02 PM

Krinkle edited projects, added Platform Engineering; removed Platform Engineering (Icebox).Oct 26 2020, 5:40 PM

Requesing re-triage of production error still seen in production one year later.

Krinkle updated the task description. (Show Details)Oct 26 2020, 5:46 PM

Change 640270 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[mediawiki/extensions/Scribunto@master] Fix invalid UTF-8 in LuaSandbox profiler data

https://gerrit.wikimedia.org/r/640270

gerritbot added a project: Patch-For-Review.Nov 9 2020, 11:49 PM

If we ever compile our own Lua package, it would be nice to increase LUA_IDSIZE. It does say you are meant to change it:

/*
@@ LUA_IDSIZE gives the maximum size for the description of the source
@* of a function in debug information.
** CHANGE it if you want a different size.
*/
#define LUA_IDSIZE	60

I don't think it's worth making a new package just for that though. It's used for a fixed-size buffer on the stack, so it can't be infinity, but 300 seems reasonable.

• Clarakosi assigned this task to tstarling.Nov 10 2020, 2:51 PM

• Clarakosi edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.

Change 640270 merged by jenkins-bot:
[mediawiki/extensions/Scribunto@master] Fix invalid UTF-8 in LuaSandbox profiler data

https://gerrit.wikimedia.org/r/640270

jenkins-bot mentioned this in rELUAeb01c6ad227c: Fix invalid UTF-8 in LuaSandbox profiler data.Nov 10 2020, 3:18 PM

• Clarakosi closed this task as Resolved.Nov 10 2020, 3:28 PM

• Clarakosi moved this task from Later to Done on the Platform Team Workboards (Clinic Duty Team) board.

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.18; 2020-11-17).Nov 10 2020, 4:00 PM

Maintenance_bot removed a project: Patch-For-Review.Nov 10 2020, 4:11 PM