Page MenuHomePhabricator
Create Task
Maniphest T365036

JSON serialization failures on media files
Open, Needs TriagePublic
Actions

Description

See https://logstash.wikimedia.org/goto/12b0f637fc0a134638415fa4558737c3

Seems to be related to the backport of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1030124 which happened at that time, but which fixed other bugs on Discussion Tools.

Details

SubjectRepoBranchLines +/-
Fix serialization errors in PageBundle extensiondatamediawiki/corewmf/1.43.0-wmf.5+75 -6
POC: detect getText before cache savemediawiki/coremaster+9 -0
Fix serialization errors in PageBundle extensiondatamediawiki/coremaster+75 -6
[JsonCodec, ParserCache] Improve debugging of serializability failuresmediawiki/corewmf/1.43.0-wmf.5+84 -28
[JsonCodec, ParserCache] Improve debugging of serializability failuresmediawiki/coremaster+84 -28
[ParserCache] Preserve information from the JsonException when logging failuresmediawiki/corewmf/1.43.0-wmf.5+1 -1
Customize query in gerrit

Related Objects

Event Timeline

cscott created this task.Wed, May 15, 3:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, May 15, 3:03 PM
gerritbot added a comment.Wed, May 15, 3:10 PM

Change #1031918 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@wmf/1.43.0-wmf.5] [ParserCache] Preserve information from the JsonException when logging failures

https://gerrit.wikimedia.org/r/1031918

gerritbot added a project: Patch-For-Review.Wed, May 15, 3:10 PM
cscott added a comment.Wed, May 15, 3:11 PM

Questions are: /who/ is adding non-serializable data to the ParserOutput, and /when/. Our suspicion is that somehow ParserOutput::getText() is being called on the ParserOutput *before* it is written to the cache, so that modifications made in the OutputTransform pipeline are being retroactively applied to the cached content.

TheresNoTime subscribed.Wed, May 15, 8:07 PM
gerritbot added a comment.Wed, May 15, 8:11 PM

Change #1031918 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.5] [ParserCache] Preserve information from the JsonException when logging failures

https://gerrit.wikimedia.org/r/1031918

Stashbot added a comment.Wed, May 15, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-15T20:12:15Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:1031918|[ParserCache] Preserve information from the JsonException when logging failures (T365036)]]

Stashbot added a comment.Wed, May 15, 8:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-15T20:15:01Z] <samtar@deploy1002> cscott and samtar: Backport for [[gerrit:1031918|[ParserCache] Preserve information from the JsonException when logging failures (T365036)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Wed, May 15, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-15T20:28:56Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:1031918|[ParserCache] Preserve information from the JsonException when logging failures (T365036)]] (duration: 16m 41s)

Maintenance_bot removed a project: Patch-For-Review.Wed, May 15, 8:30 PM
ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.5; 2024-05-14).Wed, May 15, 9:00 PM
gerritbot added a comment.Wed, May 15, 9:02 PM

Change #1032046 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] [JsonCodec, ParserCache] Improve debugging of serializability failures

https://gerrit.wikimedia.org/r/1032046

gerritbot added a project: Patch-For-Review.Wed, May 15, 9:02 PM
cscott added a comment.Wed, May 15, 9:06 PM

The 1031918 patch was backported and worked, but the information gleaned was a little disappointing: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.05.15?id=qKHwfY8BS8vmb5K1Na1f

Exactly the same message was in json_message as in message, which means this isn't something that JsonCodec::detectNonSerializableData caught, which (in turn) means it's something /inside/ of a JsonSerializable. That is, some object's ::jsonSerialize() method is returning an array containing non-serializable things, and detectNonSerializableData doesn't bother looking inside of the array returned by jsonSerialize() because it assumes you know what you're doing if you implement that.

https://gerrit.wikimedia.org/r/1032046 is a "moar debugging!" patch, which adds additional information (type, class name) to all of the places which throw an "Unable to serialize JSON" JsonException, as well as adding a "try harder" version of detectNonSerializableData used by ParserCache when it detects a serialization failure in order to try to pinpoint the exact property path leading to the problematic content.

ihurbain subscribed.Thu, May 16, 8:47 AM
gerritbot added a comment.Thu, May 16, 10:06 AM

Change #1032411 had a related patch set uploaded (by Isabelle Hurbain-Palatin; author: Isabelle Hurbain-Palatin):

[mediawiki/core@master] Fix serialization errors in PageBundle extensiondata

https://gerrit.wikimedia.org/r/1032411

gerritbot added a comment.Thu, May 16, 7:40 PM

Change #1032046 merged by jenkins-bot:

[mediawiki/core@master] [JsonCodec, ParserCache] Improve debugging of serializability failures

https://gerrit.wikimedia.org/r/1032046

gerritbot added a comment.Thu, May 16, 7:44 PM

Change #1032435 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@wmf/1.43.0-wmf.5] [JsonCodec, ParserCache] Improve debugging of serializability failures

https://gerrit.wikimedia.org/r/1032435

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.6; 2024-05-21); removed MW-1.43-notes (1.43.0-wmf.5; 2024-05-14).Thu, May 16, 8:00 PM
gerritbot added a comment.Thu, May 16, 9:49 PM

Change #1032435 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.5] [JsonCodec, ParserCache] Improve debugging of serializability failures

https://gerrit.wikimedia.org/r/1032435

Stashbot added a comment.Thu, May 16, 9:49 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T21:49:54Z] <jsn@deploy1002> Started scap: Backport for [[gerrit:1032435|[JsonCodec, ParserCache] Improve debugging of serializability failures (T365036)]]

Stashbot added a comment.Thu, May 16, 9:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T21:52:46Z] <jsn@deploy1002> cscott and jsn: Backport for [[gerrit:1032435|[JsonCodec, ParserCache] Improve debugging of serializability failures (T365036)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.5; 2024-05-14); removed MW-1.43-notes (1.43.0-wmf.6; 2024-05-21).Thu, May 16, 10:00 PM
Stashbot added a comment.Thu, May 16, 10:25 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T22:25:10Z] <jsn@deploy1002> Started scap: Backport for [[gerrit:1032435|[JsonCodec, ParserCache] Improve debugging of serializability failures (T365036)]]

Stashbot added a comment.Thu, May 16, 10:27 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T22:27:41Z] <jsn@deploy1002> jsn and cscott: Backport for [[gerrit:1032435|[JsonCodec, ParserCache] Improve debugging of serializability failures (T365036)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Thu, May 16, 10:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T22:47:08Z] <jsn@deploy1002> Finished scap: Backport for [[gerrit:1032435|[JsonCodec, ParserCache] Improve debugging of serializability failures (T365036)]] (duration: 21m 57s)

gerritbot added a comment.Thu, May 16, 11:15 PM

Change #1032605 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] POC: detect getText before cache save

https://gerrit.wikimedia.org/r/1032605

gerritbot added a comment.Fri, May 17, 2:55 PM

Change #1032411 merged by jenkins-bot:

[mediawiki/core@master] Fix serialization errors in PageBundle extensiondata

https://gerrit.wikimedia.org/r/1032411

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.6; 2024-05-21); removed MW-1.43-notes (1.43.0-wmf.5; 2024-05-14).Fri, May 17, 3:00 PM
gerritbot added a comment.Fri, May 17, 3:05 PM

Change #1032807 had a related patch set uploaded (by C. Scott Ananian; author: Isabelle Hurbain-Palatin):

[mediawiki/core@wmf/1.43.0-wmf.5] Fix serialization errors in PageBundle extensiondata

https://gerrit.wikimedia.org/r/1032807

ihurbain mentioned this in T365433: Cannot save VisualEditor content in File namespace.Tue, May 21, 2:46 PM
gerritbot added a comment.Thu, May 23, 8:32 AM

Change #1032807 abandoned by Isabelle Hurbain-Palatin:

[mediawiki/core@wmf/1.43.0-wmf.5] Fix serialization errors in PageBundle extensiondata

Reason:

wmf.6 is deployed

https://gerrit.wikimedia.org/r/1032807

MSantos edited projects, added Content-Transform-Team-WIP, Essential-Work; removed Parsoid.Thu, May 23, 2:22 PM
MSantos moved this task from Backlog to To Deploy on the Content-Transform-Team-WIP board.Fri, May 24, 10:48 AM
cscott mentioned this in T366395: CommonsMetadata DataCollector::verifyAttributeMetadata should use ParserOutput::getRawText().Fri, May 31, 10:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL