Page MenuHomePhabricator

Page_props allows max of 64KB values, forcing invalid graph definitions JSON
Closed, ResolvedPublic

Description

Very large graph definitions, totaling over 64KB, don't fit into page_props DB table, and thus end up trimmed / invalid JSON. Need to switch to an alternative storage.

Details

Related Gerrit Patches:
mediawiki/extensions/Graph : masterCompress graph_specs page property

Event Timeline

Yurik created this task.May 31 2015, 4:29 PM
Yurik raised the priority of this task from to Needs Triage.
Yurik updated the task description. (Show Details)
Yurik added a subscriber: Yurik.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2015, 4:29 PM
Yurik triaged this task as High priority.May 31 2015, 4:32 PM
Yurik set Security to None.

Note that TemplateData also uses page_props. It uses an edit hook to prevent oversized blobs from being saved in the first place. Doesn't solve the size problem, but at least ensures data integrity.

It also compresses the JSON. See T53740 for discussion and patches. There were numerous complaints about limited length before, not a single one afterwards. :)

brion added a subscriber: brion.Jul 15 2015, 2:33 PM
brion added a comment.Jul 15 2015, 2:43 PM

Lemme do a quick pass on compression and if we do something fancier than that, great :D

brion added a comment.Jul 15 2015, 3:08 PM

I don't actually see anything *using* the page_props value after it's set... ?

brion added a comment.Jul 15 2015, 3:13 PM

Ok checked in with Yuri, it all makes sense now. ;) Graphoid fetches this via API, needs tweaking.

brion added a comment.Jul 15 2015, 3:39 PM

Ok intermediate fix could be something like:

  • gzip the graph_specs page props data
  • add API query page props extension to provide the decompressed graph specs
  • tweak graphoid to use that instead of raw page_props

Proper fix for this is to do the data transformations on the MediaWiki side and bundle everything into a data blob indexed by (a fuller) hash value, but that's farther out.

  • gzip the graph_specs page props data

This is what we do with TemplateData, FWIW.

Per discussions with Yuri and JamesF we prefer to do this the right way. :) removing the dep on the compression bit.

Change 255914 had a related patch set uploaded (by Yurik):
(WAIT) Compress graph_specs page property

https://gerrit.wikimedia.org/r/255914

Yurik added a comment.EditedDec 2 2015, 11:42 PM

In the above patch, I added the compression, since now there is an API to get the data from. Waiting for the api to ride the train first, and update the Graphoid service

Change 255914 merged by jenkins-bot:
Compress graph_specs page property

https://gerrit.wikimedia.org/r/255914

Yurik closed this task as Resolved.Dec 17 2015, 10:25 PM
Yurik claimed this task.