Page MenuHomePhabricator

Page_props allows max of 64KB values, forcing invalid graph definitions JSON
Closed, ResolvedPublic

Description

Very large graph definitions, totaling over 64KB, don't fit into page_props DB table, and thus end up trimmed / invalid JSON. Need to switch to an alternative storage.

Event Timeline

Yurik raised the priority of this task from to Needs Triage.
Yurik updated the task description. (Show Details)
Yurik added a subscriber: Yurik.
Yurik triaged this task as High priority.May 31 2015, 4:32 PM
Yurik set Security to None.

Note that TemplateData also uses page_props. It uses an edit hook to prevent oversized blobs from being saved in the first place. Doesn't solve the size problem, but at least ensures data integrity.

It also compresses the JSON. See T53740 for discussion and patches. There were numerous complaints about limited length before, not a single one afterwards. :)

Lemme do a quick pass on compression and if we do something fancier than that, great :D

I don't actually see anything *using* the page_props value after it's set... ?

Ok checked in with Yuri, it all makes sense now. ;) Graphoid fetches this via API, needs tweaking.

Ok intermediate fix could be something like:

  • gzip the graph_specs page props data
  • add API query page props extension to provide the decompressed graph specs
  • tweak graphoid to use that instead of raw page_props

Proper fix for this is to do the data transformations on the MediaWiki side and bundle everything into a data blob indexed by (a fuller) hash value, but that's farther out.

  • gzip the graph_specs page props data

This is what we do with TemplateData, FWIW.

Per discussions with Yuri and JamesF we prefer to do this the right way. :) removing the dep on the compression bit.

Change 255914 had a related patch set uploaded (by Yurik):
(WAIT) Compress graph_specs page property

https://gerrit.wikimedia.org/r/255914

In the above patch, I added the compression, since now there is an API to get the data from. Waiting for the api to ride the train first, and update the Graphoid service

Change 255914 merged by jenkins-bot:
Compress graph_specs page property

https://gerrit.wikimedia.org/r/255914

Yurik claimed this task.