Page MenuHomePhabricator

templatedata does not work with PostgreSQL (JSON data is not stored correctly)
Closed, ResolvedPublic

Description

Steps to reproduce:

  1. Set up a MediaWiki instance that uses PostgreSQL
  2. Set up TemplateData
  3. Add templatedata to a template and save (or do a null edit to a page with existing templatedata)

Expected results:
VisualError should show information about that template.

Actual results:
VisualEditor is unable to show any information about the template. Looking at the API requests in the browser console reveals that this is because the templatedata API fails with the following error:

Code: templatedata-corrupt
Message: templatedata contains invalid data: Syntax error in JSON.

So I looked at the database and indeed, pp_value values for templatedata look really wrong...

mediawiki@localhost:wiki> select json_agg(t) from (select * from page_props where pp_propname='templatedata') t
+-----------------------------------------------------------------------------------------+
| json_agg                                                                                |
|-----------------------------------------------------------------------------------------|
| [{"pp_page":235,"pp_propname":"templatedata","pp_value":"\u001f?\b","pp_sortkey":null}] |
+-----------------------------------------------------------------------------------------+

It looks like the TemplateData extension stores the templatedata JSON as a gzipped string. However the pp_value column type is text, which means the binary data is getting truncated at the first null byte.

For some reason, changing the column type to bytea manually (with alter table page_props alter column pp_value type bytea using (pp_value::bytea)) does not fix the issue. The data is still visibly getting truncated at the same place. Any help would be appreciated.

Event Timeline

leoetlino created this task.Sep 7 2018, 9:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2018, 9:21 PM
Restricted Application added a project: VisualEditor. · View Herald TranscriptSep 7 2018, 11:05 PM
Deskana triaged this task as Low priority.Sep 11 2018, 6:55 PM
Deskana moved this task from To Triage to Freezer on the VisualEditor board.

Change 521193 had a related patch set uploaded (by saper; owner: saper):
[mediawiki/extensions/TemplateData@master] Do not always store template data compressed

https://gerrit.wikimedia.org/r/521193

saper claimed this task.Jul 8 2019, 1:51 AM

Is there a column type that does support arbitrary blobs? If so, seems like we should instead alter the schema in core to use that. There is no need for this to be "text". Afaik TemplateData isn't the only extension that stores binary here, and given the precedent of PHP to treat strings as binary, I think this is an expected behaviour of the interface provided by the Parser. Not something an individual extension should handle.

I don't think we should be changing core schema to fix this little thing :( but I'll think about the JSON abstraction in core that can be used by other parts of the code.

saper moved this task from Backlog to Doing on the TemplateData board.Sep 30 2019, 12:45 PM

Change 521193 merged by jenkins-bot:
[mediawiki/extensions/TemplateData@master] Do not always store template data compressed

https://gerrit.wikimedia.org/r/521193

matmarex closed this task as Resolved.Oct 1 2019, 12:00 AM
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptOct 1 2019, 12:00 AM