VisualEditor: Compress POST data in the client
Closed, ResolvedPublic

Description

Until Parsoid removes metadata, large articles are going to consist of several megabytes of data (en:Barack_Obama = 3.4MiB). As browsers doesn't provide any mechanism for compressing post data, this is takes about 20s to upload on a typical 1Mbps up ADSL line.

Using a JS implementation of deflate, we could achieve 80-90% compression in a few hundred ms on a decent machine: http://jsperf.com/js-deflate

A couple of considerations with performing such a complex calculation in pure JS:

  • Really bad JS engines or slow devices (old browsers, IE, mobile) may offer to little to overall speed benefit. We may want to detect these cases by user agent or performance profiling.
  • The compression function will be synchronous and lock browser interaction. On slower machines this may give the appearance of crashing, or with memory leaks may actually crash. If this proves to be a significant problem we could look into encoding in chunks of 100k at a time so we could at least provide progress, and maybe improve memory usage (at the cost of overall compression).

Version: unspecified
Severity: enhancement
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=64171

bzimport set Reference to bz66914.
Esanders created this task.Via LegacyJun 21 2014, 1:40 PM
matmarex added a comment.Via ConduitJun 21 2014, 1:42 PM

Supporting section editing (bug 48429), and submitting only modified section to the server, might be a better way to increase performance.

gerritbot added a comment.Via ConduitJun 24 2014, 12:55 PM

Change 141678 had a related patch set uploaded by Esanders:
Compress HTML data with deflate before POSTing

https://gerrit.wikimedia.org/r/141678

Esanders added a comment.Via ConduitJun 24 2014, 3:32 PM

There are a number of things we can do to avoid this heavy payload:

  • Section editing
  • Separating Parsoid metadata
  • Sending differential linmod data using a server-side converter

None of these are ready or close to being ready yet, and 20s to save a page is fairly significant problem in the present that compression can mitigate.

matmarex added a comment.Via ConduitJun 25 2014, 10:39 PM
  • Bug 59659 has been marked as a duplicate of this bug. ***
Mattflaschen added a comment.Via ConduitJun 27 2014, 10:42 PM

(In reply to Ed Sanders from comment #0)

  • The compression function will be synchronous and lock browser interaction.

You could look at doing it asynchronously in the background, with a Web Worker (i.e. thread) (https://developer.mozilla.org/en-US/docs/Web/API/Worker), in supported browsers (IE 10+, and Firefox/Chrome/Safari/Opera going back pretty far).

gerritbot added a comment.Via ConduitJun 30 2014, 11:43 PM

Change 141678 merged by jenkins-bot:
Compress HTML data with deflate before POSTing

https://gerrit.wikimedia.org/r/141678

Mattflaschen removed a subscriber: Mattflaschen.Via WebDec 3 2014, 5:20 AM

Add Comment

Column Prototype
This is a very early prototype of a persistent column. It is not expected to work yet, and leaving it open will activate other new features which will break things. Press "\" (backslash) on your keyboard to close it now.