Page MenuHomePhabricator

Consider Using a Binary Exchange Format (e.g. Apache Avro) for Communication between the Evaluator and Orchestrator
Closed, ResolvedPublic

Description

... as the title says! This should make the requests smaller.

Event Timeline

In early tests, doing this will decrease the size of the request by a factor somewhere between four and six.

> obj
{
  Z1K1: { Z1K1: 'Z9', Z9K1: 'Z10' },
  Z10K1: { Z1K1: 'Z9', Z9K1: 'Z7' },
  Z10K2: { Z1K1: 'Z6', Z6K1: 'hello' }
}
> JSON.stringify(obj, null, 4).length;
200  # Length of a request sent as bare JSON.
> let f = formatNormalForSerialization(obj);
undefined  # our ZObjects have to be converted to a slightly different format to make Avro happy
> let buff = u.toBuffer( f );
undefined  # u is a type defined by an Avro schema
> buff;
<Buffer 04 06 08 5a 31 4b 31 04 06 5a 31 30 0a 5a 31 30 4b 31 04 04 5a 37 0a 5a 31 30 4b 32 02 0a 68 65 6c 6c 6f 00>
> buff.toString().length
36  # more than six times smaller than before!
> obj.Z10K2.Z6K1 = 'lsdjf;aksjfd;lkajfdssdfsafd;lkajsfd;lkjsafdsf';
'lsdjf;aksjfd;lkajfdssdfsafd;lkajsfd;lkjsafdsf'
> f = formatNormalForSerialization(obj);
{
  'ztypes.Z1': { valueMap: { Z1K1: [Object], Z10K1: [Object], Z10K2: [Object] } }
}
> buff = u.toBuffer( f );
<Buffer 04 06 08 5a 31 4b 31 04 06 5a 31 30 0a 5a 31 30 4b 31 04 04 5a 37 0a 5a 31 30 4b 32 02 5a 6c 73 64 6a 66 3b 61 6b 73 6a 66 64 3b 6c 6b 61 6a 66 64 73 ... 26 more bytes>
> JSON.stringify(obj,null,4).length
240  # Z6K1's length has increased
> buff.toString().length
76  # now the savings are more like 3x

Change 838862 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[mediawiki/services/function-schemata@master] Add Avro schema to create a compressed binary representation of ZObjects.

https://gerrit.wikimedia.org/r/838862

Oh, wow. It's actually a much bigger difference on real data:

function call is 15127 chars long
string version is 1307 chars long
function call is 3832626 chars long
string version is 132311 chars long

Change 838862 merged by jenkins-bot:

[mediawiki/services/function-schemata@master] Add Avro schema to create a compressed binary representation of ZObjects.

https://gerrit.wikimedia.org/r/838862

Change 840147 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840147

Change 840148 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/services/function-orchestrator@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840148

Change 840149 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840149

Change 840205 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840205

Change 840206 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/tools/wikilambda-cli@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840206

Change 840205 abandoned by Jforrester:

[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)

Reason:

https://gerrit.wikimedia.org/r/840205

Change 840148 merged by jenkins-bot:

[mediawiki/services/function-orchestrator@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840148

Change 840149 merged by jenkins-bot:

[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840149

Change 840206 merged by jenkins-bot:

[mediawiki/tools/wikilambda-cli@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840206

Change 840147 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (d41dee3)

https://gerrit.wikimedia.org/r/840147