... as the title says! This should make the requests smaller.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | cmassaro | T319467 Re-enable @broken Map test in WikiLambda integration tests | |||
Resolved | BUG REPORT | cmassaro | T318363 Request Entity Too Large!!! | ||
Resolved | cmassaro | T319435 Consider Using a Binary Exchange Format (e.g. Apache Avro) for Communication between the Evaluator and Orchestrator |
Event Timeline
In early tests, doing this will decrease the size of the request by a factor somewhere between four and six.
> obj { Z1K1: { Z1K1: 'Z9', Z9K1: 'Z10' }, Z10K1: { Z1K1: 'Z9', Z9K1: 'Z7' }, Z10K2: { Z1K1: 'Z6', Z6K1: 'hello' } } > JSON.stringify(obj, null, 4).length; 200 # Length of a request sent as bare JSON. > let f = formatNormalForSerialization(obj); undefined # our ZObjects have to be converted to a slightly different format to make Avro happy > let buff = u.toBuffer( f ); undefined # u is a type defined by an Avro schema > buff; <Buffer 04 06 08 5a 31 4b 31 04 06 5a 31 30 0a 5a 31 30 4b 31 04 04 5a 37 0a 5a 31 30 4b 32 02 0a 68 65 6c 6c 6f 00> > buff.toString().length 36 # more than six times smaller than before! > obj.Z10K2.Z6K1 = 'lsdjf;aksjfd;lkajfdssdfsafd;lkajsfd;lkjsafdsf'; 'lsdjf;aksjfd;lkajfdssdfsafd;lkajsfd;lkjsafdsf' > f = formatNormalForSerialization(obj); { 'ztypes.Z1': { valueMap: { Z1K1: [Object], Z10K1: [Object], Z10K2: [Object] } } } > buff = u.toBuffer( f ); <Buffer 04 06 08 5a 31 4b 31 04 06 5a 31 30 0a 5a 31 30 4b 31 04 04 5a 37 0a 5a 31 30 4b 32 02 5a 6c 73 64 6a 66 3b 61 6b 73 6a 66 64 3b 6c 6b 61 6a 66 64 73 ... 26 more bytes> > JSON.stringify(obj,null,4).length 240 # Z6K1's length has increased > buff.toString().length 76 # now the savings are more like 3x
Change 838862 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):
[mediawiki/services/function-schemata@master] Add Avro schema to create a compressed binary representation of ZObjects.
Oh, wow. It's actually a much bigger difference on real data:
function call is 15127 chars long string version is 1307 chars long
function call is 3832626 chars long string version is 132311 chars long
Change 838862 merged by jenkins-bot:
[mediawiki/services/function-schemata@master] Add Avro schema to create a compressed binary representation of ZObjects.
Change 840147 had a related patch set uploaded (by Jforrester; author: Jforrester):
[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840148 had a related patch set uploaded (by Jforrester; author: Jforrester):
[mediawiki/services/function-orchestrator@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840149 had a related patch set uploaded (by Jforrester; author: Jforrester):
[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840205 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):
[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840206 had a related patch set uploaded (by Jforrester; author: Jforrester):
[mediawiki/tools/wikilambda-cli@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840205 abandoned by Jforrester:
[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)
Reason:
Change 840148 merged by jenkins-bot:
[mediawiki/services/function-orchestrator@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840149 merged by jenkins-bot:
[mediawiki/services/function-evaluator@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840206 merged by jenkins-bot:
[mediawiki/tools/wikilambda-cli@master] Update function-schemata sub-module to HEAD (d41dee3)
Change 840147 merged by jenkins-bot:
[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (d41dee3)