Page MenuHomePhabricator
Create Task
Maniphest T405130

Abandon Avro serialization; use the raw JSON request format and, optionally, compression
Closed, ResolvedPublic
Actions

Description

Description

The evalutaor request format should be sufficient for our needs. Avro is a clunky dependency and one that doesn't play well with Rust. If we find that our request sizes are unacceptable, we can use a compression algorithm like gzip or fflate.

  • Regular versions
  • Rust version
  • Migrate all tests to only use the current exchange format
  • Drop old Avro code in evaluator/orchestrator/schemata

Remove all the non-applicable tags from the "Tags" field, leave only the tags of the projects/repositories related to this task

Completion checklist

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Update function-schemata sub-module to HEAD (eb9b98ae)Genoveva Galarzamediawiki/extensions/WikiLambdamaster+1 -1
Update function-schemata sub-module to HEAD (3ad3836)Cory Massaromediawiki/extensions/WikiLambdamaster+1 -1
Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.Cory Massarooperations/deployment-chartsmaster+1 -1
Wikifunctions: Update function-evaluators from 2025-10-15-120631 to 2025-10-21-143846.Cory Massarooperations/deployment-chartsmaster+2 -2
Update function-schemata sub-module to HEAD (960dd2c)Jforrestermediawiki/extensions/WikiLambdamaster+1 -1
wikifunctions: Upgrade orchestrator from 2025-10-06-225918 to 2025-10-14-194525Jforresteroperations/deployment-chartsmaster+1 -1
wikifunctions: Upgrade evaluators from 2025-10-06-215412 to 2025-10-09-001812Jforresteroperations/deployment-chartsmaster+2 -2
wikifunctions: Upgrade orchestrator from 2025-09-25-181720 to 2025-10-06-225918Jforresteroperations/deployment-chartsmaster+1 -1
wikifunctions: Upgrade evaluators from 2025-09-30-194529 to 2025-10-06-215412Jforresteroperations/deployment-chartsmaster+2 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Eliminate everything that had to do with Avro.repos/abstract-wiki/wikifunctions/function-schemata!286apineapine-avropocalypsemain
Use exchange format instead of Avro in Rust.repos/abstract-wiki/wikifunctions/function-evaluator!425apineapine-rust-no-avromain
Add exchange format v2.1.0 for Rust.repos/abstract-wiki/wikifunctions/function-schemata!284apineapine-rust-no-avromain
Update function-schemata sub-module to HEAD (8181338)repos/abstract-wiki/wikifunctions/wikilambda-cli!92jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (8181338)repos/abstract-wiki/wikifunctions/function-evaluator!424jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (8181338)repos/abstract-wiki/wikifunctions/function-orchestrator!468jforrestersync-function-schematamain
Remove all trace of Avro from Node code.repos/abstract-wiki/wikifunctions/function-schemata!282apineapine-no-avro-nodemain
Migrate remaining tests off of Avro.repos/abstract-wiki/wikifunctions/function-evaluator!423apineapine-moarmain
Move (or copy) all executor-specific test files to a subdirectory.repos/abstract-wiki/wikifunctions/function-evaluator!422apineapine-exchange-format-testsmain
Move all the Z7-style test cases to a separate directory and combine with expectations.repos/abstract-wiki/wikifunctions/function-evaluator!421apineapine-test-changesmain
Update function-schemata sub-module to HEAD (960dd2c)repos/abstract-wiki/wikifunctions/wikilambda-cli!91jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (960dd2c)repos/abstract-wiki/wikifunctions/function-evaluator!420jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (960dd2c)repos/abstract-wiki/wikifunctions/function-orchestrator!462jforrestersync-function-schematamain
Add exchange format v2.1.0, which dispenses with Avro.repos/abstract-wiki/wikifunctions/function-schemata!281apineapine-avrolessmain
Show related patches Customize query in GitLab

Related Objects
Search...

Event Timeline

cmassaro created this task.Sep 19 2025, 10:40 PM
Jdforrester-WMF triaged this task as Medium priority.Sep 24 2025, 5:38 PM
Jdforrester-WMF added a parent task: T403669: Rust evaluator messaging protocol should handle arbitrary buffer sizes.
Jdforrester-WMF moved this task from To Triage to Engineering Backlog on the Abstract Wikipedia team board.
Jdforrester-WMF mentioned this in T402478: Rectify double/long issue with remainingTime in different Avro schemata.Sep 25 2025, 9:45 AM
cmassaro added a parent task: T405624: Fuzz testing for the Rust evaluator layer.Sep 25 2025, 4:55 PM
cmassaro claimed this task.Sep 26 2025, 11:54 AM
CodeReviewBot added a project: Patch-For-Review.Sep 30 2025, 7:41 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/281

Add exchange format v2.1.0, which dispenses with Avro.

CodeReviewBot added a comment.Oct 1 2025, 8:42 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/281

Add exchange format v2.1.0, which dispenses with Avro.

gerritbot added a comment.Oct 1 2025, 8:56 PM

Change #1192975 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (960dd2c)

https://gerrit.wikimedia.org/r/1192975

CodeReviewBot added a comment.Oct 1 2025, 8:57 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/462

Update function-schemata sub-module to HEAD (960dd2c)

CodeReviewBot added a comment.Oct 1 2025, 8:57 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/420

Update function-schemata sub-module to HEAD (960dd2c)

CodeReviewBot added a comment.Oct 1 2025, 8:57 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/91

Update function-schemata sub-module to HEAD (960dd2c)

CodeReviewBot added a comment.Oct 1 2025, 9:29 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/91

Update function-schemata sub-module to HEAD (960dd2c)

CodeReviewBot added a comment.Oct 1 2025, 9:30 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/462

Update function-schemata sub-module to HEAD (960dd2c)

CodeReviewBot added a comment.Oct 1 2025, 10:06 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/420

Update function-schemata sub-module to HEAD (960dd2c)

Jdforrester-WMF mentioned this in T403818: Generate Avro-serialized test data in Rust tests.Oct 2 2025, 4:16 PM
cmassaro mentioned this in rMSFEb9334249d2d2: Update function-schemata sub-module to HEAD (960dd2c).Oct 3 2025, 9:45 AM
cmassaro moved this task from Engineering Backlog to 26Q2 (Oct–Dec) on the Abstract Wikipedia team board.Oct 5 2025, 10:25 AM
cmassaro edited projects, added: Abstract Wikipedia team (26Q2 (Oct–Dec)); removed: Abstract Wikipedia team.
cmassaro mentioned this in T321196: For speed, don't serialise requests to the evaluator into Avro unless over a certain size threshold.Oct 5 2025, 10:28 AM
cmassaro mentioned this in T402476: Figure out how to resolve Avro Unions against the tag.
cmassaro mentioned this in T334236: Consider Also Serializing Evaluator Responses and Reentrant Calls with Avro.
cmassaro mentioned this in T324626: Come up with a way to lint .avsc (Avro) schemata files for alignment with our coding style.
cmassaro moved this task from Incoming to In Engineering on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.Oct 5 2025, 11:58 AM
Jdforrester-WMF updated the task description. (Show Details)Oct 7 2025, 5:21 PM
Jdforrester-WMF updated the task description. (Show Details)Oct 7 2025, 5:23 PM
CodeReviewBot added a comment.Oct 8 2025, 10:47 AM

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/421

Draft: Move all the Z7-style test cases to a separate directory and combine with expectations.

CodeReviewBot added a comment.Oct 8 2025, 12:01 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/422

Move (or copy) all executor-specific test files to a subdirectory.

gerritbot added a comment.Oct 8 2025, 12:53 PM

Change #1194619 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-09-30-194529 to 2025-10-06-215412

https://gerrit.wikimedia.org/r/1194619

gerritbot added a comment.Oct 8 2025, 12:53 PM

Change #1194620 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-09-25-181720 to 2025-10-06-225918

https://gerrit.wikimedia.org/r/1194620

CodeReviewBot added a comment.Oct 8 2025, 1:56 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/421

Move all the Z7-style test cases to a separate directory and combine with expectations.

Jdforrester-WMF mentioned this in rMSFE704736012749: Move all the Z7-style test cases to a separate directory and combine with….Oct 8 2025, 1:57 PM
gerritbot added a comment.Oct 8 2025, 2:03 PM

Change #1194619 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-09-30-194529 to 2025-10-06-215412

https://gerrit.wikimedia.org/r/1194619

gerritbot added a comment.Oct 8 2025, 2:09 PM

Change #1194620 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-09-25-181720 to 2025-10-06-225918

https://gerrit.wikimedia.org/r/1194620

CodeReviewBot added a comment.Oct 8 2025, 10:15 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/422

Move (or copy) all executor-specific test files to a subdirectory.

cmassaro mentioned this in rMSFEffba312f75cd: Move (or copy) all executor-specific test files to a subdirectory..Oct 8 2025, 10:16 PM
CodeReviewBot added a comment.Oct 8 2025, 11:11 PM

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/423

Migrate remaining tests off of Avro.

CodeReviewBot added a comment.Oct 8 2025, 11:12 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/282

Remove all trace of Avro from Node code.

CodeReviewBot added a comment.Oct 9 2025, 12:18 AM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/423

Migrate remaining tests off of Avro.

Jdforrester-WMF mentioned this in rMSFEc80717ea890b: Migrate remaining tests off of Avro..Oct 9 2025, 12:18 AM
gerritbot added a comment.Oct 9 2025, 1:28 PM

Change #1194936 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-06-215412 to 2025-10-09-001812

https://gerrit.wikimedia.org/r/1194936

gerritbot added a comment.Oct 9 2025, 1:31 PM

Change #1194936 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-06-215412 to 2025-10-09-001812

https://gerrit.wikimedia.org/r/1194936

CodeReviewBot added a comment.Oct 10 2025, 1:40 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/282

Remove all trace of Avro from Node code.

CodeReviewBot added a comment.Oct 14 2025, 2:35 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/468

Update function-schemata sub-module to HEAD (8181338)

CodeReviewBot added a comment.Oct 14 2025, 2:36 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/424

Update function-schemata sub-module to HEAD (8181338)

CodeReviewBot added a comment.Oct 14 2025, 2:36 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/92

Update function-schemata sub-module to HEAD (8181338)

Jdforrester-WMF updated the task description. (Show Details)Oct 14 2025, 2:37 PM
Jdforrester-WMF mentioned this in rMSFE6ae4f14742a4: Update function-schemata sub-module to HEAD (8181338).Oct 14 2025, 3:00 PM
CodeReviewBot added a comment.Oct 14 2025, 5:56 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/284

Add exchange format v2.1.0 for Rust.

CodeReviewBot added a comment.Oct 14 2025, 6:08 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/425

Draft: Use exchange format instead of Avro in Rust.

CodeReviewBot added a comment.Oct 14 2025, 7:31 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/284

Add exchange format v2.1.0 for Rust.

cmassaro mentioned this in rMSFE46c55e428103: Update function-schemata sub-module to HEAD (6baadb2).Oct 14 2025, 7:44 PM
CodeReviewBot added a comment.Oct 14 2025, 7:45 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/468

Update function-schemata sub-module to HEAD (8181338)

CodeReviewBot added a comment.Oct 14 2025, 7:45 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/92

Update function-schemata sub-module to HEAD (8181338)

CodeReviewBot added a comment.Oct 15 2025, 10:19 AM

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/286

Draft: Eliminate everything that had to do with Avro.

cmassaro updated the task description. (Show Details)Oct 15 2025, 10:35 AM
CodeReviewBot added a comment.Oct 15 2025, 12:06 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/424

Update function-schemata sub-module to HEAD (8181338)

Jdforrester-WMF mentioned this in rMSFEe0e4a0419aed: Update function-schemata sub-module to HEAD (8181338).Oct 15 2025, 12:06 PM
CodeReviewBot added a comment.Oct 15 2025, 12:18 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/286

Eliminate everything that had to do with Avro.

gerritbot added a comment.Oct 15 2025, 1:29 PM

Change #1196439 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-06-225918 to 2025-10-14-194525

https://gerrit.wikimedia.org/r/1196439

gerritbot added a comment.Oct 15 2025, 2:02 PM

Change #1196439 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-06-225918 to 2025-10-14-194525

https://gerrit.wikimedia.org/r/1196439

CodeReviewBot added a comment.Oct 15 2025, 9:40 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/425

Use exchange format instead of Avro in Rust.

Jdforrester-WMF mentioned this in rMSFE46cbc8e393f5: Use exchange format instead of Avro in Rust..Oct 15 2025, 9:41 PM
Jdforrester-WMF moved this task from In Engineering to Needs Sign-off on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.Oct 15 2025, 9:41 PM
Jdforrester-WMF subscribed.

Everything is now deployed, or is in non-deployed code, so straight to sign-off.

gerritbot added a comment.Oct 15 2025, 9:41 PM

Change #1192975 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (960dd2c)

https://gerrit.wikimedia.org/r/1192975

cmassaro closed this task as Resolved.Oct 15 2025, 9:55 PM
ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.24; 2025-10-21).Oct 15 2025, 10:00 PM
CodeReviewBot added a comment.Oct 15 2025, 10:31 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/427

Draft: Update function-schemata sub-module to HEAD (4af872b)

cmassaro mentioned this in rMSFEe2c0f6bf1bd4: Update function-schemata sub-module to HEAD (4af872b).Oct 15 2025, 10:31 PM
CodeReviewBot added a comment.Oct 15 2025, 10:31 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/469

Update function-schemata sub-module to HEAD (4af872b)

CodeReviewBot added a comment.Oct 15 2025, 10:56 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/469

Update function-schemata sub-module to HEAD (4af872b)

CodeReviewBot added a comment.Oct 16 2025, 8:45 PM

ecarg merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/427

Update function-schemata sub-module to HEAD (4af872b)

ecarg mentioned this in rMSFE17bd16f29445: Update function-schemata sub-module to HEAD (4af872b).Oct 16 2025, 8:45 PM
gerritbot added a comment.Oct 22 2025, 12:25 PM

Change #1198036 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[operations/deployment-charts@master] Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.

https://gerrit.wikimedia.org/r/1198036

gerritbot added a comment.Oct 22 2025, 12:32 PM

Change #1198039 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[operations/deployment-charts@master] Update function-evaluators from 2025-10-15-120631 to 2025-10-21-143846.

https://gerrit.wikimedia.org/r/1198039

gerritbot added a comment.Oct 22 2025, 2:07 PM

Change #1198039 merged by jenkins-bot:

[operations/deployment-charts@master] Wikifunctions: Update function-evaluators from 2025-10-15-120631 to 2025-10-21-143846.

https://gerrit.wikimedia.org/r/1198039

gerritbot added a comment.Oct 22 2025, 2:13 PM

Change #1198036 merged by jenkins-bot:

[operations/deployment-charts@master] Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.

https://gerrit.wikimedia.org/r/1198036

gerritbot added a comment.Oct 28 2025, 9:06 PM

Change #1199502 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (3ad3836)

https://gerrit.wikimedia.org/r/1199502

gerritbot added a comment.Oct 30 2025, 9:22 AM

Change #1200003 had a related patch set uploaded (by Genoveva Galarza; author: Genoveva Galarza):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (eb9b98ae)

https://gerrit.wikimedia.org/r/1200003

gerritbot added a comment.Oct 30 2025, 10:14 AM

Change #1199502 abandoned by Genoveva Galarza:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (3ad3836)

Reason:

Abandoning this in favor of 1200003

https://gerrit.wikimedia.org/r/1199502

gerritbot added a comment.Oct 30 2025, 1:17 PM

Change #1200003 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (eb9b98ae)

https://gerrit.wikimedia.org/r/1200003

CodeReviewBot added a comment.Nov 5 2025, 3:02 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/93

Update function-schemata sub-module to HEAD (eebbf96)

CodeReviewBot added a comment.Nov 5 2025, 3:47 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/93

Update function-schemata sub-module to HEAD (eebbf96)

DSantamaria added a project: Essential-Work.Dec 9 2025, 6:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits