Page MenuHomePhabricator
Create Task
Maniphest T406625

Create and use a shared set of fully-expanded Z7s for testing the evaluators, so we can assert stable performance and equivalence between Rust and non-Rust evaluators
Closed, ResolvedPublic
Actions

Description

  • Collate together.
  • Add to function-schemata
  • Wire up to regular evaluators, replacing current tests
  • Use for measurement
  • Wire up to Rust evaluator

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Use example/test files from function-schemata.mediawiki/extensions/WikiLambdamaster+36 -1 K
wikifunctions: Upgrade orchestrator from 2025-10-28-205854 to 2025-11-04-215809operations/deployment-chartsmaster+1 -1
wikifunctions: Upgrade evaluators from 2025-10-28-150053 to 2025-11-05-063501operations/deployment-chartsmaster+2 -2
Update function-schemata sub-module to HEAD (eb9b98ae)mediawiki/extensions/WikiLambdamaster+1 -1
Update function-schemata sub-module to HEAD (3ad3836)mediawiki/extensions/WikiLambdamaster+1 -1
Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.operations/deployment-chartsmaster+1 -1
Wikifunctions: Update function-evaluators from 2025-10-15-120631 to 2025-10-21-143846.operations/deployment-chartsmaster+2 -2
wikifunctions: Upgrade evaluators from 2025-10-06-215412 to 2025-10-09-001812operations/deployment-chartsmaster+2 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Update function-schemata sub-module to HEAD (eebbf96)repos/abstract-wiki/wikifunctions/wikilambda-cli!93jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (63675b5)repos/abstract-wiki/wikifunctions/function-evaluator!435apinesync-function-schematamain
Update function-schemata sub-module to HEAD (63675b5)repos/abstract-wiki/wikifunctions/function-orchestrator!479apinesync-function-schematamain
Add examples from the WikiLambda extension as system-wide tests.repos/abstract-wiki/wikifunctions/function-schemata!288apineapine-add-wikilambda-extension-examplesmain
Replace z7_style directory with full_system_tests from schemata.repos/abstract-wiki/wikifunctions/function-evaluator!429apineapine-use-schemata-z7smain
Update function-schemata sub-module to HEAD (4af872b)repos/abstract-wiki/wikifunctions/function-orchestrator!469apinesync-function-schematamain
Update function-schemata sub-module to HEAD (4af872b)repos/abstract-wiki/wikifunctions/function-evaluator!427apinesync-function-schematamain
Add a set of test cases that are useful in all Wikifunctions components.repos/abstract-wiki/wikifunctions/function-schemata!285apineapine-full-system-testsmain
Move all the Z7-style test cases to a separate directory and combine with expectations.repos/abstract-wiki/wikifunctions/function-evaluator!421apineapine-test-changesmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

Jdforrester-WMF created this task.Oct 7 2025, 5:22 PM
CodeReviewBot added a comment.Oct 8 2025, 10:47 AM

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/421

Draft: Move all the Z7-style test cases to a separate directory and combine with expectations.

cmassaro claimed this task.Oct 8 2025, 11:36 AM
CodeReviewBot added a comment.Oct 8 2025, 1:56 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/421

Move all the Z7-style test cases to a separate directory and combine with expectations.

Jdforrester-WMF mentioned this in rMSFE704736012749: Move all the Z7-style test cases to a separate directory and combine with….Oct 8 2025, 1:57 PM
gerritbot added a comment.Oct 9 2025, 1:28 PM

Change #1194936 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-06-215412 to 2025-10-09-001812

https://gerrit.wikimedia.org/r/1194936

gerritbot added a project: Patch-For-Review.Oct 9 2025, 1:28 PM
gerritbot added a comment.Oct 9 2025, 1:31 PM

Change #1194936 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-06-215412 to 2025-10-09-001812

https://gerrit.wikimedia.org/r/1194936

Jdforrester-WMF changed the task status from Open to In Progress.Oct 9 2025, 3:17 PM
Jdforrester-WMF triaged this task as Medium priority.
Jdforrester-WMF moved this task from Incoming to In Engineering on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.
Jdforrester-WMF updated the task description. (Show Details)Oct 9 2025, 3:21 PM
cmassaro added a parent task: T406936: Undertake capacity planning exercise once, on the current system.Oct 9 2025, 9:15 PM
CodeReviewBot added a comment.Oct 15 2025, 10:18 AM

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/285

Draft: Add a set of test cases that are useful in all Wikifunctions components.

CodeReviewBot added a comment.Oct 15 2025, 12:23 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/285

Add a set of test cases that are useful in all Wikifunctions components.

CodeReviewBot added a comment.Oct 15 2025, 10:31 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/427

Draft: Update function-schemata sub-module to HEAD (4af872b)

cmassaro mentioned this in rMSFEe2c0f6bf1bd4: Update function-schemata sub-module to HEAD (4af872b).Oct 15 2025, 10:31 PM
CodeReviewBot added a comment.Oct 15 2025, 10:31 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/469

Update function-schemata sub-module to HEAD (4af872b)

CodeReviewBot added a comment.Oct 15 2025, 10:56 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/469

Update function-schemata sub-module to HEAD (4af872b)

CodeReviewBot added a comment.Oct 16 2025, 8:45 PM

ecarg merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/427

Update function-schemata sub-module to HEAD (4af872b)

ecarg mentioned this in rMSFE17bd16f29445: Update function-schemata sub-module to HEAD (4af872b).Oct 16 2025, 8:45 PM
CodeReviewBot added a comment.Oct 17 2025, 11:00 AM

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/429

Replace z7_style directory with full_system_tests from schemata.

cmassaro updated the task description. (Show Details)Oct 17 2025, 9:38 PM
CodeReviewBot added a comment.Oct 17 2025, 10:01 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/429

Replace z7_style directory with full_system_tests from schemata.

DMartin-WMF mentioned this in rMSFE1005bdcbff6b: Replace z7_style directory with full_system_tests from schemata..Oct 17 2025, 10:02 PM
gerritbot added a comment.Wed, Oct 22, 12:25 PM

Change #1198036 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[operations/deployment-charts@master] Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.

https://gerrit.wikimedia.org/r/1198036

gerritbot added a comment.Wed, Oct 22, 12:32 PM

Change #1198039 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[operations/deployment-charts@master] Update function-evaluators from 2025-10-15-120631 to 2025-10-21-143846.

https://gerrit.wikimedia.org/r/1198039

gerritbot added a comment.Wed, Oct 22, 2:06 PM

Change #1198039 merged by jenkins-bot:

[operations/deployment-charts@master] Wikifunctions: Update function-evaluators from 2025-10-15-120631 to 2025-10-21-143846.

https://gerrit.wikimedia.org/r/1198039

gerritbot added a comment.Wed, Oct 22, 2:13 PM

Change #1198036 merged by jenkins-bot:

[operations/deployment-charts@master] Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.

https://gerrit.wikimedia.org/r/1198036

CodeReviewBot added a comment.Mon, Oct 27, 9:51 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/288

Add examples from the WikiLambda extension as system-wide tests.

CodeReviewBot added a comment.Tue, Oct 28, 3:20 AM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/288

Add examples from the WikiLambda extension as system-wide tests.

gerritbot added a comment.Tue, Oct 28, 9:06 PM

Change #1199502 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (3ad3836)

https://gerrit.wikimedia.org/r/1199502

cmassaro updated the task description. (Show Details)Wed, Oct 29, 2:30 PM

We actually can't wire these up to the Rust evaluator until we write the exchange format generation code for Rust. We won't do *that* until we start working on the Rust orchestrator. I've stricken that requirement from the task.

cmassaro updated the task description. (Show Details)Wed, Oct 29, 2:31 PM
cmassaro updated the task description. (Show Details)
cmassaro moved this task from In Engineering to Needs Sign-off on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.
cmassaro closed this task as Resolved.Wed, Oct 29, 2:38 PM
gerritbot added a comment.Thu, Oct 30, 9:22 AM

Change #1200003 had a related patch set uploaded (by Genoveva Galarza; author: Genoveva Galarza):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (eb9b98ae)

https://gerrit.wikimedia.org/r/1200003

gerritbot added a comment.Thu, Oct 30, 10:14 AM

Change #1199502 abandoned by Genoveva Galarza:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (3ad3836)

Reason:

Abandoning this in favor of 1200003

https://gerrit.wikimedia.org/r/1199502

gerritbot added a comment.Thu, Oct 30, 1:17 PM

Change #1200003 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (eb9b98ae)

https://gerrit.wikimedia.org/r/1200003

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.1; 2025-11-05).Thu, Oct 30, 2:00 PM
CodeReviewBot added a comment.Mon, Nov 3, 8:25 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/479

Update function-schemata sub-module to HEAD (63675b5)

CodeReviewBot added a comment.Mon, Nov 3, 8:26 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/435

Update function-schemata sub-module to HEAD (63675b5)

cmassaro mentioned this in rMSFE2367d94e1fff: Update function-schemata sub-module to HEAD (63675b5).Mon, Nov 3, 8:28 PM
CodeReviewBot added a comment.Mon, Nov 3, 10:00 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/479

Update function-schemata sub-module to HEAD (63675b5)

DMartin-WMF mentioned this in rMSFEadb0758f7170: Update function-schemata sub-module to HEAD (63675b5).Mon, Nov 3, 10:09 PM
CodeReviewBot added a comment.Tue, Nov 4, 5:47 AM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/435

Update function-schemata sub-module to HEAD (63675b5)

DMartin-WMF mentioned this in rMSFEac087975e479: Update function-schemata sub-module to HEAD (63675b5).Tue, Nov 4, 5:49 AM
gerritbot added a comment.Wed, Nov 5, 2:25 PM

Change #1202148 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-28-150053 to 2025-11-05-063501

https://gerrit.wikimedia.org/r/1202148

gerritbot added a comment.Wed, Nov 5, 2:25 PM

Change #1202149 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-28-205854 to 2025-11-04-215809

https://gerrit.wikimedia.org/r/1202149

CodeReviewBot added a comment.Wed, Nov 5, 3:02 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/93

Update function-schemata sub-module to HEAD (eebbf96)

gerritbot added a comment.Wed, Nov 5, 3:05 PM

Change #1202148 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-28-150053 to 2025-11-05-063501

https://gerrit.wikimedia.org/r/1202148

gerritbot added a comment.Wed, Nov 5, 3:12 PM

Change #1202149 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-28-205854 to 2025-11-04-215809

https://gerrit.wikimedia.org/r/1202149

CodeReviewBot added a comment.Wed, Nov 5, 3:47 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/93

Update function-schemata sub-module to HEAD (eebbf96)

gerritbot added a comment.Wed, Nov 5, 8:10 PM

Change #1199080 had a related patch set uploaded (by Jforrester; author: Cory Massaro):

[mediawiki/extensions/WikiLambda@master] Use example/test files from function-schemata.

https://gerrit.wikimedia.org/r/1199080

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits