Maniphest T376053

Reduce Test Flake in Evaluator CI
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	cmassaro
	Sep 30 2024, 5:21 PM

Tags

Referenced Files

None

Subscribers

Description

Description

A few tests are still flaky. The main sources of flake are

test-load, which fails because the runtime expectations are too stringent;
lost artifacts due to the long wasmtime compile step, which usually manifests in javascript - test debug

Desired behavior/Acceptance criteria (returned value, expected error, performance expectations, etc.)

the tests are not flaky anymore

Completion checklist

Before closing this task, review one by one the checklist available here: https://www.mediawiki.org/wiki/Abstract_Wikipedia_team/Definition_of_Done#Back-end_Task/Bug_completion_checklist

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wikifunctions: Upgrade evaluators from 2024-09-24-221243 to 2024-10-08-175510	operations/deployment-charts	master	+2 -2

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Rework the CI pipeline.	repos/abstract-wiki/wikifunctions/function-evaluator!262	apine	apine-base-images	main
	Loosen expectations in the load test to reduce flake.	repos/abstract-wiki/wikifunctions/function-evaluator!260	apine	apine-tolerance	main

Customize query in GitLab

Related Objects

Mentioned In: rMSFEe4559a674369: Rework the CI pipeline.
rMSFE93f70bb8e2af: Loosen expectations in the load test to reduce flake.

Event Timeline

cmassaro created this task.Sep 30 2024, 5:21 PM

cmassaro claimed this task.

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/260

Draft: Loosen expectations in the load test to reduce flake.

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/260

Loosen expectations in the load test to reduce flake.

• Sharvaniharan triaged this task as High priority.Oct 2 2024, 4:35 PM

• Sharvaniharan lowered the priority of this task from High to Medium.

• Sharvaniharan moved this task from To Triage to 25Q2 (Oct–Dec) on the Abstract Wikipedia team board.

• Sharvaniharan edited projects, added Abstract Wikipedia team (25Q2 (Oct–Dec)); removed Abstract Wikipedia team.

• Sharvaniharan moved this task from Incoming to Ready, Features on the Abstract Wikipedia team (25Q2 (Oct–Dec)) board.Oct 2 2024, 4:40 PM

apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/262

Draft: Build and compile wasmedge stuff a single time.

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/262

Rework the CI pipeline.

Change #1078950 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-09-24-221243 to 2024-10-08-175510

https://gerrit.wikimedia.org/r/1078950

gerritbot added a project: Patch-For-Review.Oct 9 2024, 2:03 PM

Jdforrester-WMF moved this task from Ready, Features to Needs Sign-off on the Abstract Wikipedia team (25Q2 (Oct–Dec)) board.Oct 9 2024, 2:13 PM

Jdforrester-WMF added a project: Abstract Wikipedia Fix-It tasks.

Change #1078950 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-09-24-221243 to 2024-10-08-175510

https://gerrit.wikimedia.org/r/1078950

Jdforrester-WMF closed this task as Resolved.Nov 1 2024, 7:58 PM

cmassaro mentioned this in rMSFE93f70bb8e2af: Loosen expectations in the load test to reduce flake..Oct 3 2025, 9:42 AM

Jdforrester-WMF mentioned this in rMSFEe4559a674369: Rework the CI pipeline..Oct 3 2025, 9:42 AM