Page MenuHomePhabricator
Create Task
Maniphest T368002

Test Wikifunctions Services with 'Catalyst'
Closed, ResolvedPublic
Actions

Description

Background

We currently do not have a dependable way to test our services – to deploy patches and test them. This prevents us from shipping quality updates consistently.
It will be a good investment of our efforts to hook up the next evolution of the testing platform, Catalyst, to WikiLambda and see if it works.
This will enable us to test our services as well. We would be an early adopter for the QTE team, and the bet is long-term stability and dependability in our testing infrastructure. Catalyst will provide the necessary testing capabilities.

Approach

  • Use the Catalyst platform to deploy on-demand Web-accessible instances of the Wikifunctions stack (MediaWiki + WikiLambda, orchestrator, and evaluator), with the ability to use specified merged versions of the services and specified merged or unmerged patches of WikiLambda
  • Trigger the creation of the above stack as part of the GitLab CI / pre-merge development experience for the function-orchestrator

Acceptance Criteria ("work is done")

  • Milestone 0: Use the Catalyst platform to deploy on-demand Web-accessible instances of the Wikifunctions stack
    • Full stack is MediaWiki + WikiLambda, orchestrator, and evaluator
    • Can specify merged versions of the services
    • Can specify merged patches of WikiLambda
    • Can specify unmerged patches of WikiLambda
    • At the end of of milestone 0, can make an API call to manually create an instance for testing unmerged patches
  • Milestone 1: Trigger the creation of the above stack as part of the GitLab CI / pre-merge development experience for the function-orchestrator
  • Stretch Goals (optional / Not expected to be done in Q2 ) -> this scope is now in separate ticket, T394557
    • Milestone 1a: Integration into the GitLab pre-merge workflow for function-evaluator patches.
    • Milestone 2: Integration into the gerrit pre-merge workflow for WikiLambda patches.

Success Metrics ("work has impact")

  • Can deploy WikiFunctions ad hoc with a patch to orchestrator or evaluator
  • Use of Catalyst is integrated into the GitLab CI pipeline for the orchestrator.
  • Team can now do the manual testing using Catalyst that was previously using DUCT (extension testing)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wikifunctions: Update orchestrator from 2025-04-08-183631 to 2025-04-16-192052operations/deployment-chartsmaster+1 -1
Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job votingintegration/configmaster+6 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Sharvaniharan created this task.Jun 19 2024, 10:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 19 2024, 10:51 PM
• Sharvaniharan renamed this task from [25Q1] Testing Wikifunctions Services with Catalyst to [25Q1] Test Wikifunctions Services with Catalyst.Jun 19 2024, 10:53 PM
• Sharvaniharan renamed this task from [25Q1] Test Wikifunctions Services with Catalyst to [25Q1] Test Wikifunctions Services with 'Catalyst'.
• Sharvaniharan updated the task description. (Show Details)
• Sharvaniharan triaged this task as Low priority.Jul 3 2024, 4:34 PM
• Sharvaniharan assigned this task to • Mcastro.Jul 8 2024, 3:54 PM
Jdforrester-WMF claimed this task.Aug 19 2024, 7:45 PM
Jdforrester-WMF added a subscriber: • Mcastro.
vaughnwalters removed a subscriber: • Mcastro.Sep 25 2024, 4:03 PM
vaughnwalters subscribed.
Jdforrester-WMF renamed this task from [25Q1] Test Wikifunctions Services with 'Catalyst' to [25Q2] Test Wikifunctions Services with 'Catalyst'.Sep 25 2024, 7:41 PM
Jdforrester-WMF removed Jdforrester-WMF as the assignee of this task.
Jdforrester-WMF moved this task from 25Q1 (Jul–Sep) to 25Q2 (Oct–Dec) on the Abstract Wikipedia team board.
Jdforrester-WMF edited projects, added Abstract Wikipedia team (25Q2 (Oct–Dec)); removed Abstract Wikipedia team (25Q1 (Jul–Sep)).
Jdforrester-WMF moved this task from Incoming to Epics for this Quarter on the Abstract Wikipedia team (25Q2 (Oct–Dec)) board.
Jdforrester-WMF subscribed.
vaughnwalters updated the task description. (Show Details)Sep 26 2024, 5:18 PM
vaughnwalters added a subscriber: Etonkovidova.

After conversation with @Etonkovidova, removed the Acceptance Criteria/Success Metrics to expand test coverage by 10%. Can add new selenium tests as needed, but want to get the current ones working and stable with Catalyst first.

• Sharvaniharan added a project: Epic.Oct 4 2024, 10:30 PM
Jdforrester-WMF added a subtask: T374864: Catalyst API can deploy Wikifunctions.Oct 8 2024, 1:55 PM
Jdforrester-WMF updated the task description. (Show Details)Oct 8 2024, 2:12 PM
vaughnwalters claimed this task.Oct 8 2024, 11:08 PM
vaughnwalters updated the task description. (Show Details)
vaughnwalters added a comment.Oct 8 2024, 11:15 PM

Changed the task description based on the Q2 PRD: Testing WF Services with Catalyst.

vaughnwalters updated the task description. (Show Details)Oct 9 2024, 7:08 AM
jnuche subscribed.Oct 9 2024, 1:52 PM
Jdforrester-WMF updated the task description. (Show Details)Oct 9 2024, 2:41 PM
Jdforrester-WMF updated the task description. (Show Details)
Jdforrester-WMF added a subtask: T375689: Remove WikiLambda from list of included repos on Patch demo .Oct 15 2024, 7:35 PM
vaughnwalters updated the task description. (Show Details)Oct 16 2024, 8:14 PM
vaughnwalters updated the task description. (Show Details)Oct 16 2024, 8:24 PM
Maryann-Onyinye subscribed.Oct 28 2024, 12:17 PM
thcipriani changed the status of subtask T374864: Catalyst API can deploy Wikifunctions from Open to Stalled.Nov 4 2024, 5:43 PM
jnuche closed subtask T374864: Catalyst API can deploy Wikifunctions as Resolved.Nov 7 2024, 12:51 PM
vaughnwalters added a subtask: T379574: Catalyst API: Verify WikiFunctions integration with an AW engineer.Nov 20 2024, 5:28 PM
jnuche closed subtask T379574: Catalyst API: Verify WikiFunctions integration with an AW engineer as Resolved.Nov 27 2024, 11:34 AM
vaughnwalters added a subtask: T381359: Prototype using catalyst to deploy function-orchestrator patches ("milestone 1").Dec 5 2024, 10:34 PM
Jdforrester-WMF updated the task description. (Show Details)Dec 18 2024, 5:48 PM
Jdforrester-WMF closed subtask T375689: Remove WikiLambda from list of included repos on Patch demo as Resolved.Dec 19 2024, 3:38 PM
Jdforrester-WMF renamed this task from [25Q2] Test Wikifunctions Services with 'Catalyst' to Test Wikifunctions Services with 'Catalyst'.Jan 15 2025, 8:15 PM
Jdforrester-WMF edited projects, added Abstract Wikipedia team (25Q3 (Jan–Mar)); removed Epic, Abstract Wikipedia team (25Q2 (Oct–Dec)).
Jdforrester-WMF moved this task from Incoming to Essential work on the Abstract Wikipedia team (25Q3 (Jan–Mar)) board.
jnuche closed subtask T381359: Prototype using catalyst to deploy function-orchestrator patches ("milestone 1") as Resolved.Mar 4 2025, 4:06 PM
Jdforrester-WMF edited projects, added Abstract Wikipedia team (25Q4 (Apr–Jun)); removed Abstract Wikipedia team (25Q3 (Jan–Mar)).Apr 8 2025, 8:19 PM

Moving to the Q4 board as this is clearly still in-progress.

Jdforrester-WMF added a project: Essential-Work.Apr 9 2025, 5:26 PM
Jdforrester-WMF moved this task from Incoming to Ready on the Abstract Wikipedia team (25Q4 (Apr–Jun)) board.
gerritbot added a comment.Apr 10 2025, 1:36 PM

Change #1135722 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting

https://gerrit.wikimedia.org/r/1135722

gerritbot added a project: Patch-For-Review.Apr 10 2025, 1:36 PM
vaughnwalters added a subtask: T391733: Run all selenium tests .Apr 12 2025, 12:02 AM
vaughnwalters updated the task description. (Show Details)
vaughnwalters updated the task description. (Show Details)
DSantamaria changed the status of subtask T391733: Run all selenium tests from Open to In Progress.Apr 14 2025, 2:01 PM
Jdforrester-WMF moved this task from Ready to In Engineering on the Abstract Wikipedia team (25Q4 (Apr–Jun)) board.Apr 15 2025, 9:15 PM
gerritbot added a comment.Apr 15 2025, 10:02 PM

Change #1135722 merged by jenkins-bot:

[integration/config@master] Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting

https://gerrit.wikimedia.org/r/1135722

Stashbot added a comment.Apr 15 2025, 10:02 PM

Mentioned in SAL (#wikimedia-releng) [2025-04-15T22:02:34Z] <James_F> Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting, for T368002

Jdforrester-WMF updated the task description. (Show Details)Apr 15 2025, 10:03 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 15 2025, 10:31 PM
DSantamaria changed the task status from Open to In Progress.Apr 16 2025, 1:01 PM
gerritbot added a comment.Apr 16 2025, 9:06 PM

Change #1137076 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-04-08-183631 to 2025-04-16-192052

https://gerrit.wikimedia.org/r/1137076

gerritbot added a project: Patch-For-Review.Apr 16 2025, 9:06 PM
gerritbot added a comment.Apr 16 2025, 9:11 PM

Change #1137076 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-04-08-183631 to 2025-04-16-192052

https://gerrit.wikimedia.org/r/1137076

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2025, 9:32 PM
Jdforrester-WMF closed subtask T391733: Run all selenium tests as Resolved.Apr 29 2025, 1:05 PM
vaughnwalters updated the task description. (Show Details)May 16 2025, 9:51 PM
vaughnwalters updated the task description. (Show Details)EditedMay 16 2025, 9:56 PM

All original AC is met with the exception of stretch goal Milestone 1a: Integration into the GitLab pre-merge workflow for function-evaluator patches. which is now contained on a new ticket T394557

Also adding a note here about this related ticket to make the catalyst build and selenium tests blocking in the orchestrator gitlab pipeline T394558

vaughnwalters closed this task as Resolved.May 16 2025, 10:11 PM
In T368002#10831626, @vaughnwalters wrote:

All original AC is met with the exception of stretch goal Milestone 1a: Integration into the GitLab pre-merge workflow for function-evaluator patches. which is now contained on a new ticket T394557

Also adding a note here about this related ticket to make the catalyst build and selenium tests blocking in the orchestrator gitlab pipeline T394558

And because the remaining scope of this work is contained in another ticket to work on stretch goal 1A, I am marking this ticket as resolved. Nice work yall!! 🎉 🎉 🎉

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits