Page MenuHomePhabricator
Create Task
Maniphest T228101

Pick a test runner (our own or an existing one)
Closed, ResolvedPublic
Actions

Description

Candidates:

  • Phester
  • strst
  • tavern
  • behat
  • codeception
  • cypress.io
  • RobotFramework
  • SoapUI
  • dredd

Criteria:

  • runtime
  • execution model
  • ease of running locally
  • easy of running in CI
  • test language
  • ease of editing
  • easy of migration
  • scope/purpose fit
  • stability/support
  • cost to modify/maintain
  • control over development
  • documentation
  • license model
  • license sympathy
  • recursive body matches
  • regex matches
  • variables
  • variable injection
  • global fixtures
  • JSON output
  • scan for test files
  • filter tests by tag
  • parallel execution

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledAtienoT219873 Create a suite of end-to-end API test for MediaWiki core
 Resolved• CCicalese_WMFT228101 Pick a test runner (our own or an existing one)
 ResolveddanielT222100 Decide whether creating Phester is actually worth while
 ResolveddanielT228001 Create an initial set of API integration tests using variables (part 1)
 Resolved• ClarakosiT225614 Create an initial set of basic API integration tests
 Resolved• ClarakosiT228111 API integration test: anonymous page creation and editing via API:Edit
 Resolved• ClarakosiT228113 API integration test: re-parse of dependent pages via API:Edit
 Resolved• ClarakosiT228119 API integration test: page history with edit summary, size diff
 Resolved• ClarakosiT228124 API integration test: recent changes with edit summary, size diff, etc
 Resolved• ClarakosiT228125 API integration test: renaming/moving a page (basic) via API:Move
 Resolved• ClarakosiT228126 API integration test: pre-save transform (PSR) (via API:Edit)
 Resolved• ClarakosiT228127 API integration test: template transclusion via API:Parsing wikitext
 Resolved• ClarakosiT228128 API integration test: some parser functions via API:Parsing wikitext
 Resolved• ClarakosiT228129 API integration test: some magic words via API:Parsing wikitext
 Resolved• ClarakosiT228130 API integration test: diffs via API:Compare
 Resolved• ClarakosiT228131 API integration test: fetching different kinds of links / reverse links
 Resolved• ClarakosiT228132 API integration test: listing category contents
 Resolved• ClarakosiT228133 API integration test: basic Kask tests
 Resolved• ClarakosiT228134 API integration test: basic RESTbase tests
 ResolveddanielT228323 Document test runner decision and rationale
 Resolved• ClarakosiT228324 Create an initial set of API integration tests using variables (part 2)

Event Timeline

• CCicalese_WMF created this task.Jul 15 2019, 8:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 15 2019, 8:07 PM
• CCicalese_WMF added a subtask: T222100: Decide whether creating Phester is actually worth while.Jul 15 2019, 8:08 PM
• CCicalese_WMF added a parent task: T227999: Full implementation of test runner.Jul 15 2019, 8:14 PM
• CCicalese_WMF triaged this task as Medium priority.Jul 16 2019, 3:41 AM
• CCicalese_WMF added a project: Platform Team Workboards (Epics).Jul 17 2019, 6:28 PM
• CCicalese_WMF moved this task from Epic Backlog to User Story Tasks being created on the Platform Team Workboards (Epics) board.
• WDoranWMF moved this task from User Story Tasks being created to Need User Stories on the Platform Team Workboards (Epics) board.Jul 24 2019, 7:33 PM
• WDoranWMF moved this task from Need User Stories to Identify Hardware Needs and RfCs on the Platform Team Workboards (Epics) board.Jul 25 2019, 3:38 PM
• WDoranWMF moved this task from Identify Hardware Needs and RfCs to Need User Stories on the Platform Team Workboards (Epics) board.
• WDoranWMF moved this task from Need User Stories to User Story Tasks being created on the Platform Team Workboards (Epics) board.Jul 25 2019, 3:45 PM
• WDoranWMF moved this task from User Story Tasks being created to Need User Stories on the Platform Team Workboards (Epics) board.
• WDoranWMF moved this task from API Integration Tests to mop on the Platform Engineering board.Jul 26 2019, 7:09 PM
• WDoranWMF edited projects, added Core Platform Team Initiatives (API Integration Tests); removed Platform Engineering (API Integration Tests).
• CCicalese_WMF moved this task from Need User Stories to User Story Tasks being created on the Platform Team Workboards (Epics) board.Jul 31 2019, 8:27 PM
• CCicalese_WMF moved this task from User Story Tasks being created to Ready for Engineering on the Platform Team Workboards (Epics) board.Jul 31 2019, 8:30 PM
• CCicalese_WMF moved this task from Ready for Engineering to Doing on the Platform Team Workboards (Epics) board.
• CCicalese_WMF closed subtask T222100: Decide whether creating Phester is actually worth while as Resolved.Aug 12 2019, 6:32 PM
• CCicalese_WMF closed this task as Resolved.Aug 12 2019, 6:49 PM
• CCicalese_WMF claimed this task.

Decision made: we will not build Phester. We'll use SuperTest and Mocha instead. Rationale to be documented on https://www.mediawiki.org/wiki/Core_Platform_Team/Decisions_Architecture_Research_Documentation/Using_SuperTest_Rather_Than_Creating_an_API_Integration_Test_Runner

• CCicalese_WMF removed a parent task: T227999: Full implementation of test runner.Aug 12 2019, 7:03 PM
• CCicalese_WMF mentioned this in T219879: Create a test runner for end-to-end API tests (Phester).Aug 12 2019, 7:07 PM
• CCicalese_WMF moved this task from Doing to Done! on the Platform Team Workboards (Epics) board.Oct 22 2019, 7:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits