I would love to have Quibble mentioned as a way to simplify how to run test. Although its primary case has been CI, the tool supports running test suites from a local development environment and provides a single too to run all test suites.

A lot of the devil is to figure out why a test actually fail (code or environment?) and having a way to replay just the failing one which is not trivial and varies depending on the testing framework.

(Programming note)

This session was accepted and will be scheduled.

Notes to the session leader

• Please continue to scope this session and post the session's goals and main questions into the task description.
  • If your topic is too big for one session, work with your Program Committee contact to break it down even further.
  • Session descriptions need to be completely finalized by November 1, 2019.
• Please build your session collaboratively!
  • You should consider breakout groups with report-backs, using posters / post-its to visualize thoughts and themes, or any other collaborative meeting method you like.
  • If you need to have any large group discussions they must be planned out, specific, and focused.
  • A brief summary of your session format will need to go in the associated Phabricator task.
  • Some ideas from the old WMF Team Practices Group.
• If you have any pre-session suggested reading or any specific ideas that you would like your attendees to think about in advance of your session, please state that explicitly in your session’s task.
  • Please put this at the top of your Phabricator task under the label “Pre-reading for all Participants.”

Notes to those interested in attending this session

(or those wanting to engage before the event because they are not attending)

• If the session leader is asking for feedback, please engage!
• Please do any pre-session reading that the leader would like you to do.

Why would I want to run tests locally? If I can just push to Gerrit and wait for an half an hour, I get all tests run, see code coverage etc. thanks to sonarqube.

If I do run tests locally (apart from linters, which are simple and reasonably fast), I only run phpunit tests for the specific classes I am working on to get quick feedback. Running anything else feels too slow and cumbersome. Just targeting specific class tests is already not so nice. Maybe with IDE integration it could be nicer.

In T234638#5630299, @Nikerabbit wrote:

Why would I want to run tests locally? If I can just push to Gerrit and wait for an half an hour, I get all tests run, see code coverage etc. thanks to sonarqube.

If I do run tests locally (apart from linters, which are simple and reasonably fast), I only run phpunit tests for the specific classes I am working on to get quick feedback. Running anything else feels too slow and cumbersome. Just targeting specific class tests is already not so nice. Maybe with IDE integration it could be nicer.

I agree.
This is generally my workflow.
I only run the few specific tests that I am working on locally and let Jenkins do the rest.

But of course, it is always nice to be able to run tests locally easily.
And also be able to run them locally in the same environment as CI

@kostajh and I had several exchanges and we even prepared a quick slide deck. Seems we orient ourselves toward:

• 20 minutes or so presentation of the test frameworks, recent work done to PHPUnit, Quibble 101
• break out to gather ideas for improvements
• build a list of actions to follow up on after the conference

Notes from session Etherpad:

Wikimedia Technical Conference
Atlanta, GA USA
November 12 - 15, 2019

Session Name / Topic
Running tests locally: an introduction
Session Leader: Antoine + Kosta; Facilitator: Aubrey; Scribe: Brennen
https://phabricator.wikimedia.org/T234638

Session Attendees
James F, Tyler, Alex, Elena, Lars, Jeena, TPT, Cormac, Volker, Tobi,
Daniel Z, ...

Note-taking meta:
    Likely I messed up or missed some names here.

Notes:

•  [slide] Running tests locally
•  AM: We're going to do a quick 15-20 min presentation on running tests locally, then gather feedback from people using / wanting to run tests.  Issues you might have, room for improvement, etc.
• [slide] Current state
• KH: Do you run tests locally?
  • Most people show hands
  • KH: Some people push to CI (and wait)
• [slide] Mediawiki test systems
  • We have three main roles
    • linting
    • unit/integration (mostly PHPUnit + QUnit -- which are slightly slower) -- 6 minutes to run MW/core tests
    • e2e tests -- using selenium -- mw/core has ~20tests
      • wikibase probably has 10k of these
•  AM: A lot of extensions probably don't even have tests.
• [slide] Linters
  • composer test && npm test
  • Useful in pre-commit hooks
  • Versions maintaind by CI people and consistent across MW repos
  •  When you use composer test or npm test you don't have to worry about linting -- it will just do it
• [slide] phpunit.php
  • test/phpunit/phpunit.php (yo dawg, I heard you like php)
  • Extends the maintenance class, allows passing in options
  • Has been the main thing that's existed for a timescale that only James_F can comprehend, effectively
  • modern tools that parallelize PHP are incompatible with our script that wraps around PHPUnit so it's a bit of a problem
    • KH: There's an unconf session later to discuss this
• [slide] PHPunit for unit + integration tests
  • we've been working on using /vendor/bin/phpunit rather than our homegrown test/phpunit/phpunit.php
  • Doesn't care anything about MW - just needs to know this is the class
  • Don't need a LocalSettings file, don't need a database, you can just run your tests
  • [video]
  • Can plug in PHPUnit with IDE, use debugger, generate code coverage, etc.
  • Can run test for individual functions/generate coverage for each function
  • Really nice to have this integrated with tools.
• Theme: The less magic wrapper stuff and the more modular our test runners:  The better.
• [slide] MW QUnit
  • AM: Historically a lot of JS
  • Timo wrote a page Special:JavaScriptTest which has all MW logic, will execute all JS tests in the browser
  • npm run qunit uses karma to run tests from the specialpage
  • [[Special:JavaScriptTest]]
• [slide] node QUnit
  • QUNit unit tests for js - used by some extensions / skins
  • [video demo]
  • KH: Similar to plain unit tests - much faster (than thekarma version using the special page -- no need for a browser)
  • Can do coverage summary
• [slide] webdriver.io / Selenium
  • Zeljko did an experiment here
  • npm run selenium
  • Requires chromium, chromedriver, admin (user, password) on wiki
  • Can run headless - magically in the background
  • mw/core, some extensions/skins
  • KH: There's an unconference about Selenium tests
• [slide] api-testing
  • CPT Purple made this
  • Homemade JS suite
  • e2e testing for the MW API
  • Uses for /rest.php new entrypoint
  • mediawiki/tools/api-testing.git
  • AM: The bad news is that this system is going to be running for every single MW patch you propose...
• [slide] Quibble - a test runner for MediaWiki
  • AM: This is complicated because you need a bunch of stuff
• [slide] Running tests is hard
  • Know the commands, set up MW, bring in dependencies, clean database, run install.php, tweak your LocalSettings.php
  • Local environments may have version discrepancies in PHP, Chromium + Chromedriver, NodeJS, etc.
  • AM: Usually took me 15-20 minutes just to set up a fresh enviornment just to get started...
• [slide] all-in-one command
  • Just run quibble
  • ...and a lot of things happen
  • Started at a hackathon, used by CI since 2018
  • CI system has fully migrated to it
• [slide] Install
• [slide] Environment matter
  • python3, php & composer, nodejs and npm, chromium & chromedriver, sqlite/mariadb/pgsql...
  • OR a container; i.e., docker pull docker-registry.wikimedia.org/releng/quibble-stretch-php72
  • If there's an issue in CI, you should be able to reproduce the build using this container, because it's the same one we're using.
• [slide] Examples
  • https://doc.wikimedia.org/quibble/
  • "quibble -c" executes a command on your running container with all your pre-requisites involved
• [slide] What next?
  • AM: I'm personally using quibble on my local machine.  The challenge is that if I'm working on a specific js module, I don't know how to run tests for that specific one - have to read docs every time for phpunit...

Discussion

• KH: Do you run tests locally?  Almost everyone said yes, but when / why, something you always do?
  • L: Mostly PHP - but with JavaScript in the IDE - really depend on being able to use the IDE rather than switch between terminal and IDE
  • CP: Didn't know PHPStorm could run tests like the video demo - answer: it is fairly new!
  • Tobi: Didn't write code for some time now, but the engineers I work with do - a lot of engineers are using TDD, being able to see what fails, being able to run one specific test rather than the whole suite.
    • If all this environment would come with the local development environment and would just work there, that'd be great.

• KH: How many running tests through IDE?
  • A few via IDE/PHPStorm, most using the CLI
  • James: indignant :)
  • [text editor flamewar narrowly averted]
  • Selenium tests, never run them locally since they are taking too long.
  • Tobi: One thing that happened in the past, often if you were running locally it'd pass/fail - and then you'd get a different result from CI.  I understand this might be solved by using Quibble, but I'm not sure...
  • L: That could always happen...
  • AM: There were issues because Debian changed packages, puppet updated the runner, but it's less of a concern now
• L: Is anyone running the linter locally?  phplint or jslint?
  • Most do! (great)
• tpt: I love "composer fix"! 

• KH: What is painful / confusing with our current test runners?
  • SN: some of the javascript testing libraries we're integrated with are out of date: 
    • sinon.js is outdated
    • It's not always clear what best practices are
    •  lots of ways to write test, not clear which test class you should subclass, tradeoffs of a mock - not a strong alignment around this in the Foundation
    • would love to see guidelines written 
  •  Alex: My problem isn't with the test runners - issues with Gerrit and merging (i.e., cannot merge in the current state of your repo (because they don't allow non-fast-forward merges for ops/puppet)
    • Tyler: We can make your git history uglier and it'll work
  • JF: On the library thing - we used to have the problem that CI? set the version
• KH: Do you feel like outdated libraries impacts running the tests?
  • JF: There are speed issues...  But no, mostly just a coordination problem.
  • V: Is there a possibility to have a mandate
  • JF: If somebody files a task it'll get done eventually, but if no one does
• Tyler: How many people were familiar with quibble before this talk?
  • 5-7 hands raised (mostly folks who worked on quibble [plus 2 folks] -- asserts thcipriani)
• Z: Why don't we ???
  • JF: Even the smallest test upgrade is hard, eventually it becomes an epic.
  • Z: Why don't we fix the problem that core runs everything?
  • JF: MW is a monolith.
  • L: I think we can separate it for extensions, but...
  • S: I don't dispute the need for integration, but I'd rather run unit tests as fast as possible
• Lars:  If the problem is this becomes too painful to do an upgrade - how about once per quarter we have a week of upgrading everything?
• KH: Last chance to complain about the test runners
  • James: Everyone in this room has scary stuff on your laptops -- seriously consider running in a locked-down container -- running npm install is dangerous
  • AM: when you run npm install it has access to things on your local machine, which can use your ssh keys php keys whatever 
  • [trust rant]
  • Z: Timo and others wrote a tool that'll execute tests in a local container
  • ...or just use Quibble in the container
  • KH: My painful, confusing thing - Quibble allows you to reproduce a build that failed in CI, but the process to set that up is still time-consuming, slow, a little confusing, doesn't integrate that well with existing codebase that you have open, etc.

• KH: Improvements - what do we want to see changed?
  • KH/SN: Better process to update libraries
  • Lars: Speed -- test suites that take longer than 300ms are too slow
    • JF: [heavy sigh]
  • Tobi: Whatever local environment we're setting up should include test runners!
  • Z: Documentation up to date?
    • AM: Ehhhhh - the thing is that the use case 2 years ago was reproducing a build in CI, you needed to know puppet, apt.wikimedia.org packages, shell script you had to clone, JJB, all the MW tweaking that we have to do in LocalSettings.php, figure out which command to run
      • Quibble use case was to gather that all in a single file.
      • Migrated CI to use that system.
      • You can use it as a developer, but there's barely any documentation for it, there are a bunch of obscure settings.  Not a pleasant user experience.
      • You can try it, give feedback, we'd like to fix that.
      • We need to write docs, tutorials, step-by-steps.  Blocked on time.
      • Tyler: Is there a plan to make all this easier?  How many people can reproduce tests now?
      • AM: I know WMDE people are using it now...
      • Lars: Would you say that there're at least 200% more people who can reproduce builds?
      • Tyler: Is there a roadmap to make that easier / a goal to democratize it?
      • AM: I think that's the point of this session - gathering info about how devs run stuff locally
  • CP: I'm still trying to figure out whether I should run Quibble myself.  Should I be running stuff in Quibble locally?
    • AM: Yes, you should.
    • CP: Is it quicker?
    • AM: No.  The thing is it spins up a fresh environment for you.  If you want to run Selenium tests, I just give Quibble an option.
      • You can think of Quibble as a helper.  An easier way run to tests. 
      • Adds a guarantee that you run a fresh database.
      • Can see it as an helper to not have to remember all the commands you need
    • KH: I think (one of) the things that's an improvement - Quibble could be easier to run.  And maybe you don't need to run Selenium locally.
      • The main use case I've had is reproducing CI builds.
  • Stephen: I've been away for a couple months and ... the command is difficult to remember, etc., workflow for reproducing a CI build could be improved
    • "quibble" vs "docker run etc"
    • Quibble might blow away my local changes since it checkout repositories (answer: use --skip-zuul)
    • Overall UX should be improved to make it way easier to run
    • Enhancement to run a specific test? 
  • QUESTION: Lars: When we do have a failure in CI and need to reproduce it - Gerrit gets an item in the comments, do we have a command to reproduce?
    • Tyler: I want to say there's a bajillion caveats here because of Zuul's speculative future states - your patch could be failing because you pushed your patch but didn't pull master first, or something else is about to land in master
    • JF: We're not very clear that we took your patch and rebased on master...
    • Tyler: I don't know why we don't, but we don't give people access to the speculative future states.  You can't clone from Zuul.
    • AM: There's a task for it.
    • AM: The problem is that Zuul does all the speculative future on the CI machine...  Don't know how to expose them securely.
      • ACTION: find task about opening the zuul-merger git repositories for public comsumption.
    • Z: I've never in my years here had something fail because of this...  Usually there's something different in the environment.
    • Tyler: There's a hierarchy of needs for producing CI failures...  Speculative future states probably isn't one of the most minor ones
    • Z: Nice to solve those, but I think the trivial stuff like different dependencies are more of it - 99%
  • - KH: There appears to be a documentation or awareness issue with some things - how do we approach doing that?
    • Put some notes on a mw.org page somewhere?
    • L: Notes on ... 
    • AM: https://www.mediawiki.org/wiki/Selenium/Node.js
    • JF: There's scraps of documentation around - if you go to mw vagrant page - there're some copy-and-pasteable lines, but it's hard to find
    • Some discussion: https://www.mediawiki.org/wiki/Manual:PHP_unit_testing
    • [crosstalk]
    • (AM: good news we have documentation!)
    • Jeena:  Is there any reference to this documentation in the README?
  • James: How do people feel about a regular e-mail with best practices about local development and testing?
    • (volunteer talk)
    • KH: Especially if the e-mail's linking to documentation...
    • Alex: Some people will ignore both...
    • Tyler: What you need is a blog post that you can link from an email that links to the documentation that links to the wiki that...
    • Liw: Someone to visit every developer personally
    • Z: Go in pairs and knock on people's doors
    • JF: Have you heard the good news...