Copying @hashar's comment on Trello:
Sorry for the rather long delay. To run tests with Parsoid as a backend, we would need a job that:
- spawn a Parsoid daemon like we do for some Parsoid integration tests
- setup MediaWiki behind a web server which we do for qunit tests
The mwext-Flow-testextension job execute the PHPUnit test suite which combines unit tests for the Flow function as well as integration tests against mediawiki/core.
It is totally doable to spawn a Parsoid daemon and run whatever tests you want. That being said, I would prefer to avoid rushing a solution that would match just the Flow extension in favor of a system that would work for other use cases (such as VisualEditor).
During the summer, I worked on a set of utilities that would let us clone core + vendor + extensions and run them all together with the appropriate patches/branches checked out. That will yield a state of code to be tested and is subject of an RFC: https://www.mediawiki.org/wiki/RFC/Extensions_continuous_integration
Once we have that stack of code available, everything is loaded via include() and we will run: the PHPUnit test suite the qunit page
That will supersede all the testextension and extension-qunit jobs in favor of a two jobs that are used by all repositories.
We can come up with a third job that would have a bit more backend configured such as ElasticSearch or Parsoid. But that will take a bit more time since I want the other two jobs to be setup first.
As I understand, running Flow tests with Parsoid is more of a nice to have feature than a breaker and I guess it can wait a bit. In short: yes, but a bit later and on a wider scale :-)
I am subscribed to this Trello card so we can keep the discussion here 8-)
So i know that doing this "correct" is of big benefit, but could we have most of the benefits now just by setting Flow config to point to already existing parsoid in labs? Without having to install or boot a parsoid instance the changes required become trivial, as long as the jenkins runners have network access to the labs machines.
@hashar: How far of do you think those proposed ideas are?
Would it make sense to contact an existing Parsoid, until we can properly clone & run it in isolation?
We're not in a huge rush, but it would be nice to have our tests run with Parsoid.
Possible implementation suggestion, copied from Trello:
http://www.mediawiki.org/wiki/Parsoid/API says "A production version of Parsoid on the Wikimedia cluster can be accessed at http://parsoid-lb.eqiad.wikimedia.org/", so I guess that one would be good to use.
I'm no Jenkins expert, but I think we should add a build step "Execute shell" with the following command on https://integration.wikimedia.org/ci/view/Extensions/job/mwext-Flow-testextension/configure - this would override the default Parsoid settings:
# Load context (namely: MW_INSTALL_PATH) . "/srv/deployment/integration/slave-scripts/bin/mw-set-env.sh" # Configure Parsoid echo -e \ "<?php\n\$wgFlowParsoidURL = 'http://parsoid-lb.eqiad.wikimedia.org/';\n"\ "\$wgFlowParsoidPrefix = 'enwiki';\n"\ "\$wgFlowParsoidTimeout = 100;\n" >> "$MW_INSTALL_PATH/LocalSettings.php"