Just to make sure this isn't lost, here is a list of potential candidates that @Fjalapeno sent me:

• https://www.cypress.io/
• https://robotframework.org/
• https://www.soapui.org/open-source.html
• https://github.com/apiaryio/dredd

Very VERY brief assessment of the tools suggested by corey:

• https://www.cypress.io/ based on node.js, tests are written in node.js; seems to want to avoid the kind of fixture creation described in the phester spec.
• https://robotframework.org/ written in python. Cucumber-ish tests
• https://www.soapui.org/open-source.html A drag-and drop UI approach to test specification
• https://github.com/apiaryio/dredd Written in node.js, tests are YAML files defining APIs. Mature

In terms of functionality, dredd seems to be a good fit at a glance. Requiring node.js for running tests doesn't seem ideal, but will probably become a lot less annoying with better containerization of the development and testing environment. If dredd was written in python or php, I'd probably go for it.

This week I was tasked with testing Dredd and I wanted to provide a full summary since we’ll be making the final decision soon.

Dredd accepts two different file types, OpenAPI and API Blueprint. OpenAPI has strict guidelines against duplicating endpoints and methods so it’s not surprising that Dredd errors out when you attempt to bypass those guidelines. This is not feasible for the Action API as there is only one endpoint. Even if the Action API was a REST API it would still limit the tests to target each HTTP method only once.

The other option is API Blueprint. I’ve wrestled with the specification in the last couple of days (mainly because it has VERY limited documentation and low adoption). Unlike OpenAPI/Dredd, API Blueprint doesn’t error out when you provide it duplicate endpoints or methods but it does provide a warning. Also, note that although the API Blueprint seems to be more forgiving with certain violations it still would tie the monitoring and integration tests to API Blueprint which doesn’t seem to have added much in 3 years.

To get a sense of how our simple ActionAPI test looks with Dredd/API Blueprint look here. When you compare it to Phester it's over 4x larger and requires a schema of responses in order to compare the results and provide regex support. When considering monitoring, this isn’t really a con as you’ll be testing a handful of endpoints but when testing a large and complex API like the Action API this can quickly evolve to large files that are prone to errors.

One of the earlier selling points of Dredd for me was the variable extraction as shown here. On one hand, it provides a lot of flexibility to be able to provide unique instructions before/after endpoints are tested but it also doesn’t lend itself to DRY practices to explicitly extract/insert local variables for multiple requests as I expect we’d be doing for the Action API.

Overall, I think Dredd is a workable solution for small APIs but not the best tool for testing large APIs like the Action API. In this case, I’d lean more towards further developing Phester.

It's obvious a few people have put a lot of thought and effort in this. Existing options were considered and the decision was made to make a prototype. As far as I can tell, it's working well enough to consider further development.

In general, I dislike tests that are not written in a programming language. I can see the value in this case, but we should be very careful.

Advantages

The biggest advantage to tests not written in a programming language is that people not familiar with that language can read and write tests. That is valuable only if people writing tests would not be familiar with the code. If all/most people writing/reading tests already know the language (PHP) then writing tests in the language might make more sense.

Disadvantages

I do like yaml in general, but I am not sure it's a good choice for a big test suite. (As far as I understood it, phester would be used to create a big test suite.) To have a taste of why it might not be a good idea to write a big project in yaml, take a look at [[ https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master | integration/config ]] repository, [[ https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/jjb/ | jjb ]] folder. It contains jenkins job definitions written in yaml. It's transformed into xml that jenkins uses internally (as far as I understand it). It gets complicated quickly. I fear the same could happen to the tests written in yaml.

If a lot of tests would be similar, with very small differences (that's the case in jjb example) reusing existing code might get complicated.

Recommendation

My recommendation would be to implement a small but representative test suite in both yaml and a programming language (like php) using a testing framework like phpunit. The suite should have small number of very simple tests (list all pages, list all users...), small number of tests of usual complexity (create account > log in > edit page...), and the vast majority of the tests would be complicated tests, running multiple fixtures and testing complicated workflows, trying to reuse existing code with small differences. That is the place where I think the tool will either shine or crash. Duplication of effort (for creating two test suites initially) should not take too much time and I think it would be very valuable. Comparing how easy it is to write and read code written in both would be valuable, I think.

Even if phester doesn't support all the features that more complicated tests require, I think the tests should be written anyway, to develop the test format.

Thanks to @thcipriani for a humorous page explaining why you should use yaml for everything https://noyaml.com/

Decision matrix (tentative): https://docs.google.com/spreadsheets/d/1G50XPisubSRttq4QhakSij8RDF5TBAxrJBwZ7xdBZG0/edit#gid=0

It doesn't look like Behat got scored on all criteria; I've used it extensively over the past few years and would be happy to talk through its strengths and weaknesses if anyone on CPT would like.

In T222100#5283732, @kostajh wrote:

Decision matrix (tentative): https://docs.google.com/spreadsheets/d/1G50XPisubSRttq4QhakSij8RDF5TBAxrJBwZ7xdBZG0/edit#gid=0

It doesn't look like Behat got scored on all criteria; I've used it extensively over the past few years and would be happy to talk through its strengths and weaknesses if anyone on CPT would like.

I'd love to hear your impression. Can you start by putting comments into the matrix? Or values, if you feel like it.

So I played with codeception a bit. Here's the basic CRUD tests:

It's not too bad, but I find YAML more convenient for representing JSON structures and HTTP headers.

Also, codeception produces over 5000 (!) lines of generated code as scaffolding around this.

Codeception is pretty flexible. I'm wondering how hard it would be to implement the functionality we want for phester (including YAML best test specs) into codeception.

If we were to use codeception for other things as well (perhaps instead of selenium, or as a wrapper around phpunit, both of which it supports), this would make sense. But if all we want to do is validate HTTP request/response pairs, codeception seems to be overkill, and gets in the way more than it helps. And I'm not sure how easy (or hard) it would be to make it play nicely with tests defined by extensions.

Also, codeception doesn't seem a good fit for monitoring live services. Though it's probably possible.

I'd love to hear your impression. Can you start by putting comments into the matrix? Or values, if you feel like it.

It’s on my TODO list :)

Meanwhile you could have a look at https://github.com/deminy/behat-rest-testing

In T222100#5308491, @kostajh wrote:

Meanwhile you could have a look at https://github.com/deminy/behat-rest-testing

Thanks, played a bit with it. Here's the CRUD test in Behat:

It's pretty compact, but only because I added logic to the RestContext.

Encoding the POST data as a string is awkward, especially because it requires manual URL encoding, but this could probably be fixed. There actually is a version that uses table syntax, but that has JSON encoding for the POST body hard coded, so I didn't use it.

Overall, the Cucumber approach of matching natural language with regular expressions and mapping that to PHP code seems error prone for the use case of API tests. Does the response should contain "different text" do a substring match or regular expression? Is it case sensitive? How is I send a POST request to "/api.php?action=edit&format=json" with form data from I send a POST request to "/api.php?action=edit&format=json" with values?

While being able to read the scenarios as English sentences is nice, it hides what'S actually going on below. When testing an API, the high level "behavior" is not the only thing under test. The other thing is compliance in the nitty gritty. Cache control headers, content negotiation all that. It's possible to do all that in Cucumber, but the extra layer of indirection seems to get in the way more than it is helpful. But maybe it's just a matter of getting used to it?...

One thing that isn't clear to me is how I'd pass variables within a scenario. E.g. after I created a page, I want to extract the page's ID from the response and use it in the next step of the scenario. The only option I found was using state in the context object. That's fine for a login or something, but different scenarios may need completely different things to be passed between steps. If that would require a specialized Context class, that would be a show stopper, I'm afraid.

I found another tool that is relatively close to phester conceptually: htt the HTTP Test Tool (hosted on sourceforge - I didn't know that was still a thing). As far as I can tell, htt was created by the FSF in 2011 and as seen little activity between 2013 and 2019, but had version 2.4 release this year.

Like phester, htt is purely declarative and centered on modeling HTTP requests and responses. It's very low level though, and I don't see anything like fixtures or variables. I don't think it's a viable alternative, but it's similar enough that we should look to it for inspiration and for pitfalls.

In other news, I'm investigating the possibility of making a codeception plugin for phester style tests. Seems quite doable, but I have only just started to poke around.

Decision made: we will not build Phester. We'll use SuperTest and Mocha instead. Rationale to be documented on https://www.mediawiki.org/wiki/Core_Platform_Team/Decisions_Architecture_Research_Documentation/Using_SuperTest_Rather_Than_Creating_an_API_Integration_Test_Runner