Let's separate the MCS tests into three parts:
- small (unit) tests
- medium (integration) tests
- large (system) tests
The idea and the following table come from [1].
What's the difference between these tests?
| Feature | Small | Medium | Large |
| Network access | No | localhost only | Yes |
| Database | No | Yes | Yes |
| File system access | No | Yes | Yes |
| Use external systems | No | Discouraged | Yes |
| Multiple threads | No | Yes | Yes |
| Sleep statements | No | Yes | Yes |
| System properties | No | Yes | Yes |
| Time limit (seconds) | 60 | 300 | 900+ |
What should we do?
We already have the small (unit) tests separated. So, what's left is to separate the feature tests into medium and large tests, and change the names.
We should probably discuss and think some more about how we want the folder structure to look like when this is done. (test/small, test/medium, test/large or just have them all combined in one folder but use annotations for medium and large tests?)
Medium tests could use the fixtures we get from sepia. Would be good if we add something to our sepia usage to better identify the fixtures per test. (Maybe have a folder per test?)
- Regular test runs used by CI run small and medium tests.
- A developer can chose to just run the small tests if the medium tests take too long.
- A special test job should be setup for large tests against live project data. T177896: Create a CI task for MCS periodic tests
- As a bonus, it would be great to also allow running against a local Parsoid installation or against beta cluster.
Benefits
- Regular test runs are more stable and run faster.
- Periodic tests against various Parsoid instances allow us to catch changes in Parsoid sooner, and also would allow the Parsoid team to run them.