Page MenuHomePhabricator

Create code analysis tool that checks all A/B tests are implemented
Closed, ResolvedPublic5 Estimated Story Points

Description

As a developer,
I want to be sure my A/B testing configuration is correctly implemented,
so that I can safely develop new A/B tests

The configuration file campaigns.yml makes no assumptions about how (and if) the A/B tests are implemented and if all code paths exist. Therefore, the configuration file should be checked against the actual code. The central reference point of that is the ChoiceFactory class, where each group of each campaign should be present.

AC:

  • Create a validation class that analyzes the factory code and checks the 1st parameter of calls to the feature toggle library against the current campaigns.yaml configuration from T196337. If the code contains calls for campaigns or group names that don't exist, exit the script with an error message listing all mismatches between code and configuration. In that case, the console should return a non-zero exit code. To be discussed: Issue warnings or errors for entries that are in campaigns.yml but not in the code?
  • Add the validation to our CI checks (make ci). This can be either a standalone console command or as a custom PHPStan rule

Background:
The easiest way to check this would be checking the code with regular expressions. However, this can be brittle and unprofessional. A better way would be to use either a custom PHPStan rule or at least using the PHP Tokenizer.

Event Timeline

gabriel-wmde renamed this task from Protect against misconfigurations to Create code analysis tool that cheks all A/B tests are implemented.Jun 13 2018, 7:51 AM
gabriel-wmde renamed this task from Create code analysis tool that cheks all A/B tests are implemented to Create code analysis tool that checks all A/B tests are implemented.
gabriel-wmde updated the task description. (Show Details)
kai.nissen set the point value for this task to 8.Jun 18 2018, 10:14 AM
kai.nissen changed the point value for this task from 8 to 5.Jun 18 2018, 10:22 AM
Vvjjkkii renamed this task from Create code analysis tool that checks all A/B tests are implemented to 28aaaaaaaa.Jul 1 2018, 1:04 AM
Vvjjkkii removed Tim_WMDE as the assignee of this task.
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed the point value for this task.
Vvjjkkii removed a subscriber: Aklapper.
kai.nissen renamed this task from 28aaaaaaaa to Create code analysis tool that checks all A/B tests are implemented.Jul 2 2018, 7:00 AM
kai.nissen assigned this task to Tim_WMDE.
kai.nissen lowered the priority of this task from High to Medium.
kai.nissen updated the task description. (Show Details)
kai.nissen set the point value for this task to 5.
kai.nissen added a subscriber: Aklapper.
Tim_WMDE moved this task from Doing to Review on the WMDE-FUN-Sprint-2018-06-25 board.