Page MenuHomePhabricator

Add tests to ensure consistency between OAD example and OpenAPI linter
Closed, ResolvedPublic3 Estimated Story Points

Description

The OpenAPI description (OAD) example and the OpenAPI linter are designed to help API maintainers create high quality OpenAPI descriptions. It's important for these tools to work well together and provide consistent guidance, and one of the best ways to ensure that is to test the linter against the example.

In scope for this task:

  • Decide where to store the OAD example for use in tests. Note that this will be a temporary location until the linter code gets reorganized. When that happens, we'll probably want to keep the example alongside linter rules. Possible options:
  • Create tests that lint the OAD example
  • Decide what to do when tests fail. Tests failing might indicate an error in linter rules, or an error in the OAD example.

Details

Other Assignee
KBach
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Fix issues in the OAD exampletoolforge-repos/wmf-openapi-linter!21kbachfix-oad-issuesadd-fullspec-example-test
Add the example spec to the teststoolforge-repos/wmf-openapi-linter!20wikigitadd-fullspec-example-testmain
Customize query in GitLab

Event Timeline

BPirkle triaged this task as Medium priority.Mar 11 2026, 1:27 PM
HCoplin-WMF set the point value for this task to 3.Mar 12 2026, 3:52 PM

Notes from Estimation:

  • For now, we will reference a copy of the Wikimedia Pet Store OAD into testing instead of referencing the repo directly (since it's in the tech writer repo)
  • When we move to a library implementation, we can include the file as part of the package
  • Might multiple smaller and simpler examples be more helpful than a single spec that reflects everything? (especially as rules get more complicated)
    • Potentially consider this if/when we have more rules and start grouping them into related sets (eg: related rules are bundled, and there is a shorter/simpler OAD for each ruleset)
NOTE: If this ticket drags, please flag it to the team so we can reassess the approach.

From the technical writers' perspective, it's OK to move the OAD example to the linter repository and deprecate its current, separate repo. We're no longer making active changes to the example and most future changes will likely be related to compatibility with the linter, so it makes sense to keep the example and the linter together.

I've moved all the fiels from the doc repo (including the README) into a dedicated sub-folder in the tests, and made a special test to run all rules on the example OAD yaml.

However -- the tests fail for the moment. We need a followup task to go over the errors and see if there are issues in the spec, or in the rules. This should probably be a followup, so the test itself is marked skip. It can be run locally by removing the .skip(...) directive.

I can investigate the issues with the OAD example on Monday unless someone else takes care of this earlier. I'm not sure if we need a follow-up task for the investigation itself (it's already covered in the last point of the task description as Decide what to do when tests fail...). If the problems are with the example, I'll be able to address them directly. For issues with the linter, I'll create follow-up tasks.

I linted the OAD example to check why the test fails and discovered the following:

  • The OAD example had a few missing fields. I submitted a merge request that fixes this.
  • The linter reports a lot of issues with severity hint when linting the example. I don't think these need fixing at the moment, and would suggest that we exclude hints from the test.
  • The linter incorrectly validates parameters in the link object using the same rule as for parameters in operations. I reported this in T425920.
  • The linter still incorrectly handles date fields similarly to the problem described in T414974. In that task, this was caused by the oas3-valid-schema-example rule. Here, the same error is raised by oas3-valid-media-example.

I linted the OAD example to check why the test fails and discovered the following:

  • The OAD example had a few missing fields. I submitted a merge request that fixes this.

Perfect, merged.

  • The linter reports a lot of issues with severity hint when linting the example. I don't think these need fixing at the moment, and would suggest that we exclude hints from the test.

Yes, that's not planned to be a normal thing and I will be removing it from the jest tests.

I'm still testing how to make the hints appear in the terminal results but not be blocking, so in the skipped test I wanted to make sure we *see* all of them, but the test should really mostly check if errors (maybe warnings) are emitted.

I'll be iterating on that, but it's okay if they're the only thing that's left, the jest tests should not be failing on hints.

  • The linter incorrectly validates parameters in the link object using the same rule as for parameters in operations. I reported this in T425920.
  • The linter still incorrectly handles date fields similarly to the problem described in T414974. In that task, this was caused by the oas3-valid-schema-example rule. Here, the same error is raised by oas3-valid-media-example.

Let's make sure to prioritize the fixes for these?

Marking as resolved as part of sprint close out.