Page MenuHomePhabricator

Community configuration: Introduce validation warnings
Closed, ResolvedPublic

Description

MediaWiki-extensions-CommunityConfiguration validates all configuration against a JSON schema. Validation happens in two cases:

  1. before writing new configuration (to ensure users cannot write invalid configuration),
  2. before returning configuration from the store (so that client extensions can rely on the configuration being valid, even in case of store hiccups)

Sometimes, enforcing all validation rules are met in 100% of cases is not welcomed. For example, implementing a schema migration system (T357532) requires schema changes to be backwards and forwards compatible (this is because it is not possible to hook into train deployment/rollbacks, and immediately follow them by a maintenance script which migrates the config; for this reason, there will be always at least a couple of hours in which the version of the schema and of the data is different). A similar example is T359193: Community configuration: Decide whether the schema PHP classes need support for `required` and fresh deployment of MediaWiki-extensions-CommunityConfiguration – in that case, the configuration pages do not exist yet, and as such, configuration that has all required fields cannot be returned.

To support those cases, we should categorise validation issues into two categories: validation errors and validation warnings. CommunityConfiguration (or to be more precise, IConfigurationProvider::storeValidConfiguration) should refuse to write configuration that results in either a validation error or a validation warning.

On the other hand, CommunityConfiguration (IConfigurationProvider::loadValidConfiguration) should refuse to return configuration that contains a validation error, but it should return configuration when it has a validation warning (it should still inform the warning is present, but the configuration should be returned regardless).

Unless explicitly defined otherwise, any validation issue should be considered a validation error (in other words, unless defined otherwise, validation issues are blocking for both writes and reads).

List of validation warnings
  1. A property is marked as required, but it is not present in the data
  2. Schema disallows additional properties, but one is present
Acceptance Criteria
  • When a configuration store contains a validation warning (see above), IConfigurationStore::loadValidConfiguration() returns that configuration, but informs about the warning in an appropriate manner (directly to the caller, or to Logstash, or in an alternative manner)
  • When a configuration store contains any other validation issue (that is not explicitly listed as a warning), IConfigurationStore::loadValidConfiguration() returns a fatal error and refuses to return the configuration
  • When any validation issue is present, IConfigurationStore::storeValidConfiguration refuses to write the invalid data
  • When a configuration store contains a validation warning, Special:CommunityConfiguration loads the config properly, and allows changes to be made.
  • When a configuration store contains a validation error, Special:CommunityConfiguration fails to load with a validation error.

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
KStoller-WMF raised the priority of this task from Medium to High.Apr 15 2024, 6:21 PM
Michael moved this task from Incoming to Doing on the Growth-Team (Sprint 12 (Growth Team)) board.
Michael subscribed.

Not sure I understand everything in precise detail yet, but I think I can provide a quick first draft that should allow for further discussion.

Change #1022047 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/CommunityConfiguration@master] Add validation warnings

https://gerrit.wikimedia.org/r/1022047

Change #1023410 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/CommunityConfiguration@master] Add REQUIRED JsonSchema constant

https://gerrit.wikimedia.org/r/1023410

Change #1023410 merged by jenkins-bot:

[mediawiki/extensions/CommunityConfiguration@master] Add REQUIRED JsonSchema constant

https://gerrit.wikimedia.org/r/1023410

Change #1022047 merged by jenkins-bot:

[mediawiki/extensions/CommunityConfiguration@master] Add validation warnings

https://gerrit.wikimedia.org/r/1022047

@Michael, @Urbanecm_WMF - I tested client side validation and some results & open issues are summarized in T360971: [QA task] Community configuration test cases

Basically, this one was tested (although it's still work-in-progress):

  1. before writing new configuration (to ensure users cannot write invalid configuration)

Only some testing was done for validation

  1. before returning configuration from the store (so that client extensions can rely on the configuration being valid, even in case of store hiccups)

To test more, I need some advice on how to manipulate configuration store (in local env) to trigger validation error/warning coming from configuration store.

Yeah, in its current form, I'm not sure how many user-facing changes there are. This really comes into play when schemas are changed (by extension/skin developers).

So maybe, "seemingly nothing breaks in normal operation" is sufficient QA for this particular task?

But @Urbanecm_WMF has probably better guidance. I still don't have a good intuition yet for what kind of task/story should receive what kind of QA.

Yeah, in its current form, I'm not sure how many user-facing changes there are. This really comes into play when schemas are changed (by extension/skin developers).

So maybe, "seemingly nothing breaks in normal operation" is sufficient QA for this particular task?

But @Urbanecm_WMF has probably better guidance. I still don't have a good intuition yet for what kind of task/story should receive what kind of QA.

Thanks, @Michael! I agree that it's rather tricky to describe (and test) the cases for

  1. before returning configuration from the store (so that client extensions can rely on the configuration being valid, even in case of store hiccups)

Basically, what I'm looking for is

  • changes in schema that cannot be handled by UI form, e.g. required properties status is changed, added/removed properties.
  • extension.json or configuration store becomes unavailable

Hi @Etonkovidova and @Michael, thanks for the ping. This task is probably best tested locally, by temporarily disabling CommunityConfiguration (rendering the validation moot), making any change you want and then re-enabling the extension. That should allow you to create MW pages with invalid content pretty easily. Another approach can be to create the page first, and only then change the schema. With both approaches, you are likely to run into cache-caused issues (that would not actually happen in production), so I recommend to test with caching disabled (setting wgMainCacheType to CACHE_NONE temporarily).

If you want to test in beta, you'd need to find some way to "sneak in" an invalid config. In the past, that used to be possible by importing XML files or moving pages, but we might have fixed those holes already. Testing locally is probably the easiest way forward, I think.

Hope this helps!

Hi @Etonkovidova and @Michael, thanks for the ping. This task is probably best tested locally, by temporarily disabling CommunityConfiguration (rendering the validation moot), making any change you want and then re-enabling the extension. That should allow you to create MW pages with invalid content pretty easily. Another approach can be to create the page first, and only then change the schema. With both approaches, you are likely to run into cache-caused issues (that would not actually happen in production), so I recommend to test with caching disabled (setting wgMainCacheType to CACHE_NONE temporarily).

If you want to test in beta, you'd need to find some way to "sneak in" an invalid config. In the past, that used to be possible by importing XML files or moving pages, but we might have fixed those holes already. Testing locally is probably the easiest way forward, I think.

Hope this helps!

It helps a lot - thanks, @Urbanecm_WMF! Please review the test cases below (1) are such cases realistic enough? (2) the results are expected? (3) any additional testing is needed? I've documented all details of testing in this document (probably there are too many details, so please ask if you need some clarification).

What was checkedHow it was checkedResults
(1) CommunityConfiguration extension is not present/unreachable)mv CommunityConfiguration CommunityConfiguration_newMediaWiki is unable to load the extension CommunityConfiguration. Please check that the extension's name is correct and all of its files are properly installed.
Screen Shot 2024-05-14 at 2.32.23 PM.png (1×2 px, 326 KB)
(2) CommunityConfiguration extension is not installedwfLoadExtension( 'CommunityConfiguration' is commented outNo such special page for Special:CommunityConfiguration
Screen Shot 2024-05-14 at 3.10.22 PM.png (700×1 px, 107 KB)
Special:SpecialPages produces an error page: [4d304c1441ff380b0c39ca60] /wiki/Special:SpecialPages Wikimedia\Services\NoSuchServiceException: No such service: CommunityConfiguration.WikiPageConfigReader
(3)Adding properties to json pages when CommunityConfiguration extension is disabledan additional property was added to MediaWiki:ExampleConfig.jsonWhen CommunityConfiguration extension gets enabled again, no errors on MediaWiki:ExampleConfig.json or Special:CommunityConfiguration

That mostly looks good / as expected to me.

The error on Special:SpecialPages might come from CommunityConfigurationExample having CC as a hard dependency? The stack trace unfortunately doesn't say, but I can reproduce it with CommunityConfigurationExample still enabled in the config and it goes away with it being disabled.
We should maybe consider making CommunityConfiguration a soft dependency there, but that is beyond the scope of this task.

Etonkovidova updated the task description. (Show Details)