Page MenuHomePhabricator

Create a Jenkins check to verify hooks.yaml formatting
Closed, DeclinedPublic

Description

See T115388 for context.

Event Timeline

Tgr created this task.Oct 28 2015, 8:28 PM
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr added a subscriber: Tgr.
Restricted Application added a project: Documentation. · View Herald TranscriptOct 28 2015, 8:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Creating a specific job for each use cases turned out to be a nightmare. So instead the Jenkis jobs have been made dumb and just invoke entry points listed at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points

In short the available ones are more or less:

  • composer install && composer test
  • npm install && npm test
  • tox
  • bundle install && bundle exec rake

That gives freedom to developers to implement their checks however they want without having to wait for a Jenkins job to be crafted. Hook in the entry point, add your test, propose to Gerrit and it is run by the entry point jobs.

For hooks.txt to hooks.yaml the question to ask is what is going to consume them. If it is a PHP script, PHP lacks built-in support for YAML. @bd808 has a PHP extension but I don't think we want to depend on it. packagist must have a pure PHP yaml, I think ContentTranslation or Translate uses that.

From there one can add a test in mediawiki core phpunit suite that will attempt to parse the /docs/hooks.yaml file and can even validate it against a schema. Then fail on error which will break the phpunit job we have that runs all the other tests.

Tgr added a comment.Oct 28 2015, 11:09 PM

Creating a specific job for each use cases turned out to be a nightmare. So instead the Jenkis jobs have been made dumb and just invoke entry points listed at https://www.mediawiki.org/wiki/Continuous_integration/Entry_points

Agreed, I just worded the task sloppily.

For hooks.txt to hooks.yaml the question to ask is what is going to consume them. If it is a PHP script, PHP lacks built-in support for YAML. @bd808 has a PHP extension but I don't think we want to depend on it. packagist must have a pure PHP yaml, I think ContentTranslation or Translate uses that.

Symfony has a decent YAML parser. That (or whichever library we end up with - the one used by Translate/OSM does not seem powerful enough though) should probably be a dev dependency for core, both for this task and for T115959.

Indeed we have a bunch of them scattered around in multiple extensions:

OpenStackManagerSpyc.phphttps://github.com/mustangostang/spyc/
Translate phpyamlrecommended, now clue were sources are
Translatemustangostang/spyc Suggested by composer.json

Note OpenStackManager has a Yaml ContentHandler which we might want to incorporate in core. Thought it is based on Spyc.

I am a fan of Symfony components. They are usually well thought of and that has the potential of attracting developers already familiar with that Framework.

Sounds like yet another Epic task :-(

@bd808 pecl extension is: http://bd808.com/pecl-file_formats-yaml/ which uses libyaml C bindings.

@bd808 pecl extension is: http://bd808.com/pecl-file_formats-yaml/ which uses libyaml C bindings.

This is the phpyaml that can be used as $wgTranslateYamlLibrary in the Translate extension. See https://pecl.php.net/package/yaml for links to everything.

The TL;DR on YAML support in PHP is pretty much either use symfony/yaml or use the PECL module which is also available for HHVM (and built into the WMF HHVM build). There are other pure PHP implementations than symfony/yaml but they are generally inferior.

Having C bindings to libyaml is faster than a pure PHP parser, but in general not needed unless you are processing a lot of YAML. I got involved in bringing the php-yaml module to PECL because the framework I was working on at the time used YAML for all configuration and we found that the other C bindings (syck) were buggy and that pure PHP solutions were measurably slow for our use case. A typical request/response cycle on that framework parsed >5K of YAML files though so it was a bit extreme.

If we wanted to get really fancy we could make a Wikimedia library that wraps both symfony/yaml and the yaml PECL module and uses the C acceleration when available automatically.

Tgr added a comment.Oct 30 2015, 8:16 PM

The problem there is that libYAML implements YAML 1.1 while Symfony implements a subset of YAML 1.2.

For the documentation use cases, we don't really care about speed - hooks.yaml is largish but CI hooks are slow anyway, for maintenance scripts speed doesn't matter that much, for using YAML data in wiki templates speed is more problematic but post-YAML-parsing data can be cached most of the time.

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 9 2015, 1:17 PM
hashar closed this task as Declined.Jun 16 2017, 11:12 AM

Can be reopened if one day there is activity on the parent task T115388: Convert hooks.txt to YAML [Outreachy microtask]