Page MenuHomePhabricator
Create Task
Maniphest T85335

Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n
Closed, ResolvedPublic
Actions

Description

JSON files have been added to pywikibot/i18n, which now has python and JSON files with the same messages. The JSON files are not used yet, as the code changes to enable JSON have exposed packaging problems that are the subject of RFC https://www.mediawiki.org/wiki/Requests_for_comment/pywikibot_2.0_packaging

We need syntax validation of these JSON files for gerrit submissions, as message changes typically need to be approved quickly (and without errors) so they can be merged and the core & compat i18n submodule updated, before the messages can be used in code changes.

Details

SubjectRepoBranchLines +/-
Don't hardcode every directory in Gruntfile.jspywikibot/i18nmaster+2 -49
Run jsonlint jobs on pywikibot/i18n patchesintegration/configmaster+3 -0
Add grunt test to validate i18n JSONpywikibot/i18nmaster+114 -0
pywikibot-i18n-npmintegration/configmaster+6 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

jayvdb created this task.Dec 26 2014, 7:59 AM
jayvdb claimed this task.
jayvdb raised the priority of this task from to High.
jayvdb updated the task description. (Show Details)
jayvdb added projects: Pywikibot-i18n, I18n.
jayvdb added subscribers: Unknown Object (MLST), valhallasw, siebrand and 2 others.
gerritbot added a project: Patch-For-Review.Dec 26 2014, 8:00 AM

Change 181370 had a related patch set uploaded (by John Vandenberg):
Add grunt test to validate i18n JSON

https://gerrit.wikimedia.org/r/181370

Patch-For-Review

jayvdb mentioned this in T85336: Missing attribution of messages in pywikibot/i18n JSON files..Dec 26 2014, 8:18 AM
gerritbot added a comment.Jan 3 2015, 11:02 AM

Change 181370 merged by jenkins-bot:
Add grunt test to validate i18n JSON

https://gerrit.wikimedia.org/r/181370

Xqt closed this task as Resolved.Jan 3 2015, 11:03 AM
Xqt subscribed.
gerritbot added a comment.Jan 5 2015, 10:21 AM

Change 182775 had a related patch set uploaded (by Hashar):
pywikibot-i18n-npm

https://gerrit.wikimedia.org/r/182775

Patch-For-Review

gerritbot added a comment.Jan 5 2015, 10:32 AM

Change 182775 merged by jenkins-bot:
pywikibot-i18n-npm

https://gerrit.wikimedia.org/r/182775

hashar subscribed.Jan 5 2015, 10:42 AM

i have pushed a npm based Jenkins job and update Zuul configuration. Confirmed it works using the dummy change https://gerrit.wikimedia.org/r/#/c/182778/

hashar mentioned this in rCICF5ea9a7e7f506: pywikibot-i18n-npm.Jan 5 2015, 10:46 AM
jayvdb mentioned this in rPWIN7c69f77ebe06: Add grunt test to validate i18n JSON.Mar 4 2015, 8:40 AM
Diffusion mentioned this in rPWBCe4433434d647: Updated pywikibot/core Project: pywikibot/i18n….Mar 4 2015, 8:46 AM
Diffusion mentioned this in rPWBO9b9558cbcbee: Updated pywikibot/compat Project: pywikibot/i18n….Mar 4 2015, 8:53 AM
jayvdb reopened this task as Open.Mar 25 2015, 1:09 AM
jayvdb added a subscriber: XZise.

@hashar , it doesnt appears to be working by default at https://gerrit.wikimedia.org/r/#/c/196706/ (see PS6) .

Maybe it isnt run because the patch uploader isnt on the whitelist. However it was +1'd by @XZise, so I would expect the test is run. It did run when I added a 'recheck' comment.

This test is a syntax check, not running any untrusted code, so I expect it should run always, irrespective of the uploader of the patch.

jayvdb added a comment.Mar 25 2015, 1:17 AM

We also need to automatically add new JSON bundles to Gruntfile.js, otherwise the test success is misleading.

gerritbot subscribed.Mar 25 2015, 1:44 AM

Change 199559 had a related patch set uploaded (by Legoktm):
Run jsonlint jobs on pywikibot/i18n patches

https://gerrit.wikimedia.org/r/199559

gerritbot added a comment.Mar 25 2015, 3:43 AM

Change 199559 merged by jenkins-bot:
Run jsonlint jobs on pywikibot/i18n patches

https://gerrit.wikimedia.org/r/199559

Legoktm mentioned this in rCICF26bfb68aeb0c: Run jsonlint jobs on pywikibot/i18n patches.Mar 25 2015, 3:43 AM
Ladsgroup closed this task as Resolved.Mar 25 2015, 4:04 AM
Ladsgroup set Security to None.
Ricordisamoa subscribed.Mar 26 2015, 12:40 AM
jayvdb reopened this task as Open.Apr 3 2015, 3:31 AM

Re-opening again; my comments on mar 25 have not been addressed.

Nemo_bis added a project: Continuous-Integration-Infrastructure.Apr 3 2015, 5:28 AM
Legoktm subscribed.Apr 3 2015, 5:30 AM

What's still missing exactly? There's a jsonlint job that verifies all JSON files have valid syntax, and is run for all users, whitelisted or not.

Legoktm added a subscriber: Jdforrester-WMF.Apr 3 2015, 5:31 AM

Oh, I read too fast. I think you're refering to.

In T85335#1147403, @jayvdb wrote:

We also need to automatically add new JSON bundles to Gruntfile.js, otherwise the test success is misleading.

@Jdforrester-WMF said this should be possible to do with '*' wildcards or regex...?

jayvdb added a comment.Apr 3 2015, 5:32 AM

@Legoktm, new JS files (e.g. https://gerrit.wikimedia.org/r/#/c/196706/) are not checked by banana whateveritis.

Krinkle closed this task as Declined.Apr 13 2015, 4:16 PM
Krinkle subscribed.
In T85335#1177020, @jayvdb wrote:

@Legoktm, new JS files (e.g. https://gerrit.wikimedia.org/r/#/c/196706/) are not checked by banana whateveritis.

The jsonlint job validates syntax in all json files. Including new directories and files therein.

The npm job runs whatever framework you like to use. Bash, Grunt, Gulp, anything. So this task is closed. It's your responsibility (or that of other pywikibot maintainers) to come up with a test strategy and a viable way to maintain that.

For example, you can use Grunt and grunt-banana-checker to validate things specific to the "Banana" format that the Milkshake project came up with. (E.g. the concept of "qqq" files, @authors properties and all that.)

Each directory is a separate localisation group with its own qqq file. As such, in order for Banana to be able to find missing qqq entries, it needs to know about each directory.

However, there should be no need to configure each directory by hand. Pywiki's structure looks quite simple. You can scan the top directory from Gruntfile.js and collect the relevant directory names in an array automatically. You can even provide the array in a single test:

initConfig({
  banana: {
    all: [ 'foo/', 'bar/' ] // create this array programmatically after scanning the repo
  }
});

If you prefer the results (and potential errors) to be grouped in the output separately, you'd build it more like the current structure:

var projects = ...;
var bananaGroups = {};
projects.forEach( function (project) {
  bananaGroups[ project ] = project + '/';
} );

initConfig({
  banana: bananaGroups
});
Krinkle changed the task status from Declined to Resolved.Apr 13 2015, 4:16 PM
Krinkle removed a project: Patch-For-Review.
Krinkle moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
Krinkle moved this task from In-progress to Done on the Continuous-Integration-Infrastructure board.
jayvdb reopened this task as Open.Apr 13 2015, 7:24 PM
In T85335#1203335, @Krinkle wrote:

However, there should be no need to configure each directory by hand. Pywiki's structure looks quite simple. You can scan the top directory from Gruntfile.js and collect the relevant directory names in an array automatically. You can even provide the array in a single test:

initConfig({
  banana: {
    all: [ 'foo/', 'bar/' ] // create this array programmatically after scanning the repo
  }
});

And this task is open until the check is dynamic , and new bundles in changesets for review are checked.

Krinkle edited projects, added Continuous-Integration-Config; removed Continuous-Integration-Infrastructure.Apr 24 2015, 6:16 PM
Krinkle moved this task from Backlog to Archive on the Continuous-Integration-Config board.
Krinkle removed a subscriber: gerritbot.
hashar added a comment.Jun 4 2015, 1:19 PM

pywikibot/i18n has multiple directories. @jayvdb complaint about banana-checker is you have to list each sub dir explicitly. Looking at http://git.wikimedia.org/blob/pywikibot%2Fi18n.git/e7b1df619a27c80122d59892fbce7e12d24679da/Gruntfile.js we have:

 grunt.initConfig( {
       banana: {
           pywikibot: 'pywikibot/',
           thirdparty: 'thirdparty/',
           add_text: 'add_text/',
           archivebot: 'archivebot/',
           basic: 'basic/',
...

Which is error prone. Would be nice to have banana checker to support a glob pattern of some sort so we can match all sub directories. Something like:

grunt.initConfig( {
     banana: {
         all: '*/',
    }

@jayvdb you might want to fill a feature request on https://github.com/wikimedia/grunt-banana-checker/issues

Another approach is to generate the banana configuration dynamically. Maybe the Gruntfile.js can list the directories at the root of the repository and craft the parameters for us.

hashar lowered the priority of this task from High to Medium.Aug 24 2015, 2:04 PM
gerritbot added a comment.Aug 31 2015, 4:31 AM

Change 234939 had a related patch set uploaded (by Legoktm):
Don't hardcode every directory in Gruntfile.js

https://gerrit.wikimedia.org/r/234939

gerritbot added a project: Patch-For-Review.Aug 31 2015, 4:31 AM
gerritbot added a comment.Aug 31 2015, 4:44 AM

Change 234939 merged by jenkins-bot:
Don't hardcode every directory in Gruntfile.js

https://gerrit.wikimedia.org/r/234939

Legoktm mentioned this in rPWBCb6a6c0fd3a8c: Updated pywikibot/core Project: pywikibot/i18n….Aug 31 2015, 4:44 AM
Legoktm mentioned this in rPWBCa256c3b632fb: Updated pywikibot/core Project: pywikibot/i18n….
jayvdb closed this task as Resolved.Aug 31 2015, 4:45 AM

Thanks @Legoktm for helping finish this task!

Legoktm mentioned this in rPWIN7dd81cea84c4: Don't hardcode every directory in Gruntfile.js.Aug 31 2015, 5:21 AM
hashar added a comment.Aug 31 2015, 7:57 AM

Summary GruntJS support globbing patterns http://gruntjs.com/configuring-tasks#globbing-patterns. Well done!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits