For the hackathon I I will probably spend most of the week-end helping / training volunteers + catching up with people. I am usually not writing that much code.

That being said, the feature can be added to Zuul by anyone knowing python and willing to put their hands in Zuul code. It is not trivial though since Zuul is tightly coupled with having projects explicitly listed.

I am right now busy packaging it for Debian and then working on having Jenkins jobs run in isolated machines.

All extensions + skins currently have a basic phplint job running. jsonlint will soon be added to that list. https://gerrit.wikimedia.org/r/#/c/198185/ adds a test to verify that every extension and skin has *some* job configured. For that to work, we need to make sure the repository creators let CI know when new repos are being created.

We could potentially create magic composer/npm jobs that run "composer test" / "npm test" if it detects that those are configured and add them to all extensions.

Qunit tests and PHP units should be in by default and my main concern right now. There is an issue in Gather at the moment due to a broken qunit test that could be impacting other projects which have qunit jobs.

In T92909#1157611, @Legoktm wrote:

We could potentially create magic composer/npm jobs that run "composer test" / "npm test" if it detects that those are configured and add them to all extensions.

Towards this goal, a while ago @Krinkle created a gist that adds these to extensions/examples/BoilerPlate. I submitted it at https://gerrit.wikimedia.org/r/200226 , feedback welcome.

@Spage the boilerplate is great but some extensions may not choose to use it for some reason... my goal is to not have to waste time setting this sort of stuff up. We've spent about 8 hours I reckon so far dealing with zuul stuff in setting up Gather. This is very wasteful.

From https://www.mediawiki.org/wiki/Continuous_integration/meetings/2015-03-30/minutes

Extensions are created in Gerrit but almost never have an initial set of CI jobs set for them. Another solution would be to remove the right for human to vote verified +2 , which will force the repository creator to configure CI with the appropriate jobs so changes can land in the repository. That is not automatic though.

If we want a set of given job to be triggered by default, one will need to add support for it in Zuul. Until someone write the necessary code in Zuul, it is not going to be automatic.

There is no good way to know which entry points a repository needs/wants: npm, composer, publish (jsduck, doxygen, coverage). Depends on it's maturity and whether it's a MediaWiki extension or a standalone project.

Over the past year we've dramatically simplified the Zuul configuration. We also moved most bootstrapping and control over test execution to the individual repositories. Thus, setting up a job won't do much. Adding it to Zuul is a 3-line patch that is trivially merged/deployed. That is not where the overhead and primary cost is. Setting up the repository itself is, while also simple, where the main responsibility lies. After that it's trivial to add it to Zuul.

I don't think it's worth automating that. And not possible at the moment due to Zuul not having a wildcard feature (T76256).

FWIWWhat may seem like a trivial 3 line patch to you is not trivial to a pleb like me. It's a huge context switch to a repository I never spend time in. When I do spend time in it it takes too long to get code reviewed/fixed and merged (it really isn't trivial to me). At the same time however things break and the cost of not having it is far greater... Gather had no QUnit coverage via Jenkins for a good 3 weeks and tests got broken in the process due to people not running them locally. The cost of giving a repository by default more than it needs/wants seems like a good thing.

As an alternative it would be good to have a web ui for this sort of thing if that makes more sense. It would be great for me to be able to make jenkins voting on certain jobs without having to go into the large scary looking zuul config file.

The time needed to set up everything individually is a major cause of slowdown when added to a need to set up an extension. We're being asked to innovate by Lila and plebs like me shouldn't need to think about this stuff.....

cc @greg

See : https://gerrit.wikimedia.org/r/#/c/205726/ this is wasting reviewers time unnecessarily.

In T92909#1222266, @Jdlrobson wrote:

When I do spend time in it it takes too long to get code reviewed/fixed and merged (it really isn't trivial to me). At the same time however things break and the cost of not having it is far greater... Gather had no QUnit via Jenkins for a good 3 weeks and tests got broken in the process due to people not running them locally.

If you have a patch to integration/config waiting and blocking you from having tests, ask for review. Be it with IRC pings, e-mails, a Phabricator task, or via your representative on the Scrum-of-Scrums meeting. This shouldn't take more than 2-3 days at most.

In T92909#1222266, @Jdlrobson wrote:

What may seem like a trivial 3 line patch to you is not trivial to a pleb like me. It's a huge context switch to a repository I never spend time in.

It really isn't that complicated. It's just something new to get used to. It's a lot simpler than e.g. configuring JSHint, JSCS, PHPCS, PHPUnit, or QUnit. But those you've worked with before.

In T92909#1222266, @Jdlrobson wrote:

As an alternative it would be good to have a web ui for this sort of thing if that makes more sense.

Creating and maintaining a UI for this would be disproportionate in terms of cost and benefit. And having a web UI doesn't change familiarity with concepts and context switching related to continuous integration.

In fact, the configuration is already very abstracted. The YAML config file is as close to a UI as you can get with predefined labels and value options.

In T92909#1222266, @Jdlrobson wrote:

The time needed to set up everything individually is a major cause of slowdown when added to a need to set up an extension.

Bootstrapping a repository and its CI config has gotten significantly simpler compared to 6-12 months ago. We'll continue to make it easier and improve documentation. The "Example" extension can also help in this regard.

Projects can now enable one or more standard entry point jobs in Zuul config. The build is controlled via composer.json (composer test) and/or package.json (npm test). This practice has been popularised via GitHub and Travis CI, and allows our infrastructure to scale better. It also removes the disconnect between Jenkins jobs and repositories. For example, package.json is part of the repository and naturally part of branches. Whereas updating a job also affects commits to older release- and wmf-branches. This often causes failures.

Entry points allows projects to innovate better and introduce or change how they do testing. And to configure their tests exactly the way they want to without having to recompile Jenkins jobs (which is a lot more complicated than adding a line to Zuul config).

I understand.. but the reason I use github.com for personal development is the simplicity. Put frankly, I don't want to spend time on this sort of thing chasing people/writing patches. I want it automated for me and I'm just trying to help you understand that. To some devs they sadly don't even care about this sort of thing and have no interest in jumping over the hurdles to get it (hence why lots of our extensions still don't benefit from this sort of infrastructure). We should be enforcing this and forcing people in our organisation to write good quality code that doesn't break tests and adheres to coding standards and best practices.

I appreciate all the work you and @hashar have done on this so far and it's soooo much better than 3 years ago when I joined but I really want to get to the point that I'm not leaning on either of you in anyway for this sort of thing. It's an unnecessary bottleneck.

It is time to promote Wikimedia-Hackathon-2015 activities in the program (training sessions and meetings) and main wiki page (hacking projects and other ongoing activities). Follow the instructions, please. If you have questions, about this message, ask here.

I'm assuming this is not going to happen without help from @Krinkle or @hashar ...if so I guess I should remove the hackathon tag.

Did someone work on this project during Wikimedia-Hackathon-2015? If so, please update the task with the results. If not, please remove the label.

Renamed to be a generic task on this.

This also seems like it might be an EPIC, yes? I'm not sure what the needed work in ci-infra is.

It appears that there isn't the ability to have a set of generic tests that run iff they exist in the repository? (based on Krinkle's comment "There is no good way to know which entry points a repository needs/wants".)

It appears that there isn't the ability to have a set of generic tests that run iff they exist in the repository? (based on Krinkle's comment "There is no good way to know which entry points a repository needs/wants".)

That is basically T111181 (it in addition contains running it all as one job in the sense of checkout and nodepool Jenkins slave).

In T92909#1697703, @greg wrote:

This also seems like it might be an EPIC, yes? I'm not sure what the needed work in ci-infra is.

Probably, or a tracking task or a story. But I think once the blockers are done only adding a quick patch for enabling it is necessary to solve this task.

I still think a lot of this should be automated. for instance if I have package.json in my repository why can't zuul see that and run the npm job? If I have a tests/browser folder why can't it run Jenkins?

Please reconsider. This stuff is costly when you don't have +2 in zuul repository and are setting up a new extension....