Page MenuHomePhabricator

Integrate the puppet compiler in the puppet CI pipeline
Open, MediumPublic

Description

The puppet compiler is very useful, but ATM its utility is limited by the fact we cannot know automatically which servers to check when a change is introduced. Also, it cannot be triggered directly from gerrit for this exact reason.

Given the added value that PCC gives us, we should aim in making it easy to trigger a comprehensive run from gerrit.

Ideally, this could be automated in the following way:

  1. when a keyword is set as a comment in gerrit, zuul starts the job
  2. the compiler analyzes the change for changed classes and node stanzas in site.pp
  3. the compiler queries puppetdb to find which hosts have the changed classes applied
  4. the compiler finds which node stanzas to pick from site.pp
  5. the compiler runs and the results are linked by jenkins-bot on the gerrit task

In order to do that we need the following things:

  • Have the puppet compiler be able to run in multiple instances, with a proper proxy in front
  • Have the puppet compiler be able to query puppetdb (this might need a move to production given the sensitivity of the queries)
  • Implement the node selection via puppetdb.

Event Timeline

Me and @hashar was talking about this a few months ago.

I've created this https://gerrit.wikimedia.org/r/#/c/325064/ to get jjb to generate the job and also created https://gerrit.wikimedia.org/r/#/c/315994/ so that we can write in a comment and it will check against the nodes.

We have a Jenkins job T97513 which has been made to recognizes Hosts: in commit message to pass down a list of nodes to the PPC. 91b70b4612abc6443a571dcc397622c5ddbe82af

The job can be triggered by commenting check experimental in Gerrit.

That's useful, thank you.

It didn't work for https://gerrit.wikimedia.org/r/c/operations/puppet/+/397723
From https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler-test/140/console
I think it's because it added a coma behind the regex.
+ NODES='re:dns.*,'

If we compare it to this successful one: https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/15597/console
+ NODES='re:cloudcontrol.*'

I tried it on https://gerrit.wikimedia.org/r/c/operations/puppet/+/496873 as well, the extra comma adds a ghost failed host, see https://puppet-compiler.wmflabs.org/compiler1002/141/

In addition, Jenkins doesn't seem to like having more than Change-id and Bug in the footer:
https://integration.wikimedia.org/ci/job/operations-puppet-tests-stretch-docker/9843/console

In addition, Jenkins doesn't seem to like having more than Change-id and Bug in the footer:

Seems to be in the code wonder if it just needs a release

https://gerrit.wikimedia.org/r/plugins/gitiles/integration/commit-message-validator/+/refs/heads/master/commit_message_validator/validators/GerritMessageValidator.py#20

Change 518974 had a related patch set uploaded (by Jbond; owner: John Bond):
[integration/config@master] ppc check experimental: removing trailing comma

https://gerrit.wikimedia.org/r/518974

I hit trailing comma issue reported by ayounsi above and have attempted a fix. CR which triggered the error is

https://gerrit.wikimedia.org/r/c/operations/puppet/+/518960

Change 518974 merged by jenkins-bot:
[integration/config@master] ppc check experimental: removing trailing comma

https://gerrit.wikimedia.org/r/518974

We have a Jenkins job T97513 which has been made to recognizes Hosts: in commit message to passe a list of nodes to the PPC. 91b70b4612abc6443a571dcc397622c5ddbe82af

The job can be triggered by commenting check experimental in Gerrit.

Did something cxhange with this today? I tried using it on 583078, it worked for the first comment at 14:01. however the comments at 14:09 & 14:16 failed to kick of a pcc run

We have a Jenkins job T97513 which has been made to recognizes Hosts: in commit message to passe a list of nodes to the PPC. 91b70b4612abc6443a571dcc397622c5ddbe82af

The job can be triggered by commenting check experimental in Gerrit.

Did something cxhange with this today? I tried using it on 583078, it worked for the first comment at 14:01. however the comments at 14:09 & 14:16 failed to kick of a pcc run

not sure if someone kicked something or if i was just inpatient but i got a set of experimental results at result at 14:30

Just a quick not that i added the ability to parse the commit message for the Hosts entry to utils/pcc. this means that you can now use the following to instruct PCC to look at the commit message of 594708 and use the Hosts entries it finds there. similar to how check experimental works

./utils/pcc.py 594708 parse_commit

Change 625779 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] build: Use at least commit-message-validator 0.7.0

https://gerrit.wikimedia.org/r/625779

Change 625779 merged by Jbond:
[operations/puppet@production] build: Use at least commit-message-validator 0.7.0

https://gerrit.wikimedia.org/r/625779

In addition, Jenkins doesn't seem to like having more than Change-id and Bug in the footer:

Seems to be in the code wonder if it just needs a release

https://gerrit.wikimedia.org/r/plugins/gitiles/integration/commit-message-validator/+/refs/heads/master/commit_message_validator/validators/GerritMessageValidator.py#20

This got merged and released today as commit-message-validator version 0.7.0. thanks to @Legoktm and @hashar

Change 651925 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/software/puppet-compiler@master] (WIP) nodegen: add node selections based on commited files

https://gerrit.wikimedia.org/r/651925

We have a Jenkins job T97513 which has been made to recognizes Hosts: in commit message to pass down a list of nodes to the PPC. 91b70b4612abc6443a571dcc397622c5ddbe82af

The job can be triggered by commenting check experimental in Gerrit.

Just found about this, and used it for the first time, quite useful, thanks! :)