To clarify, Puppetizing Jenkins in general is definitely not a 0% chance thing. That's very much a 100% thing. For the most part I think it is puppetized already.

The story here is specifically about it's configuration, which Jenkins internally stores as an XML file on disk on gallium.wikimedia.org.

Antoine and I both would not support that moving into operations/puppet.git (e.g. as file ensured by puppet, or an erb file expanding to xml with placeholders for $::mail_smarthost etc.). Because that would effectively make it very hard for us to do your job as these configuration values change quite frequently.

It can be compared to mediawiki-config. You wouldn't want every change there to go through operations/puppet. That's why we made a hybrid solution there with operations/mediawiki-config so that it is versioned, tracked and reviewed, but with deployers for wmf-mediawiki having write access to it as well. When ops makes changes that affect mediawiki, they update the config. And day-to-day the config is handled by deployers.

I'd like to propose a similar approach for Jenkins. We'd have operations/jenkins-config containing the XML files (or something that compiles to XML). And clone the repo on gallium (like for mediawiki-config on tin and zuul-config on gallium alreadsy).

One can probably puppetize a chunk of it based on https://github.com/jenkinsci/puppet-jenkins. But I am opposed to have the Jenkins configuration maintained in puppet because we have no rights on it (i.e. no +2 / merge).

Faidon proposal was to use puppet to have the Jenkins config to rely on $::mail_smarthost . Instead, we can probably have a local smart host relay reachable on 127.0.0.1 that will be configured to relay mail to $::mail_smarthost. That would fix that specific issue.

1. Put the xml file in operations/puppet as template with placeholders for dynamic/private data and ensured in the right location on gallium.

2. Use Puppet File::notify to have Jenkins reread/reload config from disk (it has an option for that in the web ui, I hope it supports doing this from command line as well, otherwise we can do a graceful restart).

@zeljkofilipin related to puppetization of Jenkins.

Also look at how OpenStack is handling their Jenkins config ( T95049: Figure out how Jenkins conf is maintained by OpenStack ).

With the recent gallium / CI outage, lack of Jenkins config management has been a threat. Moving the task to phase out gallium project.

Moving this out of the phase out gallium project. Having part of Jenkins unpuppetized is definitely an issue but should not hold from migrating it to another machine. We can still rsync the current config.

We definitely need to address that Technical-Debt though.

Been playing it a bit with this today. On jenkintest.integration.eqiad.wmflabs I have installed a basic jenkins. It generates a few files under /var/lib/jenkins, interesting candidates are:

• secret.key 64 random bytes
• secret.key.not-so-secret an empty file to flag the randomness is good
• identity.key.enc probably a RSA private key encrypted with the secret.key somehow ( based on https://wiki.jenkins-ci.org/display/JENKINS/Instance+Identity ).

When connecting to the web interface and doing a noop change in the configuration the following dummy files are generated:

config.xml
hudson.maven.MavenModuleSet.xml
hudson.scm.CVSSCM.xml
hudson.scm.SubversionSCM.xml
hudson.tasks.Ant.xml
hudson.tasks.Mailer.xml
hudson.tasks.Maven.xml
hudson.tasks.Shell.xml
hudson.triggers.SCMTrigger.xml
jenkins.model.ArtifactManagerConfiguration.xml
jenkins.model.JenkinsLocationConfiguration.xml
jenkins.mvn.GlobalMavenConfig.xml

They are the default configuration files of:

• Jenkins itself config.xml (see below)
• built-in plugins (can be found as /plugins/*.jpi) and global "tasks" such as shell/maven etc.
• some magic secrets/hudson.util.Secret

<?xml version='1.0' encoding='UTF-8'?>
<hudson>
  <disabledAdministrativeMonitors/>
  <version>1.651.2</version>
  <numExecutors>2</numExecutors>
  <mode>NORMAL</mode>
  <useSecurity>true</useSecurity>
  <authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
  <securityRealm class="hudson.security.SecurityRealm$None"/>
  <disableRememberMe>false</disableRememberMe>
  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
  <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
  <systemMessage>Welcome</systemMessage>
  <jdks/>
  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
  <clouds/>
  <quietPeriod>5</quietPeriod>
  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
  <views>
    <hudson.model.AllView>
      <owner class="hudson" reference="../../.."/>
      <name>All</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
    </hudson.model.AllView>
  </views>
  <primaryView>All</primaryView>
  <slaveAgentPort>0</slaveAgentPort>
  <label></label>
  <nodeProperties/>
  <globalNodeProperties/>
</hudson>

We can probably gives a shoot at the config.xml. When we start inserting credentials in there, they will need to be encrypted using the secret.key though so that might not be so trivial. Maybe puppet can invoke a groovy script to do so.

Mentioned in SAL (#wikimedia-releng) [2017-03-01T17:11:30Z] <hashar> cleaned out Jenkins security matrix to drop users that are no more used/inexistent -- T69027