Jenkins: Figure out a long term solution to maintaining Jenkins configuration
Open, NormalPublic

Description

Since the start, Jenkins has been using the smtp.pmtpa.wmnet DNS entry to send its email. The DNS entry has been removed yesterday and no mails were sent anymore. I have set jenkins to use wiki-mail.wikimedia.org instead.

Faidon recommended to use the puppet variable $::mail_smarthost , but Jenkins configuration is not maintained by puppet.

We should either:

  • puppetize Jenkins (0% chance it is going to happen)
  • use Puppet to set SMTP relay listening on localhost and point Jenkins to it. The conf will be managed by puppet and the entry will be stable in Jenkins config
  • find out a stable DNS entry to use instead of wiki-mail.wikimedia.org

Version: wmf-deployment
Severity: enhancement

Details

Reference
bz67027
bzimport raised the priority of this task from to Normal.
bzimport set Reference to bz67027.
bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.Jun 24 2014, 2:20 PM

To clarify, Puppetizing Jenkins in general is definitely not a 0% chance thing. That's very much a 100% thing. For the most part I think it is puppetized already.

The story here is specifically about it's configuration, which Jenkins internally stores as an XML file on disk on gallium.wikimedia.org.

Antoine and I both would not support that moving into operations/puppet.git (e.g. as file ensured by puppet, or an erb file expanding to xml with placeholders for $::mail_smarthost etc.). Because that would effectively make it very hard for us to do your job as these configuration values change quite frequently.

It can be compared to mediawiki-config. You wouldn't want every change there to go through operations/puppet. That's why we made a hybrid solution there with operations/mediawiki-config so that it is versioned, tracked and reviewed, but with deployers for wmf-mediawiki having write access to it as well. When ops makes changes that affect mediawiki, they update the config. And day-to-day the config is handled by deployers.

I'd like to propose a similar approach for Jenkins. We'd have operations/jenkins-config containing the XML files (or something that compiles to XML). And clone the repo on gallium (like for mediawiki-config on tin and zuul-config on gallium alreadsy).

One can probably puppetize a chunk of it based on https://github.com/jenkinsci/puppet-jenkins. But I am opposed to have the Jenkins configuration maintained in puppet because we have no rights on it (i.e. no +2 / merge).

Faidon proposal was to use puppet to have the Jenkins config to rely on $::mail_smarthost . Instead, we can probably have a local smart host relay reachable on 127.0.0.1 that will be configured to relay mail to $::mail_smarthost. That would fix that specific issue.

  1. Put the xml file in operations/puppet as template with placeholders for dynamic/private data and ensured in the right location on gallium.
  1. Use Puppet File::notify to have Jenkins reread/reload config from disk (it has an option for that in the web ui, I hope it supports doing this from command line as well, otherwise we can do a graceful restart).
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2015, 9:01 AM
hashar added a comment.EditedSep 2 2015, 9:01 AM

@zeljkofilipin related to puppetization of Jenkins.

Also look at how OpenStack is handling their Jenkins config ( T95049: Figure out how Jenkins conf is maintained by OpenStack ).

hashar edited subscribers, added: thcipriani; removed: wikibugs-l-list.

With the recent gallium / CI outage, lack of Jenkins config management has been a threat. Moving the task to phase out gallium project.

Moving this out of the phase out gallium project. Having part of Jenkins unpuppetized is definitely an issue but should not hold from migrating it to another machine. We can still rsync the current config.

We definitely need to address that Technical-Debt though.

Krinkle removed a subscriber: Krinkle.Jul 27 2016, 5:54 AM
hashar added a comment.EditedMar 1 2017, 4:48 PM

Been playing it a bit with this today. On jenkintest.integration.eqiad.wmflabs I have installed a basic jenkins. It generates a few files under /var/lib/jenkins, interesting candidates are:

When connecting to the web interface and doing a noop change in the configuration the following dummy files are generated:

config.xml
hudson.maven.MavenModuleSet.xml
hudson.scm.CVSSCM.xml
hudson.scm.SubversionSCM.xml
hudson.tasks.Ant.xml
hudson.tasks.Mailer.xml
hudson.tasks.Maven.xml
hudson.tasks.Shell.xml
hudson.triggers.SCMTrigger.xml
jenkins.model.ArtifactManagerConfiguration.xml
jenkins.model.JenkinsLocationConfiguration.xml
jenkins.mvn.GlobalMavenConfig.xml

They are the default configuration files of:

  • Jenkins itself config.xml (see below)
  • built-in plugins (can be found as /plugins/*.jpi) and global "tasks" such as shell/maven etc.
  • some magic secrets/hudson.util.Secret
/var/lib/jenkins/config.xml
<?xml version='1.0' encoding='UTF-8'?>
<hudson>
  <disabledAdministrativeMonitors/>
  <version>1.651.2</version>
  <numExecutors>2</numExecutors>
  <mode>NORMAL</mode>
  <useSecurity>true</useSecurity>
  <authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
  <securityRealm class="hudson.security.SecurityRealm$None"/>
  <disableRememberMe>false</disableRememberMe>
  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
  <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
  <systemMessage>Welcome</systemMessage>
  <jdks/>
  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
  <clouds/>
  <quietPeriod>5</quietPeriod>
  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
  <views>
    <hudson.model.AllView>
      <owner class="hudson" reference="../../.."/>
      <name>All</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
    </hudson.model.AllView>
  </views>
  <primaryView>All</primaryView>
  <slaveAgentPort>0</slaveAgentPort>
  <label></label>
  <nodeProperties/>
  <globalNodeProperties/>
</hudson>

We can probably gives a shoot at the config.xml. When we start inserting credentials in there, they will need to be encrypted using the secret.key though so that might not be so trivial. Maybe puppet can invoke a groovy script to do so.

Paladox added a subscriber: Paladox.Mar 1 2017, 4:50 PM

Mentioned in SAL (#wikimedia-releng) [2017-03-01T17:11:30Z] <hashar> cleaned out Jenkins security matrix to drop users that are no more used/inexistent -- T69027