Currently our puppet compilers compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs are Jessie hosts. Until somewhat recently upgrading was blocked by dependency on an etcd package for Stretch. Now that this is available let's plan to upgrade them.
This also is a good time to update the cloud/labs project name (probably by creating a new project) to something like "puppet-diffs" since the current name "puppet3-diffs" has become outdated.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| puppet_compiler: temporarily proxy two project names with nginx | operations/puppet | production | +10 -3 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | herron | T191438 Upgrade Puppet compilers to Stretch | |||
| Resolved | aborrero | T197815 Request creation of puppet-diffs VPS project | |||
| Resolved | aborrero | T204532 Request removal of puppet3-diffs VPS project |
Took a stab at building a stretch compiler and ran into this when attempting to start etcd...
Apr 04 15:32:51 compiler-keith-stretch1 systemd[1]: Starting etcd... Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://127.0.0.1:2379 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/puppet.eqiad.wmflabs Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1000 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=100 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://127.0.0.1:2380 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_INITIAL_CLUSTER=compiler-keith-stretch1=http://127.0.0.1:2380 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://:2379 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://127.0.0.1:2380 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_NAME=compiler-keith-stretch1 Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: error verifying flags, expected IP in URL for binding (http://:2379). See 'etcd --help'. Apr 04 15:32:51 compiler-keith-stretch1 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE Apr 04 15:32:51 compiler-keith-stretch1 systemd[1]: Failed to start etcd.
updating ETCD_LISTEN_CLIENT_URLS=http://:2379 to ETCD_LISTEN_CLIENT_URLS=http://127.0.0.1:2379 in /lib/systemd/system/etcd.service fixes it, but need to figure out why that's happening (the host value seems to be empty)
The puppet-compiler itself is working on Stretch though, even when etcd is not running.
As explained in T202717, this task should be prioritized, so I am raising its priority to High.
Change 456287 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet_compiler: temporarily proxy two project names with nginx
Stretch compilers compiler1001.puppet-diffs.eqiad.wmflabs and compiler1002.puppet-diffs.eqiad.wmflabs are up and running, and a few test diffs ran locally look good.
Next step is to add the new compilers to the https://puppet-compiler.wmflabs.org reverse proxy config, and then we can enable them in Jenkins.
Regarding the patch (https://gerrit.wikimedia.org/r/456287) -- I'm thinking of temporarily proxying requests to the new stretch hosts through the existing nginx reverse proxy on compiler02.puppet3-diffs.eqiad.wmflabs. That will let us add the new Stretch compilers to Jenkins without having to remove the Jessie compilers right away. Then, once we're sure it all is working as expected, we can move the web proxy config in horizon over to the new 'puppet-diffs' project and retire the puppet3-diffs project.
Change 456287 merged by Herron:
[operations/puppet@production] puppet_compiler: temporarily proxy two project names with nginx
The config to proxy https://puppet-compiler.wmflabs.org across both jessie and stretch compiler hosts (4 hosts total) is live and has been stable for a few days.
Next step is to add and enable hosts compiler100[12].puppet-diffs.eqiad.wmflabs to the operations-puppet-catalog-compiler project in Jenkins.
Release-Engineering-Team could we coordinate a time to update the Jenkins config this week?
Stretch compilers compiler100[12].puppet-diffs.eqiad.wmflabs are now live in the operations-puppet-catalog-compiler Jenkins project, and the old Jessie compilers compiler0[23].puppet3-diffs.eqiad.wmflabs have been marked offline (thanks @Reedy!)
For the time being web requests are still being proxied through compiler02 via the web proxy config on the puppet3-diffs project.
Next step is to migrate the http://puppet-compiler.wmflabs.org web proxy from puppet3-diffs to puppet-diffs.
Cloud-VPS is there anything else involved in moving a web proxy from one project to another besides deleting it from the old project and creating it in the new project?
Mentioned in SAL (#wikimedia-operations) [2018-09-13T19:24:39Z] <herron> moving puppet-compiler.wmlabs.org proxy from puppet3-diffs to puppet-diffs project T191438
Mentioned in SAL (#wikimedia-operations) [2018-09-13T19:31:42Z] <herron> shutting down old puppet compiler instances in puppet3-diffs project T191438
Can we now delete the old instances and remove them from the Jenkins config? In Jenkins we just have to delete:
The old instances have been powered off for a few days now and afaict there have been no problems. I think it's safe to remove them.
<hasharAway> !log jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deleted
Only think left to do now is remove the old puppet3-diffs project which is being tracked in T204532. Resolving!