Page MenuHomePhabricator

Upgrade Puppet compilers to Stretch
Closed, ResolvedPublic

Description

Currently our puppet compilers compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs are Jessie hosts. Until somewhat recently upgrading was blocked by dependency on an etcd package for Stretch. Now that this is available let's plan to upgrade them.

This also is a good time to update the cloud/labs project name (probably by creating a new project) to something like "puppet-diffs" since the current name "puppet3-diffs" has become outdated.

Event Timeline

herron triaged this task as Medium priority.Apr 4 2018, 4:15 PM
herron created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2018, 4:15 PM
herron added a comment.Apr 4 2018, 4:40 PM

Took a stab at building a stretch compiler and ran into this when attempting to start etcd...

Apr 04 15:32:51 compiler-keith-stretch1 systemd[1]: Starting etcd...
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://127.0.0.1:2379
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/puppet.eqiad.wmflabs
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1000
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=100
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://127.0.0.1:2380
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_INITIAL_CLUSTER=compiler-keith-stretch1=http://127.0.0.1:2380
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://:2379
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://127.0.0.1:2380
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: recognized and used environment variable ETCD_NAME=compiler-keith-stretch1
Apr 04 15:32:51 compiler-keith-stretch1 etcd[5148]: error verifying flags, expected IP in URL for binding (http://:2379). See 'etcd --help'.
Apr 04 15:32:51 compiler-keith-stretch1 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 15:32:51 compiler-keith-stretch1 systemd[1]: Failed to start etcd.

updating ETCD_LISTEN_CLIENT_URLS=http://:2379 to ETCD_LISTEN_CLIENT_URLS=http://127.0.0.1:2379 in /lib/systemd/system/etcd.service fixes it, but need to figure out why that's happening (the host value seems to be empty)

The puppet-compiler itself is working on Stretch though, even when etcd is not running.

herron moved this task from Backlog to Working on on the User-herron board.Jun 28 2018, 6:58 PM
elukey raised the priority of this task from Medium to High.Aug 24 2018, 10:18 AM
elukey added subscribers: elukey, MoritzMuehlenhoff.

As explained in T202717, this task should be prioritized, so I am raising its priority to High.

Change 456287 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet_compiler: temporarily proxy two project names with nginx

https://gerrit.wikimedia.org/r/456287

Stretch compilers compiler1001.puppet-diffs.eqiad.wmflabs and compiler1002.puppet-diffs.eqiad.wmflabs are up and running, and a few test diffs ran locally look good.

Next step is to add the new compilers to the https://puppet-compiler.wmflabs.org reverse proxy config, and then we can enable them in Jenkins.

Regarding the patch (https://gerrit.wikimedia.org/r/456287) -- I'm thinking of temporarily proxying requests to the new stretch hosts through the existing nginx reverse proxy on compiler02.puppet3-diffs.eqiad.wmflabs. That will let us add the new Stretch compilers to Jenkins without having to remove the Jessie compilers right away. Then, once we're sure it all is working as expected, we can move the web proxy config in horizon over to the new 'puppet-diffs' project and retire the puppet3-diffs project.

Change 456287 merged by Herron:
[operations/puppet@production] puppet_compiler: temporarily proxy two project names with nginx

https://gerrit.wikimedia.org/r/456287

The config to proxy https://puppet-compiler.wmflabs.org across both jessie and stretch compiler hosts (4 hosts total) is live and has been stable for a few days.

Next step is to add and enable hosts compiler100[12].puppet-diffs.eqiad.wmflabs to the operations-puppet-catalog-compiler project in Jenkins.

Release-Engineering-Team could we coordinate a time to update the Jenkins config this week?

herron added a subscriber: Reedy.

Stretch compilers compiler100[12].puppet-diffs.eqiad.wmflabs are now live in the operations-puppet-catalog-compiler Jenkins project, and the old Jessie compilers compiler0[23].puppet3-diffs.eqiad.wmflabs have been marked offline (thanks @Reedy!)

For the time being web requests are still being proxied through compiler02 via the web proxy config on the puppet3-diffs project.

Next step is to migrate the http://puppet-compiler.wmflabs.org web proxy from puppet3-diffs to puppet-diffs.

Cloud-VPS is there anything else involved in moving a web proxy from one project to another besides deleting it from the old project and creating it in the new project?

Krenair added a subscriber: Krenair.Sep 6 2018, 8:27 PM

Cloud-VPS is there anything else involved in moving a web proxy from one project to another besides deleting it from the old project and creating it in the new project?

I don't think so, I think you just have to hope no-one nicks it in the process.

Mentioned in SAL (#wikimedia-operations) [2018-09-13T19:24:39Z] <herron> moving puppet-compiler.wmlabs.org proxy from puppet3-diffs to puppet-diffs project T191438

Mentioned in SAL (#wikimedia-operations) [2018-09-13T19:31:42Z] <herron> shutting down old puppet compiler instances in puppet3-diffs project T191438

The old instances have been powered off for a few days now and afaict there have been no problems. I think it's safe to remove them.

<hasharAway> !log jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deleted
herron closed this task as Resolved.Sep 17 2018, 7:58 PM
herron claimed this task.

Only think left to do now is remove the old puppet3-diffs project which is being tracked in T204532. Resolving!