Page MenuHomePhabricator

Upgrade cache servers to stretch
Closed, ResolvedPublic

Description

All cache hosts are currently running jessie and need to be upgraded to stretch.

Packages needed in stretch-wikimedia:

  • varnish
  • varnish-modules
  • libvmod-tbf
  • vhtcpd
  • varnishkafka
  • libvmod-netmapper (liburcu2 -> liburcu4)
  • libvmod-re2 (libre2-1 -> libre2-3)

traffic-text-stretch.traffic.eqiad.wmflabs is running stretch and has the cache::text role applied. It can be used to test the upgrade.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts:

cp3008.esams.wmnet

The log can be found in /var/log/wmf-auto-reimage/201807310939_ema_27179_cp3008_esams_wmnet.log.

Change 449426 merged by jenkins-bot:
[integration/config@master] Use backports for varnishkafka

https://gerrit.wikimedia.org/r/449426

Change 449433 had a related patch set uploaded (by Ema; owner: Ema):
[integration/config@master] varnishkafka: use debian-glue (voting)

https://gerrit.wikimedia.org/r/449433

Change 449433 merged by jenkins-bot:
[integration/config@master] varnishkafka: use debian-glue (voting)

https://gerrit.wikimedia.org/r/449433

Completed auto-reimage of hosts:

['cp3008.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts:

cp3007.esams.wmnet

The log can be found in /var/log/wmf-auto-reimage/201807311019_ema_3382_cp3007_esams_wmnet.log.

Completed auto-reimage of hosts:

['cp3007.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts:

cp3010.esams.wmnet

The log can be found in /var/log/wmf-auto-reimage/201807311059_ema_12671_cp3010_esams_wmnet.log.

Completed auto-reimage of hosts:

['cp3010.esams.wmnet']

and were ALL successful.

Change 449451 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp-misc_eqiad: upgrade to stretch

https://gerrit.wikimedia.org/r/449451

Change 449451 merged by Ema:
[operations/puppet@production] cp-misc_eqiad: upgrade to stretch

https://gerrit.wikimedia.org/r/449451

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp1045.eqiad.wmnet', 'cp1051.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201807311305_ema_13646.log.

Completed auto-reimage of hosts:

['cp1045.eqiad.wmnet', 'cp1051.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp1058.eqiad.wmnet', 'cp1061.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201807311340_ema_22414.log.

Completed auto-reimage of hosts:

['cp1058.eqiad.wmnet']

Of which those FAILED:

['cp1058.eqiad.wmnet']

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp1058.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201807311425_ema_32163.log.

Completed auto-reimage of hosts:

['cp1058.eqiad.wmnet']

and were ALL successful.

Change 449470 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp3031 (text), cp3044 (upload): upgrade to stretch

https://gerrit.wikimedia.org/r/449470

Change 449470 merged by Ema:
[operations/puppet@production] cp3031 (text), cp3044 (upload): upgrade to stretch

https://gerrit.wikimedia.org/r/449470

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3031.esams.wmnet', 'cp3044.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201807311509_ema_11545.log.

Completed auto-reimage of hosts:

['cp3031.esams.wmnet']

Of which those FAILED:

['cp3031.esams.wmnet']

Mentioned in SAL (#wikimedia-operations) [2018-08-01T09:41:46Z] <ema> cp3044 (upload) repooled after upgrade to stretch T200445

Change 449716 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp-eqsin: upgrade to stretch

https://gerrit.wikimedia.org/r/449716

Change 449716 merged by Ema:
[operations/puppet@production] cp-eqsin: upgrade to stretch

https://gerrit.wikimedia.org/r/449716

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp5006.eqsin.wmnet', 'cp5007.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808011535_ema_20424.log.

Completed auto-reimage of hosts:

['cp5007.eqsin.wmnet', 'cp5006.eqsin.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts:

['cp5008.eqsin.wmnet', 'cp5002.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808020824_ema_8652.log.

Completed auto-reimage of hosts:

['cp5002.eqsin.wmnet', 'cp5008.eqsin.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

cp5003.eqsin.wmnet

The log can be found in /var/log/wmf-auto-reimage/201808060952_ema_27699_cp5003_eqsin_wmnet.log.

Completed auto-reimage of hosts:

['cp5003.eqsin.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp5004.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808061339_ema_22611.log.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

cp5009.eqsin.wmnet

The log can be found in /var/log/wmf-auto-reimage/201808061428_ema_1461_cp5009_eqsin_wmnet.log.

Completed auto-reimage of hosts:

['cp5004.eqsin.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['cp5009.eqsin.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

cp5011.eqsin.wmnet

The log can be found in /var/log/wmf-auto-reimage/201808071615_ema_7750_cp5011_eqsin_wmnet.log.

Completed auto-reimage of hosts:

['cp5011.eqsin.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

cp5012.eqsin.wmnet

The log can be found in /var/log/wmf-auto-reimage/201808071706_ema_21743_cp5012_eqsin_wmnet.log.

Completed auto-reimage of hosts:

['cp5012.eqsin.wmnet']

Of which those FAILED:

['cp5012.eqsin.wmnet']

Change 451305 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp2022: upgrade to stretch

https://gerrit.wikimedia.org/r/451305

Change 451305 merged by Ema:
[operations/puppet@production] cp2022: upgrade to stretch

https://gerrit.wikimedia.org/r/451305

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

cp2022.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201808081215_ema_4705_cp2022_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cp2022.codfw.wmnet']

Of which those FAILED:

['cp2022.codfw.wmnet']

Change 451324 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] cpNNNN: remove all jessie installer settings

https://gerrit.wikimedia.org/r/451324

Change 451324 merged by BBlack:
[operations/puppet@production] cpNNNN: remove all jessie installer settings

https://gerrit.wikimedia.org/r/451324

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

cp5005.eqsin.wmnet

The log can be found in /var/log/wmf-auto-reimage/201808141322_ema_10769_cp5005_eqsin_wmnet.log.

Completed auto-reimage of hosts:

['cp5005.eqsin.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp5010.eqsin.wmnet', 'cp4021.ulsfo.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808150646_ema_32748.log.

Completed auto-reimage of hosts:

['cp4021.ulsfo.wmnet']

Of which those FAILED:

['cp4021.ulsfo.wmnet']

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3032.esams.wmnet', 'cp4022.ulsfo.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808150815_ema_20369.log.

Completed auto-reimage of hosts:

['cp3032.esams.wmnet', 'cp4022.ulsfo.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3049.esams.wmnet', 'cp2001.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808160924_ema_14276.log.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

cp4023.ulsfo.wmnet

The log can be found in /var/log/wmf-auto-reimage/201808160925_ema_14682_cp4023_ulsfo_wmnet.log.

Completed auto-reimage of hosts:

['cp2001.codfw.wmnet', 'cp3049.esams.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['cp4023.ulsfo.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp4024.ulsfo.wmnet', 'cp2002.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808161422_ema_14677.log.

Completed auto-reimage of hosts:

['cp4024.ulsfo.wmnet']

Of which those FAILED:

['cp4024.ulsfo.wmnet']

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp4025.ulsfo.wmnet', 'cp2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808171455_ema_9263.log.

Completed auto-reimage of hosts:

['cp2004.codfw.wmnet', 'cp4025.ulsfo.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp4032.ulsfo.wmnet', 'cp2005.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808200727_ema_24974.log.

Completed auto-reimage of hosts:

['cp2005.codfw.wmnet', 'cp4032.ulsfo.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp4031.ulsfo.wmnet', 'cp2026.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808201051_ema_31995.log.

Completed auto-reimage of hosts:

['cp2026.codfw.wmnet', 'cp4031.ulsfo.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3045.esams.wmnet', 'cp4026.ulsfo.wmnet', 'cp5001.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808210915_ema_29668.log.

Completed auto-reimage of hosts:

['cp4026.ulsfo.wmnet']

Of which those FAILED:

['cp4026.ulsfo.wmnet']

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp4027.ulsfo.wmnet', 'cp3042.esams.wmnet', 'cp2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808211006_ema_11124.log.

Completed auto-reimage of hosts:

['cp4027.ulsfo.wmnet']

Of which those FAILED:

['cp4027.ulsfo.wmnet']

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp2008.codfw.wmnet', 'cp2023.codfw.wmnet', 'cp3047.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808211149_ema_1291.log.

Completed auto-reimage of hosts:

['cp2008.codfw.wmnet', 'cp2023.codfw.wmnet', 'cp3047.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3036.esams.wmnet', 'cp4028.ulsfo.wmnet', 'cp2016.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808211256_ema_18256.log.

Completed auto-reimage of hosts:

['cp4028.ulsfo.wmnet']

Of which those FAILED:

['cp4028.ulsfo.wmnet']

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp2010.codfw.wmnet', 'cp4030.ulsfo.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808231036_ema_11516.log.

Completed auto-reimage of hosts:

['cp2010.codfw.wmnet', 'cp4030.ulsfo.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp2011.codfw.wmnet', 'cp4029.ulsfo.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808231333_ema_4992.log.

Completed auto-reimage of hosts:

['cp4029.ulsfo.wmnet']

Of which those FAILED:

['cp4029.ulsfo.wmnet']

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp2013.codfw.wmnet', 'cp3030.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808241039_ema_16794.log.

Mentioned in SAL (#wikimedia-operations) [2018-08-24T11:19:44Z] <ema> powercycle cp3030, stuck rebooting T200445

Completed auto-reimage of hosts:

['cp2013.codfw.wmnet', 'cp3030.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp2014.codfw.wmnet', 'cp3033.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808241201_ema_26878.log.

Completed auto-reimage of hosts:

['cp2014.codfw.wmnet', 'cp3033.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts:

['cp2019.codfw.wmnet', 'cp2017.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808270824_ema_26834.log.

Completed auto-reimage of hosts:

['cp2019.codfw.wmnet', 'cp2017.codfw.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts:

['cp3034.esams.wmnet', 'cp2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808270918_ema_9500.log.

Completed auto-reimage of hosts:

['cp2020.codfw.wmnet', 'cp3034.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts:

['cp3035.esams.wmnet', 'cp2024.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808271034_ema_28550.log.

Completed auto-reimage of hosts:

['cp2024.codfw.wmnet', 'cp3035.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3037.esams.wmnet', 'cp3041.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808271200_ema_18869.log.

Completed auto-reimage of hosts:

['cp3037.esams.wmnet', 'cp3041.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3038.esams.wmnet', 'cp3040.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808280548_ema_12392.log.

Completed auto-reimage of hosts:

['cp3038.esams.wmnet', 'cp3040.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3039.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808280727_ema_29676.log.

Completed auto-reimage of hosts:

['cp3039.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3043.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808280858_ema_12461.log.

Completed auto-reimage of hosts:

['cp3043.esams.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts:

['cp3046.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808280934_ema_31349.log.

Completed auto-reimage of hosts:

['cp3046.esams.wmnet']

and were ALL successful.

ema closed this task as Resolved.Aug 28 2018, 10:11 AM
ema claimed this task.

Done! \o/

The only cache host running jessie is cp1008, which will be replaced soon by cp1099: T202966.