Page MenuHomePhabricator

Upgrade cache cluster to debian buster
Closed, ResolvedPublic

Description

We are currently blocked on several tasks like rolling out TLS 1.3 till our caching cluster is upgraded to buster.

Upgrade status:

  • eqiad
    • text
      • cp1075
      • cp1077
      • cp1079
      • cp1081
      • cp1083
      • cp1085
      • cp1087
      • cp1089
    • upload
      • cp1076
      • cp1078
      • cp1080
      • cp1082
      • cp1084
      • cp1086
      • cp1088
      • cp1090
  • codfw
    • text
      • cp2001
      • cp2004
      • cp2006
      • cp2007
      • cp2010
      • cp2012
      • cp2013
      • cp2016
      • cp2019
      • cp2023
    • upload
      • cp2002
      • cp2005
      • cp2008
      • cp2011
      • cp2014
      • cp2017
      • cp2018
      • cp2020
      • cp2022
      • cp2024
      • cp2025
      • cp2026
  • esams
    • text
      • cp3050
      • cp3052
      • cp3054
      • cp3056
      • cp3058
      • cp3060
      • cp3062
      • cp3064
    • upload
      • cp3051
      • cp3053
      • cp3055
      • cp3057
      • cp3059
      • cp3061
      • cp3063
      • cp3065
  • ulsfo
    • text
      • cp4027
      • cp4028
      • cp4029
      • cp4030
      • cp4031
      • cp4032
    • upload
      • cp4021
      • cp4022
      • cp4023
      • cp4024
      • cp4025
      • cp4026
  • eqsin
    • text
      • cp5006
      • cp5007
      • cp5008
      • cp5009
      • cp5010
      • cp5011
      • cp5012
    • upload
      • cp5001
      • cp5002
      • cp5003
      • cp5004
      • cp5005
      • cp5006

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -16
operations/puppetproduction+0 -26
operations/puppetproduction+0 -15
operations/puppetproduction+0 -10
operations/puppetproduction+0 -5
operations/puppetproduction+0 -6
operations/debs/trafficservermaster+23 -1
operations/puppetproduction+0 -1
operations/puppetproduction+0 -1
operations/puppetproduction+0 -1
operations/puppetproduction+0 -1
operations/puppetproduction+3 -1
operations/puppetproduction+2 -1
operations/puppetproduction+6 -35
operations/puppetproduction+2 -0
operations/software/fifo-log-demuxmaster+6 -0
operations/software/varnish/vhtcpddebian+8 -1
operations/debs/trafficservermaster+16 -8
operations/software/varnish/libvmod-tbfdebian+24 -13
operations/software/varnish/libvmod-netmapperdebian+11 -2
operations/software/varnish/libvmod-re2debian+10 -2
operations/software/varnish/varnishkafkadebian+14 -3
operations/debs/varnish4debian-wmf+10 -2
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2020-02-12T15:02:08Z] <vgutierrez> depool cp20[07,17] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2007.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002121502_vgutierrez_122677_cp2007_codfw_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2017.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002121512_vgutierrez_127278_cp2017_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cp2007.codfw.wmnet']

Of which those FAILED:

['cp2007.codfw.wmnet']

Completed auto-reimage of hosts:

['cp2017.codfw.wmnet']

Of which those FAILED:

['cp2017.codfw.wmnet']

Mentioned in SAL (#wikimedia-operations) [2020-02-12T15:48:33Z] <vgutierrez> pool cp20[07,17] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-12T15:50:13Z] <vgutierrez> depool cp20[06,14] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2006.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002121550_vgutierrez_137927_cp2006_codfw_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2014.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002121553_vgutierrez_138370_cp2014_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cp2006.codfw.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['cp2014.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-12T16:52:14Z] <vgutierrez> pool cp20[06,14] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T05:09:44Z] <vgutierrez> depool cp20[04,11] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2004.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130511_vgutierrez_23193_cp2004_codfw_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2011.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130513_vgutierrez_23547_cp2011_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cp2004.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T05:34:57Z] <vgutierrez> pool cp2004 running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T05:35:34Z] <vgutierrez> depool cp2001 and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2001.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130535_vgutierrez_29443_cp2001_codfw_wmnet.log.

Completed auto-reimage of hosts:

['cp2011.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T05:37:47Z] <vgutierrez> pool cp2011 running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T05:38:24Z] <vgutierrez> depool cp2008 and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2008.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130538_vgutierrez_29767_cp2008_codfw_wmnet.log.

Change 571862 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reiamge {upload,text}@eqiad as buster

https://gerrit.wikimedia.org/r/571862

Completed auto-reimage of hosts:

['cp2001.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T06:03:56Z] <vgutierrez> pool cp20[01,08] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T06:04:39Z] <vgutierrez> depool cp20[02,05] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2002.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130605_vgutierrez_37262_cp2002_codfw_wmnet.log.

Change 571862 merged by Vgutierrez:
[operations/puppet@production] install_server: Reiamge {upload,text}@eqiad as buster

https://gerrit.wikimedia.org/r/571862

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2005.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130608_vgutierrez_37846_cp2005_codfw_wmnet.log.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T06:11:51Z] <vgutierrez> depool cp10[89,90] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1089.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130612_vgutierrez_38556_cp1089_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1090.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130616_vgutierrez_39272_cp1090_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp2002.codfw.wmnet']

Of which those FAILED:

['cp2002.codfw.wmnet']

Completed auto-reimage of hosts:

['cp2005.codfw.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['cp1089.eqiad.wmnet']

Of which those FAILED:

['cp1089.eqiad.wmnet']

Completed auto-reimage of hosts:

['cp1090.eqiad.wmnet']

Of which those FAILED:

['cp1090.eqiad.wmnet']

Mentioned in SAL (#wikimedia-operations) [2020-02-13T06:49:39Z] <vgutierrez> pool cp20[02,05] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T07:01:59Z] <vgutierrez> pool cp10[89,90] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T07:03:12Z] <vgutierrez> depool cp10[87,88] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1087.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130703_vgutierrez_52539_cp1087_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1088.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130708_vgutierrez_55107_cp1088_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp1087.eqiad.wmnet']

Of which those FAILED:

['cp1087.eqiad.wmnet']

Completed auto-reimage of hosts:

['cp1088.eqiad.wmnet']

Of which those FAILED:

['cp1088.eqiad.wmnet']

Mentioned in SAL (#wikimedia-operations) [2020-02-13T07:49:50Z] <vgutierrez> pool cp10[87,88] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T07:57:08Z] <vgutierrez> depool cp10[85,86] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1085.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130757_vgutierrez_66095_cp1085_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1086.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130757_vgutierrez_66140_cp1086_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp1085.eqiad.wmnet']

Of which those FAILED:

['cp1085.eqiad.wmnet']

Completed auto-reimage of hosts:

['cp1086.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T09:45:06Z] <vgutierrez> pool cp10[85,86] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T09:45:54Z] <vgutierrez> depool cp10[83,84] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1083.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130946_vgutierrez_88180_cp1083_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1084.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002130948_vgutierrez_88590_cp1084_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp1084.eqiad.wmnet']

Of which those FAILED:

['cp1084.eqiad.wmnet']

Completed auto-reimage of hosts:

['cp1083.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T10:21:34Z] <vgutierrez> pool cp10[83,84] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T10:24:13Z] <vgutierrez> depool cp10[81,82] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1081.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131024_vgutierrez_98289_cp1081_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1082.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131026_vgutierrez_98684_cp1082_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp1081.eqiad.wmnet']

Of which those FAILED:

['cp1081.eqiad.wmnet']

Completed auto-reimage of hosts:

['cp1082.eqiad.wmnet']

Of which those FAILED:

['cp1082.eqiad.wmnet']

Mentioned in SAL (#wikimedia-operations) [2020-02-13T11:02:53Z] <vgutierrez> pool cp10[81,82] and reimage as buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T11:16:37Z] <vgutierrez> depool cp10[79,80] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1079.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131117_vgutierrez_109648_cp1079_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1080.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131119_vgutierrez_110041_cp1080_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp1079.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['cp1080.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T11:52:13Z] <vgutierrez> pool cp10[79,80] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T11:53:43Z] <vgutierrez> depool cp10[77,78] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1077.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131154_vgutierrez_118940_cp1077_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1078.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131156_vgutierrez_119341_cp1078_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp1077.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['cp1078.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T12:28:14Z] <vgutierrez> pool cp10[77,78] running buster - T242093

Mentioned in SAL (#wikimedia-operations) [2020-02-13T12:29:00Z] <vgutierrez> depool cp10[75,76] and reimage as buster - T242093

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1075.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131229_vgutierrez_127126_cp1075_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp1076.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202002131231_vgutierrez_127380_cp1076_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cp1075.eqiad.wmnet']

and were ALL successful.

Completed auto-reimage of hosts:

['cp1076.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-02-13T13:00:51Z] <vgutierrez> pool cp10[75,76] running buster - T242093

Vgutierrez claimed this task.
Vgutierrez updated the task description. (Show Details)
vgutierrez@cumin1001:~$ sudo -i cumin 'A:cp' 'cat /etc/debian_version'
78 hosts will be targeted:
cp[2001-2002,2004-2008,2010-2014,2016-2020,2022-2026].codfw.wmnet,cp[1075-1090].eqiad.wmnet,cp[5001-5012].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4032].ulsfo.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(78) cp[2001-2002,2004-2008,2010-2014,2016-2020,2022-2026].codfw.wmnet,cp[1075-1090].eqiad.wmnet,cp[5001-5012].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4032].ulsfo.wmnet
----- OUTPUT of 'cat /etc/debian_version' -----
10.3