Page MenuHomePhabricator
Create Task
Maniphest T243391

Upgrade ncredir cluster to buster
Closed, ResolvedPublic
Actions

Description

ncredir instances are currently running stretch, the main blocker for this task is the lack of support for multiple stapling files on nginx. This has been solved on stretch with a custom patch (0600-stapling-multi-file.patch).

@BBlack IIRC we mentioned the possibility of dropping RSA support on ncredir, so this would render 0600-stapling-multi-file.patch unnecessary, and we could upgrade ncredir to buster and go back to the "stock" nginx.

Upgrade status:

  • ncredir1001
  • ncredir1002
  • ncredir1001
  • ncredir2002
  • ncredir3001
  • ncredir3002
  • ncredir4001
  • ncredir4002
  • ncredir5001
  • ncredir5002

Details

SubjectRepoBranchLines +/-
install_server: Reimage ncredir@eqiad as busteroperations/puppetproduction+0 -2
install_server: Reimage ncredir@codfw as busteroperations/puppetproduction+0 -2
install_server: Reimage ncredir@esams as busteroperations/puppetproduction+0 -2
install_server: Reimage ncredir@eqsin as busteroperations/puppetproduction+0 -2
install_server: Reimage ncredir@ulsfo as busteroperations/puppetproduction+0 -2
ncredir: Drop RSA supportoperations/puppetproduction+0 -3
Customize query in gerrit

Event Timeline

Vgutierrez created this task.Jan 22 2020, 11:54 AM
Restricted Application added a project: SRE. · View Herald TranscriptJan 22 2020, 11:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Vgutierrez triaged this task as Medium priority.Jan 22 2020, 11:55 AM
Vgutierrez added a comment.Feb 6 2020, 1:03 PM

@BBlack confirmed that we can drop RSA support on ncredir during the All Hands

gerritbot added a comment.Feb 6 2020, 1:05 PM

Change 570629 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ncredir: Drop RSA support

https://gerrit.wikimedia.org/r/570629

gerritbot added a project: Patch-For-Review.Feb 6 2020, 1:05 PM
gerritbot added a comment.Feb 6 2020, 2:23 PM

Change 570629 merged by Vgutierrez:
[operations/puppet@production] ncredir: Drop RSA support

https://gerrit.wikimedia.org/r/570629

gerritbot added a comment.Feb 6 2020, 2:47 PM

Change 570665 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage ncredir@ulsfo as buster

https://gerrit.wikimedia.org/r/570665

gerritbot added a comment.Feb 6 2020, 2:53 PM

Change 570665 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage ncredir@ulsfo as buster

https://gerrit.wikimedia.org/r/570665

Stashbot added a comment.Feb 6 2020, 2:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T14:56:42Z] <vgutierrez> depool and reimage ncredir4002 as buster - T243391

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 3:11 PM
Stashbot added a comment.Feb 6 2020, 3:28 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T15:28:22Z] <vgutierrez> pooling ncredir4002 running buster - T243391

Stashbot added a comment.Feb 6 2020, 3:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T15:30:30Z] <vgutierrez> depool & reimage ncredir4001 as buster - T243391

Vgutierrez updated the task description. (Show Details)Feb 6 2020, 3:36 PM
Stashbot added a comment.Feb 6 2020, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T15:56:30Z] <vgutierrez> pooling ncredir4001 running buster - T243391

gerritbot added a comment.Feb 6 2020, 4:00 PM

Change 570678 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage ncredir@eqsin as buster

https://gerrit.wikimedia.org/r/570678

gerritbot added a project: Patch-For-Review.Feb 6 2020, 4:00 PM
gerritbot added a comment.Feb 6 2020, 4:04 PM

Change 570678 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage ncredir@eqsin as buster

https://gerrit.wikimedia.org/r/570678

Stashbot added a comment.Feb 6 2020, 4:07 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T16:07:40Z] <vgutierrez> depool and reimage ncredir5002 as buster - T243391

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 4:12 PM
Stashbot added a comment.Feb 6 2020, 4:49 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T16:49:12Z] <vgutierrez> pooling ncredir5002 running buster - T243391

Vgutierrez updated the task description. (Show Details)Feb 6 2020, 4:49 PM
Stashbot added a comment.Feb 7 2020, 10:23 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T10:23:46Z] <vgutierrez> depool and reimage ncredir5001 as buster - T243391

Stashbot added a comment.Feb 7 2020, 11:25 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T11:25:22Z] <vgutierrez> pooling ncredir5001 running buster - T243391

Vgutierrez updated the task description. (Show Details)Feb 7 2020, 11:26 AM
gerritbot added a comment.Feb 7 2020, 12:44 PM

Change 570883 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage ncredir@esams as buster

https://gerrit.wikimedia.org/r/570883

gerritbot added a project: Patch-For-Review.Feb 7 2020, 12:44 PM

Change 570883 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage ncredir@esams as buster

https://gerrit.wikimedia.org/r/570883

Stashbot added a comment.Feb 7 2020, 12:51 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T12:51:50Z] <vgutierrez> depool and reimage ncredir3002 as buster - T243391

Maintenance_bot removed a project: Patch-For-Review.Feb 7 2020, 1:11 PM
Stashbot added a comment.Feb 7 2020, 2:23 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T14:23:10Z] <vgutierrez> pooling ncredir3002 running buster - T243391

Stashbot added a comment.Feb 7 2020, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T14:33:56Z] <vgutierrez> depool and reimage ncredir3001 as buster - T243391

Stashbot added a comment.Feb 7 2020, 3:20 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T15:20:50Z] <vgutierrez> pooling ncredir3001 running buster - T243391

Vgutierrez updated the task description. (Show Details)Feb 7 2020, 3:29 PM
Vgutierrez moved this task from Backlog to TLS on the Traffic board.
gerritbot added a comment.Feb 10 2020, 6:50 AM

Change 571117 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage ncredir@codfw as buster

https://gerrit.wikimedia.org/r/571117

gerritbot added a project: Patch-For-Review.Feb 10 2020, 6:50 AM

Change 571117 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage ncredir@codfw as buster

https://gerrit.wikimedia.org/r/571117

Stashbot added a comment.Feb 10 2020, 6:55 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T06:55:19Z] <vgutierrez> depool ncredir2002 and reimage as buster - T243391

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2020, 7:10 AM
Stashbot added a comment.Feb 10 2020, 8:43 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T08:43:50Z] <vgutierrez> pooling ncredir2002 with buster - T243391

Stashbot added a comment.Feb 10 2020, 10:31 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T10:31:00Z] <vgutierrez> depool ncredir2001 and reimage as buster - T243391

Stashbot added a comment.Feb 10 2020, 11:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T11:07:25Z] <vgutierrez> pooling ncredir2001 with buster - T243391

Vgutierrez updated the task description. (Show Details)Feb 10 2020, 11:09 AM
gerritbot added a comment.Feb 10 2020, 11:10 AM

Change 571263 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage ncredir@eqiad as buster

https://gerrit.wikimedia.org/r/571263

gerritbot added a project: Patch-For-Review.Feb 10 2020, 11:10 AM

Change 571263 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage ncredir@eqiad as buster

https://gerrit.wikimedia.org/r/571263

Stashbot added a comment.Feb 10 2020, 11:38 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T11:38:49Z] <vgutierrez> depool ncredir1002 and reimage as buster - T243391

Stashbot added a comment.Feb 10 2020, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T11:57:26Z] <vgutierrez> pooling ncredir1002 with buster - T243391

Stashbot added a comment.Feb 10 2020, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T11:57:47Z] <vgutierrez> depool ncredir1001 and reimage as buster - T243391

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2020, 12:11 PM
Stashbot added a comment.Feb 10 2020, 12:23 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T12:23:07Z] <vgutierrez> pooling ncredir1001 with buster - T243391

Vgutierrez closed this task as Resolved.Feb 10 2020, 12:24 PM
Vgutierrez claimed this task.
Vgutierrez updated the task description. (Show Details)
MoritzMuehlenhoff reopened this task as Open.Mar 11 2020, 9:30 AM

I ran the OS upgrade tracking script and noticed that ncredir5002 is still on Buster, reopening

Stashbot added a comment.Mar 11 2020, 9:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-11T09:46:56Z] <vgutierrez> depool and reimage ncredir5002 with buster - T243391

Stashbot added a comment.Mar 11 2020, 10:42 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-11T10:42:24Z] <vgutierrez> pool ncredir5002 - T243391

Vgutierrez closed this task as Resolved.Mar 11 2020, 10:44 AM
vgutierrez@cumin1001:~$ sudo -i cumin 'A:ncredir' 'cat /etc/debian_version'
10 hosts will be targeted:
ncredir[2001-2002].codfw.wmnet,ncredir[1001-1002].eqiad.wmnet,ncredir[5001-5002].eqsin.wmnet,ncredir[3001-3002].esams.wmnet,ncredir[4001-4002].ulsfo.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(10) ncredir[2001-2002].codfw.wmnet,ncredir[1001-1002].eqiad.wmnet,ncredir[5001-5002].eqsin.wmnet,ncredir[3001-3002].esams.wmnet,ncredir[4001-4002].ulsfo.wmnet
----- OUTPUT of 'cat /etc/debian_version' -----
10.3
================

thanks for double checking @MoritzMuehlenhoff

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL