Page MenuHomePhabricator
Create Task
Maniphest T308214

Migrate the IDPs to Bullseye
Closed, ResolvedPublic
Actions

Description

  • provision new idp-test nodes with Bullseye
  • Build new cas debs for Bullseye
  • Test everything on the new test cluster after failing over the idp-test CNAME
  • Create new idp nodes with Bullseye
  • Failover the idp CNAME
  • Remove old Buster nodes

Details

SubjectRepoBranchLines +/-
acme_chief: Remove old buster IDP hostsoperations/puppetproduction+0 -4
Switch idp1001/idp2001 to role(insetup)operations/puppetproduction+6 -15
profile::mariadb::ferm_misc: Remove old buster IDP nodesoperations/puppetproduction+1 -1
Failover idp.w.o to idp1002 (new Bullseye node)operations/dnsmaster+1 -1
Failover active IDP nodes to idp1002/idp2002operations/puppetproduction+2 -2
Remove Puppet references to idp-test1001/idp-test2001operations/puppetproduction+0 -11
Apply idp role to idp1002/idp2002operations/puppetproduction+2 -2
Remove old buster idp-test hosts, add new idp/bullseye ones from Ferm rulesoperations/puppetproduction+2 -2
Point idp-test to idp-test1002operations/dnsmaster+1 -1
idp-test: Point to the new Bullseye hostsoperations/puppetproduction+2 -2
idp::memcached: Only enable memcached_16 on Busteroperations/puppetproduction+1 -1
memcached: Untangle TLS/1.6 optionsoperations/puppetproduction+33 -8
Add repository component for TLS-enabled memcachedoperations/puppetproduction+1 -0
Only add component/memcached16 on Busteroperations/puppetproduction+6 -4
Allow new idp-test hosts in Ferm rulesoperations/puppetproduction+1 -1
idp: Remove component/idp-testoperations/puppetproduction+0 -9
Also switch idp-test1002 to idp_test roleoperations/puppetproduction+1 -1
Enable new Bullseye test IDPs in acmechief configoperations/puppetproduction+4 -0
Also add component/idp-test for Bullseyeoperations/puppetproduction+1 -0
Switch idp-test2002 to idp_test roleoperations/puppetproduction+1 -1
Add idp-test1002/2002operations/puppetproduction+20 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.May 12 2022, 8:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2022, 8:33 AM
gerritbot added a comment.May 12 2022, 11:17 AM

Change 791342 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add idp-test1002/2002

https://gerrit.wikimedia.org/r/791342

gerritbot added a project: Patch-For-Review.May 12 2022, 11:17 AM
gerritbot added a comment.May 12 2022, 11:36 AM

Change 791342 merged by Muehlenhoff:

[operations/puppet@production] Add idp-test1002/2002

https://gerrit.wikimedia.org/r/791342

Stashbot added a comment.May 13 2022, 10:55 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-13T10:55:37Z] <moritzm> installing idp-test2002 T308214

Maintenance_bot removed a project: Patch-For-Review.May 13 2022, 11:30 AM
Stashbot added a comment.May 13 2022, 11:40 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-13T11:40:58Z] <moritzm> installing idp-test1002 T308214

• Marostegui triaged this task as Medium priority.May 17 2022, 8:18 AM
gerritbot added a comment.May 20 2022, 6:19 AM

Change 793634 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch idp-test2002 to idp_test role

https://gerrit.wikimedia.org/r/793634

gerritbot added a project: Patch-For-Review.May 20 2022, 6:19 AM
gerritbot added a comment.May 20 2022, 7:25 AM

Change 793634 merged by Muehlenhoff:

[operations/puppet@production] Switch idp-test2002 to idp_test role

https://gerrit.wikimedia.org/r/793634

Maintenance_bot removed a project: Patch-For-Review.May 20 2022, 7:30 AM
gerritbot added a comment.May 20 2022, 10:58 AM

Change 793744 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only add component/memcached16 on Buster

https://gerrit.wikimedia.org/r/793744

gerritbot added a project: Patch-For-Review.May 20 2022, 10:58 AM
gerritbot added a comment.May 20 2022, 11:29 AM

Change 793751 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Also add component/idp-test for Bullseye

https://gerrit.wikimedia.org/r/793751

gerritbot added a comment.May 20 2022, 11:31 AM

Change 793751 merged by Muehlenhoff:

[operations/puppet@production] Also add component/idp-test for Bullseye

https://gerrit.wikimedia.org/r/793751

Stashbot added a comment.May 20 2022, 12:37 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-20T12:37:32Z] <moritzm> copy prometheus-mcrouter-exporter from buster-wikimedia to bullseye-wikimedia (needed for T308214)

gerritbot added a comment.May 20 2022, 12:44 PM

Change 793770 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable new Bullseye test IDPs in acmechief config

https://gerrit.wikimedia.org/r/793770

gerritbot added a comment.May 20 2022, 1:44 PM

Change 793783 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] idp: Remove component/idp-test

https://gerrit.wikimedia.org/r/793783

gerritbot added a comment.May 23 2022, 6:33 AM

Change 793770 merged by Muehlenhoff:

[operations/puppet@production] Enable new Bullseye test IDPs in acmechief config

https://gerrit.wikimedia.org/r/793770

gerritbot added a comment.May 23 2022, 6:51 AM

Change 796969 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Also switch idp-test1002 to idp_test role

https://gerrit.wikimedia.org/r/796969

gerritbot added a comment.May 23 2022, 7:04 AM

Change 796969 merged by Muehlenhoff:

[operations/puppet@production] Also switch idp-test1002 to idp_test role

https://gerrit.wikimedia.org/r/796969

gerritbot added a comment.May 23 2022, 8:52 AM

Change 793783 merged by Muehlenhoff:

[operations/puppet@production] idp: Remove component/idp-test

https://gerrit.wikimedia.org/r/793783

gerritbot added a comment.May 24 2022, 2:42 PM

Change 798709 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Allow new idp-test hosts in Ferm rules

https://gerrit.wikimedia.org/r/798709

gerritbot added a comment.May 25 2022, 8:51 AM

Change 798709 merged by Muehlenhoff:

[operations/puppet@production] Allow new idp-test hosts in Ferm rules

https://gerrit.wikimedia.org/r/798709

MoritzMuehlenhoff added a comment.May 25 2022, 10:00 AM

The IDPs needs a TLS-enabled build of the bullseye version of memcached, which was only enabled after the bullseye release (in 1.6.12). I'll create a separate component with a memcached build.

gerritbot added a comment.May 25 2022, 10:17 AM

Change 793744 merged by Muehlenhoff:

[operations/puppet@production] Only add component/memcached16 on Buster

https://gerrit.wikimedia.org/r/793744

gerritbot added a comment.May 25 2022, 10:21 AM

Change 799286 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add repository component for TLS-enabled memcached

https://gerrit.wikimedia.org/r/799286

gerritbot added a comment.May 25 2022, 10:32 AM

Change 799286 merged by Muehlenhoff:

[operations/puppet@production] Add repository component for TLS-enabled memcached

https://gerrit.wikimedia.org/r/799286

Stashbot added a comment.May 25 2022, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-25T13:15:46Z] <moritzm> imported memcached 1.6.9+dfsg-1+wmf11u1 to bullseye-wikimedia (TLS-enabled build) T308214

gerritbot added a comment.May 25 2022, 2:07 PM

Change 799348 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] memcached: Untangle TLS/1.6 options

https://gerrit.wikimedia.org/r/799348

gerritbot added a comment.May 25 2022, 3:08 PM

Change 799354 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] idp::memcached: Only enable memcached_16 on Buster

https://gerrit.wikimedia.org/r/799354

gerritbot added a comment.May 30 2022, 10:22 AM

Change 799348 merged by Muehlenhoff:

[operations/puppet@production] memcached: Untangle TLS/1.6 options

https://gerrit.wikimedia.org/r/799348

gerritbot added a comment.May 30 2022, 11:41 AM

Change 799354 merged by Muehlenhoff:

[operations/puppet@production] idp::memcached: Only enable memcached_16 on Buster

https://gerrit.wikimedia.org/r/799354

gerritbot added a comment.May 30 2022, 3:24 PM

Change 801402 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] idp-test: Point to the new Bullseye hosts

https://gerrit.wikimedia.org/r/801402

gerritbot added a comment.May 31 2022, 7:02 AM

Change 801624 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point idp-test to idp-test1002

https://gerrit.wikimedia.org/r/801624

gerritbot added a comment.May 31 2022, 9:13 AM

Change 801402 merged by Muehlenhoff:

[operations/puppet@production] idp-test: Point to the new Bullseye hosts

https://gerrit.wikimedia.org/r/801402

gerritbot added a comment.May 31 2022, 10:17 AM

Change 801624 merged by Muehlenhoff:

[operations/dns@master] Point idp-test to idp-test1002

https://gerrit.wikimedia.org/r/801624

MoritzMuehlenhoff updated the task description. (Show Details)May 31 2022, 6:28 PM
Stashbot added a comment.Jun 1 2022, 8:00 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-01T08:00:08Z] <moritzm> installing idp2002 T308214

Stashbot added a comment.Jun 1 2022, 8:49 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-01T08:49:11Z] <moritzm> installing idp1002 T308214

gerritbot added a comment.Jun 1 2022, 11:28 AM

Change 802098 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] profile::mariadb::ferm_misc: Remove old buster idp-test hosts, add new idp/bullseye ones

https://gerrit.wikimedia.org/r/802098

gerritbot added a comment.Jun 1 2022, 6:16 PM

Change 802098 merged by Muehlenhoff:

[operations/puppet@production] Remove old buster idp-test hosts, add new idp/bullseye ones from Ferm rules

https://gerrit.wikimedia.org/r/802098

gerritbot added a comment.Jun 2 2022, 9:14 AM

Change 802444 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Apply idp role to idp1002/idp2002

https://gerrit.wikimedia.org/r/802444

gerritbot added a comment.Jun 2 2022, 11:57 AM

Change 802444 merged by Muehlenhoff:

[operations/puppet@production] Apply idp role to idp1002/idp2002

https://gerrit.wikimedia.org/r/802444

gerritbot added a comment.Jun 2 2022, 3:05 PM

Change 802541 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Failover idp.w.o to idp1002 (new Bullseye node)

https://gerrit.wikimedia.org/r/802541

gerritbot added a comment.Jun 2 2022, 3:06 PM

Change 802542 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Failover active IDP nodes to idp1002/idp2002

https://gerrit.wikimedia.org/r/802542

gerritbot added a comment.Jun 3 2022, 8:09 AM

Change 802729 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove Puppet references to idp-test1001/idp-test2001

https://gerrit.wikimedia.org/r/802729

gerritbot added a comment.Jun 3 2022, 8:15 AM

Change 802729 merged by Muehlenhoff:

[operations/puppet@production] Remove Puppet references to idp-test1001/idp-test2001

https://gerrit.wikimedia.org/r/802729

ops-monitoring-bot added a comment.Jun 3 2022, 9:20 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: idp-test2001.wikimedia.org

  • idp-test2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
ops-monitoring-bot added a comment.Jun 3 2022, 9:28 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: idp-test1001.wikimedia.org

  • idp-test1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
gerritbot added a comment.Jun 8 2022, 9:00 AM

Change 802542 merged by Muehlenhoff:

[operations/puppet@production] Failover active IDP nodes to idp1002/idp2002

https://gerrit.wikimedia.org/r/802542

gerritbot added a comment.Jun 8 2022, 9:21 AM

Change 802541 merged by Muehlenhoff:

[operations/dns@master] Failover idp.w.o to idp1002 (new Bullseye node)

https://gerrit.wikimedia.org/r/802541

gerritbot added a comment.Jun 8 2022, 10:15 AM

Change 803883 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] profile::mariadb::ferm_misc: Remove old buster IDP nodes

https://gerrit.wikimedia.org/r/803883

gerritbot added a comment.Jun 8 2022, 11:39 AM

Change 803892 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch idp1001/idp2001 to role(insetup)

https://gerrit.wikimedia.org/r/803892

gerritbot added a comment.Jun 9 2022, 1:33 PM

Change 803883 merged by Muehlenhoff:

[operations/puppet@production] profile::mariadb::ferm_misc: Remove old buster IDP nodes

https://gerrit.wikimedia.org/r/803883

gerritbot added a comment.Jun 13 2022, 12:49 PM

Change 803892 merged by Muehlenhoff:

[operations/puppet@production] Switch idp1001/idp2001 to role(insetup)

https://gerrit.wikimedia.org/r/803892

gerritbot added a comment.Jun 13 2022, 3:06 PM

Change 805140 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] acme_chief: Remove old buster IDP hosts

https://gerrit.wikimedia.org/r/805140

MoritzMuehlenhoff updated the task description. (Show Details)Jun 13 2022, 3:07 PM
gerritbot added a comment.Jun 21 2022, 2:48 PM

Change 805140 merged by Muehlenhoff:

[operations/puppet@production] acme_chief: Remove old buster IDP hosts

https://gerrit.wikimedia.org/r/805140

ops-monitoring-bot added a comment.Jun 21 2022, 5:23 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: idp2001.wikimedia.org

  • idp2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
ops-monitoring-bot added a comment.Jun 21 2022, 5:38 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: idp1001.wikimedia.org

  • idp1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
MoritzMuehlenhoff closed this task as Resolved.Jun 21 2022, 5:40 PM
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff updated the task description. (Show Details)

This is complete

MoritzMuehlenhoff added a parent task: T305518: Upgrade IDPs to CAS 6.6/Bullseye and enable webauthn.Jun 23 2022, 2:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL