Page MenuHomePhabricator

Upgrade schema[12]00[12] to Debian Buster
Closed, ResolvedPublic5 Estimated Story Points

Description

Upgrade the schema* nodes to Debian Buster.

Event Timeline

elukey created this task.Jun 10 2020, 2:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 10 2020, 2:49 PM
elukey moved this task from Q2 2020/2021 to Q1 2020/2021 on the Analytics-Clusters board.

@Ottomata this should be easy to do right? Or is there any special consideration to make to avoid fireworks? :D

Should be easy peasy, there's nothing special or fancy here, and the hosts are HA LB-ed, so you should be able to just depool, reinstall, puppetize, repool.

elukey added a comment.Jul 3 2020, 6:55 AM

If I got it correctly the 4 schema hosts are VM on ganeti, so we cannot upgrade them in place. We could create 4 new VMs identical but with Buster (schema[12]00[34]), apply to them the correct puppet roles and then flip all the configs using the old nodes to the new ones. Finally we'd drop the old VMs and the upgrade would be completed.

When you do that, please use row B/D in eqiad and row C/D in codfw to better balance out our Ganeti groups.

Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM

Vms created, next steps:

  • check that vms are working fine etc.. (I checked that specs are correct but @Ottomata should verify that all the envoy services are good on all VMs just to be sure)
  • add new VMs to LVS
  • depool old ones from LVS
  • decommission old VMs

Change 622567 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add new schema[12]00[34] vms to the related LVS endpoints

https://gerrit.wikimedia.org/r/622567

Change 622567 merged by Elukey:
[operations/puppet@production] Add new schema[12]00[34] vms to the related LVS endpoints

https://gerrit.wikimedia.org/r/622567

New vms added to the LVS endpoints:

https://config-master.wikimedia.org/pybal/eqiad/schema
https://config-master.wikimedia.org/pybal/codfw/schema

If everything works correctly, we'll need to:

  • remove the old VMs from LVS
  • decommission them
elukey claimed this task.Aug 26 2020, 2:36 PM
elukey triaged this task as Medium priority.
elukey added a project: Analytics-Kanban.
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 622587 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove schema[12]00[12] from their LVS endpoint configs

https://gerrit.wikimedia.org/r/622587

Change 622587 merged by Elukey:
[operations/puppet@production] Remove schema[12]00[12] from their LVS endpoint configs

https://gerrit.wikimedia.org/r/622587

Change 622754 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove old schema[12]00[12] from puppet

https://gerrit.wikimedia.org/r/622754

Change 622754 merged by Elukey:
[operations/puppet@production] Remove old schema[12]00[12] from puppet

https://gerrit.wikimedia.org/r/622754

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema1001.eqiad.wmnet

  • schema1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema1002.eqiad.wmnet

  • schema1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema2001.codfw.wmnet

  • schema2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema2002.codfw.wmnet

  • schema2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed

Change 622755 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Remove schema[12]00[12] records (VMs decommissioned)

https://gerrit.wikimedia.org/r/622755

Change 622755 merged by Elukey:
[operations/dns@master] Remove schema[12]00[12] records (VMs decommissioned)

https://gerrit.wikimedia.org/r/622755

elukey set the point value for this task to 5.Aug 28 2020, 6:30 AM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
Nuria closed this task as Resolved.Sep 1 2020, 9:31 PM