Page MenuHomePhabricator

Create new deployment-ms-be instances running Debian Bullseye
Closed, ResolvedPublic

Description

Current ms-be instances in the deployment-prep project have been shutdown cause those are running debian 9. The current production instances are running already in Bullseye so it should be possible to spawn new instances running Debian 11

Event Timeline

Change 852245 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] deployment-prep: Add ms-be0[78] as storagehosts

https://gerrit.wikimedia.org/r/852245

Vgutierrez changed the task status from Open to In Progress.Wed, Nov 2, 5:11 PM

Change 852245 merged by Vgutierrez:

[operations/puppet@production] deployment-prep: Add ms-be0[78] as storagehosts

https://gerrit.wikimedia.org/r/852245

Change 852281 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: Add ms-be0[78] to deployment-prep cluster

https://gerrit.wikimedia.org/r/852281

Change 852281 merged by Vgutierrez:

[operations/puppet@production] swift: Add ms-be0[78] to deployment-prep cluster

https://gerrit.wikimedia.org/r/852281

Change 852290 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: Drain ms-be0[56]@deployment-prep cluster

https://gerrit.wikimedia.org/r/852290

Change 852290 merged by Vgutierrez:

[operations/puppet@production] swift: Drain ms-be0[56]@deployment-prep cluster

https://gerrit.wikimedia.org/r/852290

Change 852293 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: Fix syntax error on deployment-prep config file

https://gerrit.wikimedia.org/r/852293

Change 852293 merged by Vgutierrez:

[operations/puppet@production] swift: Fix syntax error on deployment-prep config file

https://gerrit.wikimedia.org/r/852293

Change 852740 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cluster: Add deployment-prep swift cluster definition

https://gerrit.wikimedia.org/r/852740

Change 852740 abandoned by Vgutierrez:

[operations/puppet@production] cluster: Add deployment-prep swift cluster definition

Reason:

https://gerrit.wikimedia.org/r/852740

Change 852767 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] puppetmaster::standalone: Sync swift rings

https://gerrit.wikimedia.org/r/852767

Change 852767 merged by Vgutierrez:

[operations/puppet@production] puppetmaster::standalone: Sync swift rings

https://gerrit.wikimedia.org/r/852767

Vgutierrez changed the task status from In Progress to Stalled.Thu, Nov 3, 11:56 AM
vgutierrez@deployment-cumin:~$ sudo -i cumin 'project:deployment-prep name:ms-be' 'swift-ring-builder /etc/swift/object.builder'
4 hosts will be targeted:
deployment-ms-be[05-08].deployment-prep.eqiad1.wikimedia.cloud
Ok to proceed on 4 hosts? Enter the number of affected hosts to confirm or "q" to quit 4
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |   0% (0/4) [00:00<?, ?hosts/s]
===== NODE GROUP =====                                                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |   0% (0/4) [00:01<?, ?hosts/s]
(2) deployment-ms-be[07-08].deployment-prep.eqiad1.wikimedia.cloud                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                         |   0% (0/4) [00:01<?, ?hosts/s]
----- OUTPUT of 'swift-ring-build...t/object.builder' -----                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                         |   0% (0/4) [00:01<?, ?hosts/s]
/etc/swift/object.builder, build version 39, id (not assigned)                                                                                                                                                                                                                                                                                                           65536 partitions, 2.000000 replicas, 1 regions, 3 zones, 4 devices, 0.02 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 3 (2:15:51 remaining)
The overload factor is 10.00% (0.100000)
Ring file /etc/swift/object.ring.gz is obsolete
Devices:   id region zone   ip address:port replication ip:port  name weight partitions balance flags meta
            0      1    1 172.16.5.167:6010   172.16.5.167:6010  sdb1  40.00       2622    0.02       
            1      1    1  172.16.6.12:6010    172.16.6.12:6010  sdb1  40.00       2621   -0.02       
            2      1    3 172.16.7.114:6000   172.16.7.114:6000 lv-a1 960.00      62915    0.00       
            3      1    4 172.16.7.115:6000   172.16.7.115:6000 lv-a1 960.00      62914   -0.00       
                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                         |   0% (0/4) [00:01<?, ?hosts/s]
===== NODE GROUP =====                                                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                         |   0% (0/4) [00:01<?, ?hosts/s]
(2) deployment-ms-be[05-06].deployment-prep.eqiad1.wikimedia.cloud                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                         |   0% (0/4) [00:01<?, ?hosts/s]
----- OUTPUT of 'swift-ring-build...t/object.builder' -----                                                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                         |   0% (0/4) [00:01<?, ?hosts/s]
/etc/swift/object.builder, build version 39                                                                                                                                                                                                                                                                                                                              
65536 partitions, 2.000000 replicas, 1 regions, 3 zones, 4 devices, 0.02 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 3 (2:15:51 remaining)
The overload factor is 10.00% (0.100000)
Ring file /etc/swift/object.ring.gz is up-to-date
Devices:   id region zone   ip address:port replication ip:port  name weight partitions balance flags meta
            0      1    1 172.16.5.167:6010   172.16.5.167:6010  sdb1  40.00       2622    0.02       
            1      1    1  172.16.6.12:6010    172.16.6.12:6010  sdb1  40.00       2621   -0.02       
            2      1    3 172.16.7.114:6000   172.16.7.114:6000 lv-a1 960.00      62915    0.00       
            3      1    4 172.16.7.115:6000   172.16.7.115:6000 lv-a1 960.00      62914   -0.00

waiting while the old nodes get drained

Change 852912 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: Ramp up ms-be07 balance

https://gerrit.wikimedia.org/r/852912

Change 852912 merged by Vgutierrez:

[operations/puppet@production] swift: Ramp up ms-be07 balance

https://gerrit.wikimedia.org/r/852912

Change 853401 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: Ramp up ms-be08 rebalance

https://gerrit.wikimedia.org/r/853401

Change 853401 merged by Vgutierrez:

[operations/puppet@production] swift: Ramp up ms-be08 rebalance

https://gerrit.wikimedia.org/r/853401

Change 853475 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: Ramp up ms-be05 draining

https://gerrit.wikimedia.org/r/853475

Change 853475 merged by Vgutierrez:

[operations/puppet@production] swift: Ramp up ms-be05 draining

https://gerrit.wikimedia.org/r/853475

Change 853917 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: drain ms-be06 ASAP and remove ms-be05

https://gerrit.wikimedia.org/r/853917

Change 853917 merged by Vgutierrez:

[operations/puppet@production] swift: drain ms-be06 ASAP and remove ms-be05

https://gerrit.wikimedia.org/r/853917

Mentioned in SAL (#wikimedia-releng) [2022-11-07T15:57:06Z] <vgutierrez> shutting down deployment-ms-be05 - T322231

Change 854041 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] deployment-prep: Remove ms-be05

https://gerrit.wikimedia.org/r/854041

Change 854041 merged by Vgutierrez:

[operations/puppet@production] deployment-prep: Remove ms-be05

https://gerrit.wikimedia.org/r/854041

Change 854094 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] swift: Remove ms-be06 from deployment-prep cluster

https://gerrit.wikimedia.org/r/854094

Change 854094 merged by Vgutierrez:

[operations/puppet@production] swift: Remove ms-be06 from deployment-prep cluster

https://gerrit.wikimedia.org/r/854094

Mentioned in SAL (#wikimedia-releng) [2022-11-08T06:39:41Z] <vgutierrez> delete deployment-ms-be05 - T322231

Change 854416 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] deployment-prep: Move swift expirer duties to ms-be07

https://gerrit.wikimedia.org/r/854416

Change 854416 merged by Vgutierrez:

[operations/puppet@production] deployment-prep: Move swift expirer duties to ms-be07

https://gerrit.wikimedia.org/r/854416

Change 854533 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] deployment-prep: Remove ms-be06 from swift::storagehosts

https://gerrit.wikimedia.org/r/854533

Change 854533 merged by Vgutierrez:

[operations/puppet@production] deployment-prep: Remove ms-be06 from swift::storagehosts

https://gerrit.wikimedia.org/r/854533

Mentioned in SAL (#wikimedia-releng) [2022-11-08T15:21:37Z] <vgutierrez> shutdown deployment-ms-be06 - T322231

Mentioned in SAL (#wikimedia-releng) [2022-11-08T15:26:15Z] <vgutierrez> delete deployment-ms-be06 - T322231

Vgutierrez claimed this task.