Page MenuHomePhabricator
Create Task
Maniphest T300744

Move kubernetes workers to bullseye and docker to overlayfs
Closed, ResolvedPublic
Actions

Description

All kubernetes worker nodes should be running bullseye and docker (docker.io=20.10.5+dfsg1-1+deb11u1 from debian upstream) with overlayfs (instead of devicemapper) as storage driver:

  • Kubernetes master nodes already run docker (docker.io=18.09.1+dfsg1-7.1+deb10u3) with overlayfs (on buster)
  • Changes have been prepared and tested in ml as well as staging clusters

Todos:

  • Reimage ml-codfw workers
  • Reimage ml-eqiad workers
  • Reimage staging-codfw workers
  • Reimage staging-eqiad workers
  • Add kubernetes20[1(89)|2(012)] as updated nodes to wikikube-codfw - T302208
  • cordon kubernetes200[1-4]
  • Add kubernetes10[18-22] as updated nodes to wikikube-eqiad - T293728
  • cordon kubernetes100[1-4]
  • Reimage wikikube-codfw vms kubernetes20[05,06,15,16]
  • Reimage wikikube-codfw nodes kubernetes20[07-14,17]
  • Reimage wikikube-eqiad vms kubernetes10[05,06,15,16]
  • Reimage wikikube-eqiad nodes kubernetes10[07-14,17]
  • Decom kubernetes100[1-4] (they have been refreshed, T297157) - T303045
  • Decom kubernetes200[1-4] (they have been refreshed, T286585) - T303044

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+48 -0kubernetes: apply devicemapper settings to kubernetes[12]00[1-4]
operations/puppetproduction+54 -343kubernetes: clean up extra netboot and host settings
operations/puppetproduction+7 -0Set bullseye + overlayfs for kubernetes1014
operations/puppetproduction+7 -0Set bullseye + overlayfs for kubernetes1013
operations/puppetproduction+7 -0Set bullseye + overlayfs settings for kubernetes1012
operations/puppetproduction+1 -1install_server: update netboot settings for kubernetes nodes on Stretch
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes1011
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes1010
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes1009
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes1008
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes1007
operations/puppetproduction+7 -0Set overlay settings for kubernetes1016
operations/puppetproduction+7 -0Set overlay settings for kubernetes1015
operations/puppetproduction+7 -0Set overlay settings for kubernetes1006
operations/puppetproduction+7 -0Set overlay settings for kubernetes1005
operations/puppetproduction+5 -5Set bullseye + overlay settings for kubernetes10[01][56] nodes
operations/puppetproduction+70 -7Add kubernetes1018-1022
operations/puppetproduction+7 -0Set bullseye + overlayfs for kubernetes2016
operations/puppetproduction+2 -2Set bullseye in dhcp config for kubernetes201[5,6]
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes2015
operations/puppetproduction+1 -42install_server: move kubernetes200[5,6] to the new flat-noswap recipe
operations/puppetproduction+9 -1install_server: improve the kubernetes-node-virtual-overlay recipe
operations/puppetproduction+28 -0Set simpler partman recipe for kubernetes200[5,6]
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes2006
operations/puppetproduction+8 -1Set overlayfs + bullseye for kubernetes2005
operations/puppetproduction+8 -1Set bullseye and overlayfs for kubernetes2017
operations/puppetproduction+8 -1Set bullseye + overlayfs settings for kubernetes2014
operations/puppetproduction+8 -1Set bullseye + overlayfs settings for kubernetes2013
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes2012
operations/puppetproduction+8 -1Set bullseye + overlayfs for kubernetes2011
operations/puppetproduction+8 -1Set bullseye + overlayfs settings for kubernetes2010
operations/puppetproduction+8 -1Set bullseye + overlayfs settings for kubernetes2009
operations/puppetproduction+8 -1Add bullseye + overlayfs settings to kubernetes2008
operations/puppetproduction+8 -0Set Bullseye + overlayfs settings for kubernetes2007
operations/puppetproduction+7 -0Add overlayfs settings for kubestage1003
operations/puppetproduction+7 -0Add overlayfs settings to kubestage2002's settings
operations/puppetproduction+1 -2install_server: move kubestage[12]* nodes to overlayfs partman recipe
operations/puppetproduction+1 -0install_server: set new partman recipe for kubestage1003
operations/puppetproduction+2 -1Change docker package name for kubestage2001
operations/puppetproduction+6 -0Add overlayfs settings for kubestage2001
operations/puppetproduction+1 -0install_server: set new partman recipe for kubestage2001
operations/puppetproduction+12 -0base::kernel: explicitly load overlay when overlayfs is true
operations/homer/publicmaster+2 -0Add ml-serve200[7,8] to the k8s ml-serve-codfw cluster
operations/puppetproduction+17 -6Add ml-serve200[7,8] to the k8s ml-serve-codfw cluster
operations/puppetproduction+7 -1profile::docker::engine: skip profile::base::memory_cgroup
operations/puppetproduction+13 -0profile::kubernetes::node: add grub settings for Bullseye
operations/puppetproduction+1 -1profile::rsyslog::kubernetes: add support for Bullseye
operations/puppetproduction+7 -2Add ml-serve2006 to the ml-serve-codfw k8s cluster
operations/puppetproduction+4 -2profile::kubernetes::node: avoid iptables alternatives for Bullseye
operations/debs/rsyslogdebian/bullseye-wikimedia-k8s+47 -0Add wmf-specific patches to Rsyslog
operations/debs/rsyslogdebian/bullseye-wikimedia-k8s+85 -26Add wmf-specific patches to Rsyslog
operations/puppetproduction+74 -23kubernetes: add bullseye apt configuration
operations/puppetproduction+2 -0profile::kubernetes::node: support bullseye
operations/homer/publicmaster+1 -0Add ml-serve2005 to the ml-serve-codfw k8s cluster
operations/puppetproduction+3 -1Empty profile::docker::storage settings for ml-serve2005
operations/puppetproduction+9 -3Add ml-serve2005 to the ml-serve codfw cluster
operations/puppetproduction+6 -2profile::docker::engine: add param to ignore docker storage settings
operations/puppetproduction+3 -1install_server: avoid swap in d-i when using kubernetes-node-overlay.cfg
operations/puppetproduction+1 -0install_server: set new partman recipe for ml-serve2005
operations/puppetproduction+62 -0install_server: add partman recipe kubernetes-node-overlay.cfg
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
JMeybohm updated the task description. (Show Details)Mar 4 2022, 1:54 PM
JMeybohm closed subtask T302208: setup/install kubernetes20[1(89)|2(012)] as Resolved.
JMeybohm updated the task description. (Show Details)Mar 4 2022, 1:56 PM
gerritbot added a comment.Mar 8 2022, 5:45 PM

Change 769085 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set Bullseye + overlayfs settings for kubernetes2005

https://gerrit.wikimedia.org/r/769085

gerritbot added a comment.Mar 9 2022, 2:57 PM

Change 769085 merged by Elukey:

[operations/puppet@production] Set Bullseye + overlayfs settings for kubernetes2007

https://gerrit.wikimedia.org/r/769085

gerritbot added a comment.Mar 9 2022, 4:13 PM

Change 769463 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add bullseye + overlayfs settings to kubernetes2008

https://gerrit.wikimedia.org/r/769463

gerritbot added a comment.Mar 9 2022, 4:42 PM

Change 769463 merged by Elukey:

[operations/puppet@production] Add bullseye + overlayfs settings to kubernetes2008

https://gerrit.wikimedia.org/r/769463

gerritbot added a comment.Mar 10 2022, 7:10 AM

Change 769616 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2009

https://gerrit.wikimedia.org/r/769616

gerritbot added a comment.Mar 10 2022, 7:10 AM

Change 769617 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2010

https://gerrit.wikimedia.org/r/769617

gerritbot added a comment.Mar 10 2022, 9:20 AM

Change 769616 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2009

https://gerrit.wikimedia.org/r/769616

elukey updated the task description. (Show Details)Mar 10 2022, 9:29 AM
gerritbot added a comment.Mar 10 2022, 10:06 AM

Change 769617 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2010

https://gerrit.wikimedia.org/r/769617

gerritbot added a comment.Mar 11 2022, 9:57 AM

Change 769919 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2011

https://gerrit.wikimedia.org/r/769919

gerritbot added a comment.Mar 11 2022, 9:57 AM

Change 769920 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2012

https://gerrit.wikimedia.org/r/769920

gerritbot added a comment.Mar 11 2022, 10:05 AM

Change 769919 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2011

https://gerrit.wikimedia.org/r/769919

gerritbot added a comment.Mar 11 2022, 10:44 AM

Change 769920 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2012

https://gerrit.wikimedia.org/r/769920

gerritbot added a comment.Mar 11 2022, 2:06 PM

Change 769978 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2013

https://gerrit.wikimedia.org/r/769978

gerritbot added a comment.Mar 11 2022, 2:06 PM

Change 769979 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2014

https://gerrit.wikimedia.org/r/769979

gerritbot added a comment.Mar 11 2022, 2:30 PM

Change 769978 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2013

https://gerrit.wikimedia.org/r/769978

gerritbot added a comment.Mar 11 2022, 3:24 PM

Change 769979 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes2014

https://gerrit.wikimedia.org/r/769979

gerritbot added a comment.Mar 14 2022, 7:24 AM

Change 770439 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye and overlayfs for kubernetes2017

https://gerrit.wikimedia.org/r/770439

gerritbot added a comment.Mar 14 2022, 7:24 AM

Change 770440 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1007

https://gerrit.wikimedia.org/r/770440

gerritbot added a comment.Mar 14 2022, 8:39 AM

Change 770439 merged by Elukey:

[operations/puppet@production] Set bullseye and overlayfs for kubernetes2017

https://gerrit.wikimedia.org/r/770439

elukey updated the task description. (Show Details)Mar 14 2022, 9:42 AM
elukey added a comment.EditedMar 14 2022, 9:59 AM

There are some ganeti VMs running as kubernets nodes in both clusters, with two virtual disks: vda for the boot and root partitions, and vdb as unmanaged for device mapper (used by Docker). Since we are moving to Overlay, the plan for each VM would be the following (using kubernetes2005 as example):

Assumption: Ganeti vdisks are 0 indexed, so vda == 0 vdb == 1

  • Drain kubernetes2005 via kubectl
  • add downtime
  • shutdown the instance gnt-instance shutdown kubernetes2005.codfw.wmnet
  • remove /dev/vdb gnt-instance modify --disk 1:remove kubernetes2005.codfw.wmnet
  • add 10g to vda gnt-instance grow-disk kubernetes2005.codfw.wmnet 0 10g
  • follow https://wikitech.wikimedia.org/wiki/Ganeti#Reinstall_/_Reimage_a_VM to reinstall the VM with Bullseye
gerritbot added a comment.Mar 14 2022, 10:03 AM

Change 770459 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set overlayfs + bullseye for kubernetes2005

https://gerrit.wikimedia.org/r/770459

gerritbot added a comment.Mar 14 2022, 10:19 AM

Change 770459 merged by Elukey:

[operations/puppet@production] Set overlayfs + bullseye for kubernetes2005

https://gerrit.wikimedia.org/r/770459

elukey added a comment.Mar 14 2022, 2:45 PM

All good, kubernetes2005 reimaged as planned. The only little issue that I encountered was a confirm question during d-i to use the whole disk (20G) instead of half of it as partman suggested (I think that it is related to the current values of the recipe that have 10g as max capacity).

gerritbot added a comment.Mar 15 2022, 8:06 AM

Change 770879 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2006

https://gerrit.wikimedia.org/r/770879

gerritbot added a comment.Mar 15 2022, 8:31 AM

Change 770879 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2006

https://gerrit.wikimedia.org/r/770879

gerritbot added a comment.Mar 15 2022, 1:26 PM

Change 770912 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set simpler partman recipe for kubernetes200[5,6]

https://gerrit.wikimedia.org/r/770912

gerritbot added a comment.Mar 16 2022, 7:29 AM

Change 770912 merged by Elukey:

[operations/puppet@production] Set simpler partman recipe for kubernetes200[5,6]

https://gerrit.wikimedia.org/r/770912

gerritbot added a comment.Mar 16 2022, 8:14 AM

Change 771319 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: improve the kubernetes-node-virtual-overlay recipe

https://gerrit.wikimedia.org/r/771319

gerritbot added a comment.Mar 16 2022, 8:20 AM

Change 771319 merged by Elukey:

[operations/puppet@production] install_server: improve the kubernetes-node-virtual-overlay recipe

https://gerrit.wikimedia.org/r/771319

gerritbot added a comment.Mar 16 2022, 2:03 PM

Change 771356 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: move kubernetes200[5,6] to the new flat-noswap recipe

https://gerrit.wikimedia.org/r/771356

gerritbot added a comment.Mar 16 2022, 2:39 PM

Change 771356 merged by Elukey:

[operations/puppet@production] install_server: move kubernetes200[5,6] to the new flat-noswap recipe

https://gerrit.wikimedia.org/r/771356

gerritbot added a comment.Mar 16 2022, 6:14 PM

Change 771422 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2015

https://gerrit.wikimedia.org/r/771422

gerritbot added a comment.Mar 16 2022, 6:14 PM

Change 771423 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2016

https://gerrit.wikimedia.org/r/771423

gerritbot added a comment.Mar 17 2022, 9:58 AM

Change 771422 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2015

https://gerrit.wikimedia.org/r/771422

gerritbot added a comment.Mar 17 2022, 10:00 AM

Change 771552 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye in dhcp config for kubernetes201[5,6]

https://gerrit.wikimedia.org/r/771552

gerritbot added a comment.Mar 17 2022, 10:02 AM

Change 771552 merged by Elukey:

[operations/puppet@production] Set bullseye in dhcp config for kubernetes201[5,6]

https://gerrit.wikimedia.org/r/771552

gerritbot added a comment.Mar 17 2022, 10:55 AM

Change 771423 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes2016

https://gerrit.wikimedia.org/r/771423

elukey updated the task description. (Show Details)Mar 17 2022, 11:36 AM
gerritbot added a comment.Mar 17 2022, 1:58 PM

Change 771598 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Add kubernetes1018-1022

https://gerrit.wikimedia.org/r/771598

gerritbot added a comment.Mar 17 2022, 2:02 PM

Change 771600 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlay settings for kubernetes10[01][56] nodes

https://gerrit.wikimedia.org/r/771600

gerritbot added a comment.Mar 17 2022, 2:02 PM

Change 771601 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set overlay settings for kubernetes1005

https://gerrit.wikimedia.org/r/771601

gerritbot added a comment.Mar 17 2022, 2:02 PM

Change 771602 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set overlay settings for kubernetes1006

https://gerrit.wikimedia.org/r/771602

gerritbot added a comment.Mar 17 2022, 2:02 PM

Change 771603 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set overlay settings for kubernetes1015

https://gerrit.wikimedia.org/r/771603

gerritbot added a comment.Mar 17 2022, 2:02 PM

Change 771604 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set overlay settings for kubernetes1016

https://gerrit.wikimedia.org/r/771604

gerritbot added a comment.Mar 17 2022, 2:57 PM

Change 771598 merged by Alexandros Kosiaris:

[operations/puppet@production] Add kubernetes1018-1022

https://gerrit.wikimedia.org/r/771598

gerritbot added a comment.Mar 18 2022, 8:19 AM

Change 771600 merged by Elukey:

[operations/puppet@production] Set bullseye + overlay settings for kubernetes10[01][56] nodes

https://gerrit.wikimedia.org/r/771600

gerritbot added a comment.Mar 18 2022, 8:32 AM

Change 771601 merged by Elukey:

[operations/puppet@production] Set overlay settings for kubernetes1005

https://gerrit.wikimedia.org/r/771601

akosiaris closed subtask T293728: setup/install kubernetes10[18-22] as Resolved.Mar 18 2022, 9:51 AM
akosiaris changed the status of subtask T303044: decommission kubernetes100[1-4] from Stalled to Open.
akosiaris changed the status of subtask T303045: decommission kubernetes200[1-4] from Stalled to Open.Mar 18 2022, 10:50 AM
gerritbot added a comment.Mar 21 2022, 7:59 AM

Change 771602 merged by Elukey:

[operations/puppet@production] Set overlay settings for kubernetes1006

https://gerrit.wikimedia.org/r/771602

gerritbot added a comment.Mar 21 2022, 9:21 AM

Change 771603 merged by Elukey:

[operations/puppet@production] Set overlay settings for kubernetes1015

https://gerrit.wikimedia.org/r/771603

gerritbot added a comment.Mar 21 2022, 10:40 AM

Change 771604 merged by Elukey:

[operations/puppet@production] Set overlay settings for kubernetes1016

https://gerrit.wikimedia.org/r/771604

elukey updated the task description. (Show Details)Mar 21 2022, 11:24 AM
elukey updated the task description. (Show Details)Mar 21 2022, 2:56 PM
gerritbot added a comment.Mar 22 2022, 6:51 AM

Change 770440 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1007

https://gerrit.wikimedia.org/r/770440

gerritbot added a comment.Mar 22 2022, 7:13 AM

Change 772686 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1008

https://gerrit.wikimedia.org/r/772686

gerritbot added a comment.Mar 22 2022, 8:12 AM

Change 772686 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1008

https://gerrit.wikimedia.org/r/772686

gerritbot added a comment.Mar 23 2022, 7:47 AM

Change 773181 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1009

https://gerrit.wikimedia.org/r/773181

gerritbot added a comment.Mar 23 2022, 7:48 AM

Change 773181 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1009

https://gerrit.wikimedia.org/r/773181

gerritbot added a comment.Mar 23 2022, 10:08 AM

Change 773193 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1010

https://gerrit.wikimedia.org/r/773193

gerritbot added a comment.Mar 23 2022, 1:14 PM

Change 773193 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1010

https://gerrit.wikimedia.org/r/773193

gerritbot added a comment.Mar 23 2022, 4:05 PM

Change 773278 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1011

https://gerrit.wikimedia.org/r/773278

gerritbot added a comment.Mar 23 2022, 4:07 PM

Change 773278 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1011

https://gerrit.wikimedia.org/r/773278

gerritbot added a comment.Mar 24 2022, 6:58 AM

Change 773389 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: update netboot settings for kubernetes nodes on Stretch

https://gerrit.wikimedia.org/r/773389

gerritbot added a comment.Mar 24 2022, 6:58 AM

Change 773390 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes1012

https://gerrit.wikimedia.org/r/773390

gerritbot added a comment.Mar 24 2022, 7:00 AM

Change 773389 merged by Elukey:

[operations/puppet@production] install_server: update netboot settings for kubernetes nodes on Stretch

https://gerrit.wikimedia.org/r/773389

gerritbot added a comment.Mar 24 2022, 7:08 AM

Change 773390 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs settings for kubernetes1012

https://gerrit.wikimedia.org/r/773390

gerritbot added a comment.Mar 24 2022, 8:03 AM

Change 773443 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1013

https://gerrit.wikimedia.org/r/773443

gerritbot added a comment.Mar 24 2022, 8:12 AM

Change 773443 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1013

https://gerrit.wikimedia.org/r/773443

gerritbot added a comment.Mar 24 2022, 10:05 AM

Change 773466 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1014

https://gerrit.wikimedia.org/r/773466

gerritbot added a comment.Mar 24 2022, 10:09 AM

Change 773466 merged by Elukey:

[operations/puppet@production] Set bullseye + overlayfs for kubernetes1014

https://gerrit.wikimedia.org/r/773466

elukey updated the task description. (Show Details)Mar 24 2022, 11:29 AM
gerritbot added a comment.Mar 24 2022, 2:48 PM

Change 773520 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] kubernetes: clean up extra netboot and host settings

https://gerrit.wikimedia.org/r/773520

gerritbot added a comment.Mar 25 2022, 9:43 AM

Change 773520 merged by Elukey:

[operations/puppet@production] kubernetes: clean up extra netboot and host settings

https://gerrit.wikimedia.org/r/773520

gerritbot added a comment.Mar 25 2022, 10:08 AM

Change 773751 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] kubernetes: apply devicemapper settings to kubernetes[12]00[1-4]

https://gerrit.wikimedia.org/r/773751

gerritbot added a comment.Mar 25 2022, 10:11 AM

Change 773751 merged by Elukey:

[operations/puppet@production] kubernetes: apply devicemapper settings to kubernetes[12]00[1-4]

https://gerrit.wikimedia.org/r/773751

elukey removed elukey as the assignee of this task.Mar 25 2022, 10:15 AM

Removing myself as assignee of the task, all the reimages completed. The remaining steps are:

elukey removed a project: Machine-Learning-Team (Active Tasks).Mar 25 2022, 10:16 AM
JMeybohm closed this task as Resolved.Mar 30 2022, 7:03 AM
JMeybohm claimed this task.
JMeybohm updated the task description. (Show Details)

I think the SRE part of this is all done. Thanks everyone! ❤

Papaul closed subtask T303045: decommission kubernetes200[1-4] as Resolved.Mar 31 2022, 4:16 PM
JMeybohm mentioned this in T245272: Draft a plan for upgrading kubernetes machines to buster.Apr 4 2022, 7:30 AM
JMeybohm mentioned this in T305435: Migrate kubernetes masters to bullseye.Apr 5 2022, 8:55 AM
Cmjohnson closed subtask T303044: decommission kubernetes100[1-4] as Resolved.Apr 13 2022, 5:44 PM
BTullis added a subscriber: BTullis.Aug 23 2022, 9:15 AM

@JMeybohm - something to note, when setting up the new dse-k8s-ctrl100[1-2] servers recently under T310172, I observed a race condition whereby the devicemapper storage engine got initialized before overlayfs.

After a puppet run I had to do the following:

systemctl stop docker
cd /var/lib/docker
rm -rf *
systemctl start docker

After this the overlayfs storage engine was in use.

JMeybohm added a comment.EditedAug 23 2022, 9:40 AM
In T300744#8177252, @BTullis wrote:

@JMeybohm - something to note, when setting up the new dse-k8s-ctrl100[1-2] servers recently under T310172, I observed a race condition whereby the devicemapper storage engine got initialized before overlayfs.

After a puppet run I had to do the following:

systemctl stop docker
cd /var/lib/docker
rm -rf *
systemctl start docker

After this the overlayfs storage engine was in use.

that may happen when the node is role(insetup) (which means overlayfs module is blacklisted) and then docker is installed via a role change. There is no guarantee that the overlayfs module can be loaded when docker starts. IIRC setting the hiera key profile::docker::engine::force_default_docker_storage: true prevents this (see https://gerrit.wikimedia.org/r/c/operations/puppet/+/759678)

Or not including profile::docker::storage at all. Sorry, seems like we did not clean that up after moving from devicemapper to overlay

JMeybohm mentioned this in T315977: Clean up profile::docker::storage from puppet.Aug 23 2022, 9:52 AM
Clement_Goubert closed subtask T315977: Clean up profile::docker::storage from puppet as Resolved.Aug 26 2022, 10:13 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL