Page MenuHomePhabricator
Create Task
Maniphest T307943

Update Kubernetes clusters to v1.23
Open, HighPublic
Actions

Description

We need/want to upgrade our Kubernetes clusters to Kubernetes v1.23

Kubernetes v1.23 was selected as a target:

  • because it is the last version supporting dockershim (and we don't want to move away from that together with this update)
  • v1.24 was only released 2022-05-03 and we usually don't want to upgrade to a just released major/miner

Together with the Kubernetes update, we need to update the following other components (subtasks might be a good idea for each of them when more details available):

Preparation for the Kubernetes update:

  • Double check out docker API version is supported with Kubernetes v1.23 (minimum version is still 1.26.0)
  • Check if migration from command line flags to config files is required for some Kubernetes components: T300499
    • Looks to me like it is not yet strictly needed. It might be for some specific flags, though. That we will figure out during upgrade tests I guess.
  • Ensure all our charts are compatible with Kubernetes v1.23
    • Our current validation with kubeyaml does not support this and getting support for 1.23 into kubeyaml will require time that could be better invested into replacing kubeyaml in deployment-charts CI with kubeconform: T306165
  • Read Kubernetes changelogs (yellow/red flags just linked below each version. Tick the box if all action required items have been addressed), https://relnotes.k8s.io (relevant versions)
  • v1.18
    • Action Required
    • Note
      • The following features are unconditionally enabled and the corresponding --feature-gates flags have been removed: PodPriority, TaintNodesByCondition, ResourceQuotaScopeSelectors and ScheduleDaemonSetPods (#86210, @draveness)
      • Kube-proxy: Added dual-stack IPv4/IPv6 support to the iptables proxier. (#82462, @vllry)
      • Support server-side dry-run in kubectl with --dry-run=server for commands including apply, patch, create, run, annotate, label, set, autoscale, drain, rollout undo, and expose. (#87714, @julianvmodesto)
  • v1.19
    • Action Required
      • seccomp graduates to GA, check if we need to migrate PSPs off the annotations https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#seccomp-graduates-to-general-availability (this only affects pod spec not PSPs, nothing to do here)
      • Kube-apiserver: the componentstatus API is deprecated. This API provided status of etcd, kube-scheduler, and kube-controller-manager components, but only worked when those components were local to the API server, and when kube-scheduler and kube-controller-manager exposed unsecured health endpoints. Instead of this API, etcd health is included in the kube-apiserver health check and kube-scheduler/kube-controller-manager health checks can be made directly against those components' health endpoints. (#93570, @liggitt)
      • ✅ Kubeadm now includes CoreDNS version v1.7.0.
      • ✅ Kube-apiserver: The NodeRestriction admission plugin now restricts Node labels kubelets are permitted to set when creating a new Node to the --node-labels parameters accepted by kubelets in 1.16+. (#90307, @liggitt)
    • Note
      • EndpointSlices, be aware (during debugging) https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
      • Fix bug in reflector that couldn't recover from "Too large resource version" errors (#92537, @wojtek-t) [SIG API Machinery]
      • Kubelet: add '--logging-format' flag to support structured logging (#91532, @afrouzMashaykhi)
      • Add --logging-format flag for component-base. Defaults to "text" using unchanged klog. (#89683, @yuzhiquan)
      • Kube-controller-manager: add '--logging-format' flag to support structured logging (#91521, @SataQiu)
      • Kube-scheduler: add '--logging-format' flag to support structured logging (#91522, @SataQiu)
      • The DefaultIngressClass feature is now GA. The --feature-gate parameter will be removed in 1.20. (#91957, @cmluciano)
      • The kube-controller-manager managed signers can now have distinct signing certificates and keys. See the help about --cluster-signing-[signer-name]-{cert,key}-file. --cluster-signing-{cert,key}-file is still the default. (#90822, @deads2k)2
      • Kube-apiserver, kube-scheduler and kube-controller manager now use SO_REUSEPORT socket option when listening on address defined by --bind-address and --secure-port flags, when running on Unix systems (Windows is NOT supported). This allows to run multiple instances of those processes on a single host with the same configuration, which allows to update/restart them in a graceful way, without causing downtime. (#88893, @invidian)
  • v1.20
    • Action Required
      • ✅ TokenRequest and TokenRequestProjection are now GA features. The following flags are required by the API server:
        • ✅ --service-account-issuer, should be set to a URL identifying the API server that will be stable over the cluster lifetime.
        • ✅ --service-account-key-file, set to one or more files containing one or more public keys used to verify tokens.
        • ✅ --service-account-signing-key-file, set to a file containing a private key to use to sign service account tokens. Can be the same file given to kube-controller-manager with --service-account-private-key-file. (#95896, @zshihang)
      • ✅ Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences: https://github.com/kubernetes-sigs/kubectl-check-ownerreferences
      • ✅ In dual-stack bare-metal clusters, you can now pass dual-stack IPs to kubelet --node-ip. eg: kubelet --node-ip 10.1.0.5,fd01::0005
        • ✅ In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods will now get dual-stack PodIPs.
    • Note
      • A bug was fixed in kubelet where exec probe timeouts were not respected. This may result in unexpected behavior since the default timeout (if not specified) is 1s which may be too small for some exec probes. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts.
      • Kubernetes 1.20 now enables API Priority and Fairness (APF) by default.
      • IPv4/IPv6 dual-stack has been reimplemented for 1.20 to support dual-stack Services: https://docs.k8s.io/concepts/services-networking/dual-stack/
      • On-demand metrics calculation is now available through /metrics/resources
      • kubectl alpha debug graduates from alpha to beta in 1.20, becoming kubectl debug
      • Support the node label node.kubernetes.io/exclude-from-external-load-balancers (might be an idea to exclude ganeti VM nodes from LVS?)
  • v1.21
    • Action Required
      • ✅ Kubeadm now includes CoreDNS v1.8.0
      • ✅ New admission controller DenyServiceExternalIPs is available. Clusters which do not need the Service externalIPs feature should enable this controller and be more secure. (#97395, @thockin)
      • ✅ The pause image upgraded to v3.4.1 in kubelet and kubeadm for both Linux and Windows. (#98205, @pacoxu) T322920
      • ✅ Update the latest validated version of Docker to 20.10
      • ✅ Upgrades IPv6Dualstack to Beta and turns it on by default. New clusters or existing clusters are not be affected until an actor starts adding secondary Pods and service CIDRS CLI flags as described here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack (#98969, @khenidak)
    • Note
      • Pod with multiple containers can use kubectl.kubernetes.io/default-container annotation to have a container preselected for kubectl commands.
      • Immutable Secrets and ConfigMaps graduates to GA. This feature allows users to specify that the contents of a particular Secret or ConfigMap is immutable for its object lifetime. For such instances, Kubelet will not watch/poll for changes and therefore reducing apiserver load. (Probably true for almost all of our configmap/secret objects as we roll-restart deployments on configmap changes anyways).
      • ServiceNodeExclusion, NodeDisruptionExclusion and LegacyNodeRoleBehavior features have been promoted to GA. ServiceNodeExclusion and NodeDisruptionExclusion are now unconditionally enabled, while LegacyNodeRoleBehavior is unconditionally disabled.
      • ✅ TokenRequest and TokenRequestProjection feature gates have been removed and are unconditionally enabled
      • Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default.
      • Namespace API objects now have a kubernetes.io/metadata.name label matching their metadata.name field to allow selecting any namespace by its name using a label selector.
      • Kubectl: kubectl get will omit managed fields by default now. Users could set --show-managed-fields to true to show managedFields when the output format is either json or yaml. (#96878, @knight42)
  • v1.22
    • Action Required
      • ✅ Various beta API removals. We're not affected as kubeconform would have given notice
      • ✅ controller-manager changes:
        • ✅ controller-manager MUST start with --authorization-kubeconfig and --authentication-kubeconfig correctly set to get authentication/authorization working
        • ✅ Applications that fetch metrics from controller-manager should use a dedicated service account which is allowed to access nonResourceURLs /metrics. (#96216, @knight42)
        • ✅ (don't think we use that) liveness/readiness probes to controller-manager MUST use HTTPS now, and the default port has been changed to 10257
      • ✅ Updated pause image to version 3.5, which now runs per default as pseudo user and group 65535:65535. This does not have any effect on remote container runtimes like CRI-O and containerd, which setup the pod sandbox user and group on their own. (#100292, @saschagrunert) T322920
    • Note
      • As of now both system-node-critical and system-cluster-critical pods have -997 OOM score, making them one of the last processes to be OOMKilled. If the user wants to have the pod to be OOMKilled last and the pod has system-cluster-critical priority class, it has to be changed to system-node-critical priority class to preserve the existing behavior (#99729, @ravisantoshgudimetla)
      • Server-side Apply is GA https://kubernetes.io/docs/reference/using-api/server-side-apply/
      • Default/kubeadm etcd moves to version 3.5.0
        • Data ccorruption issues with etcd 3.5.[0-2], use >=3.5.3
      • Introducing Memory quality of service support with cgroups v2 (Alpha). The MemoryQoS feature is now in Alpha. This allows kubelet running with cgroups v2 to set memory QoS at container, pod and QoS level to protect and guarantee better memory quality. This feature can be enabled through feature gate Memory QoS. (#102970, @borgerli)
      • Kube-apiserver: the alpha PodSecurity feature can be enabled by passing --feature-gates=PodSecurity=true, and enables controlling allowed pods using namespace labels. See https://git.k8s.io/enhancements/keps/sig-auth/2579-psp-replacement for more details. (#103099, @liggitt)
      • The EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. (#101048, @dims)
      • The NamespaceDefaultLabelName is promoted to GA in this release. All Namespace API objects have a kubernetes.io/metadata.name label matching their metadata.name field to allow selecting any namespace by its name using a label selector. (#101342, @rosenhouse)
  • v1.23
    • Action Required
    • Note
      • IPv4/IPv6 Dual-stack Networking graduates to GA https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack
      • PodSecurity graduates to Beta (In 1.23, the PodSecurity feature gate is enabled by default.) https://kubernetes.io/docs/concepts/security/pod-security-admission/
      • Structured logging graduate to Beta
      • Log messages in JSON format are written to stderr by default now (same as text format) instead of stdout. Users who expected JSON output on stdout must now capture stderr instead or in addition to stdout. (#106146, @pohly) (we log zu stderr anyways, so probably no change here)
      • Support for the seccomp annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io/[name] has been deprecated since 1.19, will be dropped in 1.25. Transition to using the seccompProfile API field. (#104389, @saschagrunert)
      • Ephemeral containers graduated to beta and are now available by default. (#105405, @verb)
      • The TTLAfterFinished feature gate is now GA and enabled by default. (#105219, @sahilvv)
      • Introduce a feature gate DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers. (#102507, @ostrain)
  • Read Calico changelogs
    • v3.17 > 3.17.0
      • All components that use Typha now use the same logic to discover Typha’s address. They lookup the endpoints of the service directly and connect to one at random. This avoids a dependency on kube-proxy. typha #466 (@fasaxc)
      • kube-controllers runs a non-root by default kube-controllers #566 (@caseydavenport)
    • v3.18
    • v3.19
      • Update ipables version to 1.8.4-15
      • By default, limit each node to 20 IP address blocks. This value can be overridden through IPAM configuration.
    • v3.20
      • Service-based egress rules; Calico NetworkPolicy and GlobalNetworkPolicy now support egress rules which match on Kubernetes service names. Service matches in egress rules can be used to allow or deny access to in-cluster services, as well as services typically not backed by pods (for example, the Kubernetes API). Address and port information is learned from the individual endpoints within the service.
      • Configurable BGP graceful restart timer; See the maxRestartTime configuration option in the BGPPeer API.
      • calico/node marks nodes with NetworkUnavailable=true on shutdown node #993 (@song-jiang)
      • Add IP address garbage collection to kube-controllers kube-controllers #744 (@caseydavenport)
      • Calico will now release empty IPAM blocks from nodes that no longer need them so they can be used elsewhere. kube-controllers #799 (@caseydavenport)
    • v3.21
      • For users of BGP you can now view the status of your BGP routers, including session status, RIB / FIB contents, and agent health via the new CalicoNodeStatus API: https://docs.projectcalico.org/archive/v3.21/reference/resources/caliconodestatus
      • Service-based ingress rules; In v3.20, we introduced egress policy rules that can match on Kubernetes services. In v3.21, we improved upon that in two ways. First, you can now use service matches in Calico NetworkPolicy and GlobalNetworkPolicy ingress rules. Second, you can now use service-based network policy rules on Windows nodes.
      • Option to run Calico as non-privileged and non-root; https://docs.projectcalico.org/archive/v3.21/security/non-privileged
      • ACTION REQUIRED: calico/node logs write to /var/log/calico within the container by default, in addition to stdout node #1133 (@song-jiang)
    • v3.22
      • None
    • v3.23
      • Update to CNI plugins v1.1.1 calico #5944 (@caseydavenport)
      • New per-pool IPAM metrics added calico #5706 (@pasanw)

Additional things to do together with the re-init of clusters:

  • Move to bigger Pod and Service IP range (T326617)
  • Delete etcd v2 datastore of calico: $ etcdctl -C https://$(hostname -f):2379 rm -r /calico
  • Reimage etcd clusters to bullseye (so the above is probably not needed)
  • enable DenyServiceExternalIPs admission plugin

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+4 -5profile::kubernetes::client: Switch to k8s 1.23
operations/cookbooksmaster+166 -0Add sre.k8s.wipe-cluster.py
operations/puppetproduction+41 -49Kubernetes masters: include profile::kubernetes::client
operations/puppetproduction+3 -1k8s::package: Ensure the apt component is registered first
operations/alertsmaster+15 -2KubernetesAPIErrorRate: make alert v1.23 compatible
operations/puppetproduction+10 -0kubernetes: Increase inotify limits
labs/privatemaster+4 -0Add prometheus user to the system:monitoring group
operations/deployment-chartsmaster+15 -0Pin coredns, eventrouter and helm-state-metrics for k8s 1.16
operations/deployment-chartsmaster+11 -1Update staging-codfw to k8s 1.23
operations/puppetproduction+47 -2k8s: Update staging-codfw to kubernetes 1.23
operations/puppetproduction+0 -6k8s: Remove default kubelet_cluster_domain definitions
operations/puppetproduction+4 -0k8s: Add the ClusterIP of kubernetes.default.cluster.local to cert
operations/deployment-chartsmaster+20 -4If-guard admin_ng objects no longer relevant for k8s 1.23
operations/puppetproduction+4 -22k8s: Remove authz_mode hiera key
operations/puppetproduction+429 -130k8s: Add support for PKI with k8s >= 1.23
operations/puppetproduction+89 -0pki: Add intermediate certifikates for wikikube and wikikube_staging
operations/puppetproduction+1 -2k8s: Fix duplicate definition of --service-account-key-file
operations/puppetproduction+40 -23k8s: Add a central ipv6dualstack flag to enable dual stack
operations/puppetproduction+48 -19k8s: Refactor profile::kubernetes::master::service_cluster_ip_range
operations/puppetproduction+44 -4k8s: make profile::kubernetes::cluster_cidr mandatory
operations/puppetproduction+8 -0k8s: Stop docker/runc spam from being written to syslog
operations/debs/kubernetesv1.23+6 -0Update to v1.23.14
operations/puppetproduction+26 -45calico: Allow different versions, drop pre bullseye support
operations/deployment-chartsmaster+3 -2calico: Allow calico-cni access to ipreservations
operations/deployment-chartsmaster+29 -84calico: More calico 3.23.3 additions.
operations/puppetproduction+12 -14calico: Align formatting with k8s module and profiles
operations/deployment-chartsmaster+72 -11Update calico to v3.23.3
operations/deployment-chartsmaster+30 -3calico: Allow alternative dnsConfig and disabling of IPv6
operations/puppetproduction+29 -13Add --service-account* flags for TokenRequest
operations/puppetproduction+83 -41k8s: Use the K8s::Core::V1Taint type
operations/puppetproduction+42 -16k8s: Add version switching where needed
operations/puppetproduction+66 -91Make Kubernetes version configurable
operations/puppetproduction+3 -4kubernetes: Actually use the master_fqdn instead of the cert name
operations/puppetproduction+10 -2kubernetes::master fail if user tokens are not unique
operations/puppetproduction+0 -12kubernetes::master remove apiserver_count
operations/debs/kubernetesv1.23+19 -18Update to Kubernetes v1.23.12
operations/puppetproduction+19 -94k8s: Remove all debian version if-guarding
operations/puppetproduction+2 -0aptrepo: Create versioned kubernetes components for buster
operations/puppetproduction+65 -75k8s: Align formatting along k8s module and profiles
operations/puppetproduction+2 -0aptrepo: Add bullseye components calico323 and kubernetes123
operations/debs/calicov3.23+25 -6Update calico to v3.23.3
operations/deployment-chartsmaster+921 -141Update calico-crds to v3.23.3
operations/deployment-chartsmaster+15 -1admin_ng: Allow to pin calico chart versions per environment
operations/debs/calicov3.20+8 -2Update to v3.20.6
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenJMeybohmT307943 Update Kubernetes clusters to v1.23
 ResolvedJMeybohmT270191 Add kubernetes 1.17+ topology annotations
 ResolvedJMeybohmT278329 Support multiple kubernetes versions with puppet
 ResolvedJMeybohmT270271 Target Sources (component/kubernetes-future/source/Sources) is configured multiple times
 ResolvedJMeybohmT300499 Migrate from command line flags to config files for kubernetes components
Restricted Task
 OpenayounsiT306649 Agree strategy for Kubernetes BGP peering to top-of-rack switches
 ResolvedelukeyT308418 Add missing failure domain labels to ml-serve-* clusters
 ResolvedJMeybohmT299236 Move away from system:node RBAC role
 ResolvedJMeybohmT306165 Replace kubeyaml in deployment-charts CI
 ResolvedClement_GoubertT316348 Remove kubeyaml from deployment-charts CI
 ResolvedJMeybohmT303279 Fix calico, cfssl-issuer and knative-serving Helm dependencies
 ResolvedJMeybohmT303184 High API server request latencies (LIST)
ResolvedJMeybohmT310486 Update cfssl-issuer to cert-manager 1.8.x
 ResolvedJMeybohmT310618 Define priorityClassName for istio and cert-manager deployments
 ResolvedJMeybohmT311251 Migrate kubernetes alerts away from icinga
Restricted Task
 ResolvedJMeybohmT313473 Switch to cgroup v2 and systemd as cgroup driver for docker and kubelet
 ResolvedClement_GoubertT315977 Clean up profile::docker::storage from puppet
Restricted Task
 ResolvedelukeyT321159 Import coredns 1.8.x (k8s 1.23 dependency)
 ResolvedelukeyT322193 Import istio 1.1x (k8s 1.23 dependency)
Restricted Task
 OpenNoneT277876 Reserve resources for system daemons on kubernetes nodes
 ResolvedelukeyT322920 Import pause container image >= 3.5 (k8s 1.23 dependency)
 OpenelukeyT277677 Write a cookbook to set a k8s cluster in maintenance mode
 ResolvedJMeybohmT260663 Create a cookbook for depooling one or all services from one kubernetes cluster
 OpenNoneT287491 Allow to address Kubernets API servers from NetworkPolicy
 ResolvedelukeyT323793 Import new knative-serving version (k8s 1.23 dependency for ML)
 ResolvedJMeybohmT323993 Upgrade DSE K8 Cluster to 1.23
 OpenjijikiT324959 Scrape controller-manager and scheduler metrics
 ResolvedJMeybohmT296303 New Kubernetes nodes may end up with no Pod IPv4 block assigned
 ResolvedJMeybohmT325292 Update cert-manager to 1.10.x
 ResolvedJMeybohmT326340 Update staging-codfw to k8s 1.23
 ResolvedBTullisT327799 Datahub errors in staging-codfw
 ResolvedJMeybohmT326617 Decide on new Pod and Sevice IPv4 ranges for wikikube clusters
 ResolvedMarosteguiT331508 Remove deprecated toolhub wikikube grants
 ResolvedJMeybohmT327664 Update staging-eqiad to k8s 1.23
 ResolvedJMeybohmT327751 k8s-api for k8s-staging down in codfw
 OpenJMeybohmT328291 Post Kubernetes v1.23 cleanup
 ResolvedJMeybohmT326729 Remove the .Values.kubernetesApi hack
 OpenNoneT290963 Drop the use of nonexisting groups in kubernetes infrastructure_users
 OpenJMeybohmT325066 Migrate charts away from deprecated typology annotations
 ResolvedJMeybohmT325620 Re-enable seccomProfile in cert-manager chart after k8s 1.23 migration completed
 OpenNoneT325268 Migrate use of infrastructure_users tokens to client certificates
 OpenNoneT322919 Metrics changes with Kubernetes v1.23
 ResolvedelukeyT324542 Upgrade ML clusters to Kubernetes 1.23
 ResolvedelukeyT327767 Upgrade the ml-staging-codfw cluster to k8s 1.23
 DeclinedNoneT330662 Upgrade the ml-etcd clusters to bullseye and PKI
 ResolvedNoneT330669 Upgrade the ml-serve-codfw cluster to k8s 1.23
 ResolvedelukeyT330758 Upgrade the ml-serve-eqiad cluster to k8s 1.23
 ResolvedelukeyT329556 K8s etcd on bullseye show TLS errors in logs
 ResolvedJMeybohmT329717 Switch wikikube-staging (codfw and eqiad) etcd clusters to use PKI
 ResolvedJMeybohmT329664 Update wikikube codfw to k8s 1.23
 OpenJMeybohmT329826 Kubernetes v1.23 use PKI for service-account signing (instead of cergen)
 ResolvedCDanisT329633 Upgrade the aux-eqiad cluster to k8s 1.23
 OpenNoneT330060 etcd cluster reimage strategies to use with the K8s upgrade cookbook
 ResolvedakosiarisT331126 Update wikikube eqiad to k8s 1.23

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Nov 16 2022, 9:49 AM

Change 856589 merged by JMeybohm:

[operations/puppet@production] k8s: Add a central ipv6dualstack flag to enable dual stack

https://gerrit.wikimedia.org/r/856589

gerritbot added a comment.Nov 16 2022, 9:49 AM

Change 857004 merged by JMeybohm:

[operations/puppet@production] k8s: Fix duplicate definition of --service-account-key-file

https://gerrit.wikimedia.org/r/857004

JMeybohm closed subtask Restricted Task as Resolved.Nov 18 2022, 9:22 AM
JMeybohm closed subtask Restricted Task as Resolved.
JMeybohm closed subtask Restricted Task as Resolved.Nov 18 2022, 9:24 AM
JMeybohm added a subtask: T277677: Write a cookbook to set a k8s cluster in maintenance mode.
JMeybohm added a subtask: T287491: Allow to address Kubernets API servers from NetworkPolicy.Nov 18 2022, 4:03 PM
elukey closed subtask T322920: Import pause container image >= 3.5 (k8s 1.23 dependency) as Resolved.Nov 21 2022, 10:22 AM
elukey updated the task description. (Show Details)
elukey mentioned this in T324542: Upgrade ML clusters to Kubernetes 1.23.Dec 6 2022, 9:36 AM
gerritbot added a comment.Dec 7 2022, 10:51 AM

Change 865592 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Add support for PKI with k8s >= 1.23

https://gerrit.wikimedia.org/r/865592

BTullis added a subtask: T323993: Upgrade DSE K8 Cluster to 1.23.Dec 7 2022, 1:49 PM
gerritbot added a comment.Dec 8 2022, 4:56 PM

Change 866444 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Remove authz_mode hiera key

https://gerrit.wikimedia.org/r/866444

JMeybohm updated the task description. (Show Details)Dec 9 2022, 12:28 PM
JMeybohm updated the task description. (Show Details)Dec 12 2022, 11:37 AM
JMeybohm updated the task description. (Show Details)Dec 12 2022, 11:45 AM
gerritbot added a comment.Dec 12 2022, 2:45 PM

Change 867182 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] pki: Add intermediate certifikates for wikikube and wikikube_staging

https://gerrit.wikimedia.org/r/867182

gerritbot added a comment.Dec 12 2022, 2:49 PM

Change 867182 merged by JMeybohm:

[operations/puppet@production] pki: Add intermediate certifikates for wikikube and wikikube_staging

https://gerrit.wikimedia.org/r/867182

JMeybohm added a subtask: T296303: New Kubernetes nodes may end up with no Pod IPv4 block assigned.Dec 13 2022, 8:42 AM
gerritbot added a comment.Dec 13 2022, 12:06 PM

Change 865592 merged by JMeybohm:

[operations/puppet@production] k8s: Add support for PKI with k8s >= 1.23

https://gerrit.wikimedia.org/r/865592

gerritbot added a comment.Dec 13 2022, 12:06 PM

Change 866444 merged by JMeybohm:

[operations/puppet@production] k8s: Remove authz_mode hiera key

https://gerrit.wikimedia.org/r/866444

JMeybohm closed subtask T303279: Fix calico, cfssl-issuer and knative-serving Helm dependencies as Resolved.Dec 13 2022, 2:03 PM
JMeybohm closed subtask T270191: Add kubernetes 1.17+ topology annotations as Resolved.Dec 14 2022, 3:04 PM
gerritbot added a comment.Dec 15 2022, 10:18 AM

Change 868232 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] If-guard admin_ng objects no longer relevant for k8s 1.23

https://gerrit.wikimedia.org/r/868232

gerritbot added a comment.Dec 15 2022, 12:25 PM

Change 868389 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] WIP: Update staging-codfw to k8s 1.23

https://gerrit.wikimedia.org/r/868389

JMeybohm updated the task description. (Show Details)Dec 15 2022, 12:29 PM
gerritbot added a comment.Dec 15 2022, 1:20 PM

Change 868232 merged by jenkins-bot:

[operations/deployment-charts@master] If-guard admin_ng objects no longer relevant for k8s 1.23

https://gerrit.wikimedia.org/r/868232

JMeybohm closed subtask T299236: Move away from system:node RBAC role as Resolved.Dec 15 2022, 1:44 PM
JMeybohm updated the task description. (Show Details)Dec 15 2022, 1:47 PM
JMeybohm added a subtask: T325620: Re-enable seccomProfile in cert-manager chart after k8s 1.23 migration completed.Dec 20 2022, 9:56 AM
gerritbot added a comment.Dec 22 2022, 10:57 AM

Change 870820 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Add the ClusterIP of kubernetes.default.cluster.local to cert

https://gerrit.wikimedia.org/r/870820

gerritbot added a comment.Jan 3 2023, 1:56 PM

Change 870820 merged by JMeybohm:

[operations/puppet@production] k8s: Add the ClusterIP of kubernetes.default.cluster.local to cert

https://gerrit.wikimedia.org/r/870820

JMeybohm closed subtask T325292: Update cert-manager to 1.10.x as Resolved.Jan 5 2023, 2:30 PM
gerritbot added a comment.Jan 10 2023, 10:09 AM

Change 877963 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Remove default kubelet_cluster_domain definitions

https://gerrit.wikimedia.org/r/877963

gerritbot added a comment.Jan 10 2023, 10:42 AM

Change 877990 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Update staging-codfw to kubernetes 1.23

https://gerrit.wikimedia.org/r/877990

JMeybohm updated the task description. (Show Details)Jan 10 2023, 10:50 AM
gerritbot added a comment.Jan 10 2023, 11:09 AM

Change 877963 merged by JMeybohm:

[operations/puppet@production] k8s: Remove default kubelet_cluster_domain definitions

https://gerrit.wikimedia.org/r/877963

gerritbot added a comment.Jan 10 2023, 5:10 PM

Change 877990 merged by JMeybohm:

[operations/puppet@production] k8s: Update staging-codfw to kubernetes 1.23

https://gerrit.wikimedia.org/r/877990

gerritbot added a comment.Jan 10 2023, 7:29 PM

Change 868389 merged by jenkins-bot:

[operations/deployment-charts@master] Update staging-codfw to k8s 1.23

https://gerrit.wikimedia.org/r/868389

gerritbot added a comment.Jan 11 2023, 1:47 PM

Change 878940 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Pin coredns, eventrouter and helm-state-metrics for k8s 1.16

https://gerrit.wikimedia.org/r/878940

gerritbot added a comment.Jan 11 2023, 4:57 PM

Change 878940 merged by jenkins-bot:

[operations/deployment-charts@master] Pin coredns, eventrouter and helm-state-metrics for k8s 1.16

https://gerrit.wikimedia.org/r/878940

elukey closed subtask T323793: Import new knative-serving version (k8s 1.23 dependency for ML) as Resolved.Jan 17 2023, 10:06 AM
JMeybohm updated the task description. (Show Details)Jan 23 2023, 3:52 PM
gerritbot added a comment.Jan 24 2023, 10:42 AM

Change 883130 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[labs/private@master] Add prometheus user to the system:monitoring group

https://gerrit.wikimedia.org/r/883130

gerritbot added a comment.Jan 24 2023, 10:44 AM

Change 883130 merged by JMeybohm:

[labs/private@master] Add prometheus user to the system:monitoring group

https://gerrit.wikimedia.org/r/883130

JMeybohm closed subtask T327751: k8s-api for k8s-staging down in codfw as Resolved.Jan 24 2023, 10:54 AM
JMeybohm mentioned this in rLPRIb4bf985c254e: Add prometheus user to the system:monitoring group.Jan 24 2023, 10:54 AM
JMeybohm updated the task description. (Show Details)Jan 24 2023, 5:55 PM
JMeybohm added a subtask: T327786: Update toolhub helm chart to use the mcrouter helm chart module.Jan 24 2023, 6:07 PM
gerritbot added a comment.Jan 25 2023, 10:08 AM

Change 883539 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/alerts@master] KubernetesAPIErrorRate: make alert v1.23 compatible

https://gerrit.wikimedia.org/r/883539

gerritbot added a comment.Jan 27 2023, 1:34 PM

Change 884305 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes: Incease inotify limits

https://gerrit.wikimedia.org/r/884305

gerritbot added a comment.Jan 30 2023, 12:16 PM

Change 884305 merged by JMeybohm:

[operations/puppet@production] kubernetes: Increase inotify limits

https://gerrit.wikimedia.org/r/884305

JMeybohm removed a subtask: T326729: Remove the .Values.kubernetesApi hack.Jan 30 2023, 1:15 PM
JMeybohm removed a subtask: T290963: Drop the use of nonexisting groups in kubernetes infrastructure_users.
JMeybohm removed a subtask: T325066: Migrate charts away from deprecated typology annotations.
JMeybohm removed a subtask: T325620: Re-enable seccomProfile in cert-manager chart after k8s 1.23 migration completed.
JMeybohm removed a subtask: T325268: Migrate use of infrastructure_users tokens to client certificates.
gerritbot added a comment.Jan 30 2023, 1:24 PM

Change 883539 merged by jenkins-bot:

[operations/alerts@master] KubernetesAPIErrorRate: make alert v1.23 compatible

https://gerrit.wikimedia.org/r/883539

JMeybohm closed subtask T326340: Update staging-codfw to k8s 1.23 as Resolved.Jan 30 2023, 2:10 PM
JMeybohm closed subtask T327664: Update staging-eqiad to k8s 1.23 as Resolved.Feb 1 2023, 8:41 AM
JMeybohm closed subtask T321159: Import coredns 1.8.x (k8s 1.23 dependency) as Resolved.Feb 1 2023, 8:51 AM
JMeybohm updated the task description. (Show Details)
JMeybohm removed a subtask: T322919: Metrics changes with Kubernetes v1.23.Feb 2 2023, 2:01 PM
JMeybohm updated the task description. (Show Details)Feb 2 2023, 2:26 PM
gerritbot added a comment.Feb 9 2023, 11:48 AM

Change 887981 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s::package: Ensure the apt component is registered first

https://gerrit.wikimedia.org/r/887981

Clement_Goubert mentioned this in T329193: March 2023 Datacenter Switchover Excluded services.Feb 9 2023, 2:45 PM
JMeybohm added a subtask: T324542: Upgrade ML clusters to Kubernetes 1.23.Feb 13 2023, 2:58 PM
gerritbot added a comment.Feb 14 2023, 5:07 PM

Change 887981 merged by JMeybohm:

[operations/puppet@production] k8s::package: Ensure the apt component is registered first

https://gerrit.wikimedia.org/r/887981

gerritbot added a comment.Feb 15 2023, 9:33 AM

Change 889486 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Kubernets masters: include profile::kubernetes::client

https://gerrit.wikimedia.org/r/889486

gerritbot added a comment.Feb 15 2023, 9:56 AM

Change 889486 merged by JMeybohm:

[operations/puppet@production] Kubernetes masters: include profile::kubernetes::client

https://gerrit.wikimedia.org/r/889486

JMeybohm updated the task description. (Show Details)Feb 15 2023, 10:17 AM
Jdforrester-WMF added a subscriber: Jdforrester-WMF.Feb 15 2023, 4:35 PM
CDanis added a subtask: T329633: Upgrade the aux-eqiad cluster to k8s 1.23.Feb 16 2023, 11:47 AM
Clement_Goubert mentioned this in T329319: What should happen to Toolhub during the 2023 DC switch?.Feb 17 2023, 10:36 AM
gerritbot added a comment.Feb 17 2023, 10:51 AM

Change 889956 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] Add sre.k8s.wipe-cluster.py

https://gerrit.wikimedia.org/r/889956

gerritbot added a comment.Feb 17 2023, 11:52 AM

Change 889956 merged by jenkins-bot:

[operations/cookbooks@master] Add sre.k8s.wipe-cluster.py

https://gerrit.wikimedia.org/r/889956

JMeybohm closed subtask T329633: Upgrade the aux-eqiad cluster to k8s 1.23 as Resolved.Feb 17 2023, 1:30 PM
elukey closed subtask T329556: K8s etcd on bullseye show TLS errors in logs as Resolved.Wed, Feb 22, 8:57 AM
akosiaris closed subtask T329664: Update wikikube codfw to k8s 1.23 as Resolved.Wed, Feb 22, 5:00 PM
JMeybohm added a subtask: T331126: Update wikikube eqiad to k8s 1.23.Fri, Mar 3, 1:54 PM
gerritbot added a comment.Tue, Mar 7, 10:16 AM

Change 895138 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes__deployment_server: Switch to k8s 1.23

https://gerrit.wikimedia.org/r/895138

JMeybohm closed subtask T323993: Upgrade DSE K8 Cluster to 1.23 as Resolved.Tue, Mar 7, 10:20 AM
JMeybohm closed subtask T296303: New Kubernetes nodes may end up with no Pod IPv4 block assigned as Resolved.
JMeybohm removed a subtask: T327786: Update toolhub helm chart to use the mcrouter helm chart module.Tue, Mar 7, 10:24 AM
Stashbot added a comment.Tue, Mar 7, 10:51 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-07T10:51:07Z] <akosiaris> manually label kubemaster1001, kubemaster1002 giving them role master T307943

JMeybohm closed subtask T322193: Import istio 1.1x (k8s 1.23 dependency) as Resolved.Tue, Mar 7, 11:03 AM
JMeybohm closed subtask T310618: Define priorityClassName for istio and cert-manager deployments as Resolved.Tue, Mar 7, 12:24 PM
JMeybohm updated the task description. (Show Details)Tue, Mar 7, 12:28 PM
gerritbot added a comment.Wed, Mar 8, 11:11 AM

Change 895138 merged by Alexandros Kosiaris:

[operations/puppet@production] profile::kubernetes::client: Switch to k8s 1.23

https://gerrit.wikimedia.org/r/895138

Stashbot added a comment.Wed, Mar 8, 11:26 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-08T11:26:08Z] <akosiaris> T307943 upgrade kubernetes-client on deploy1002 deploy2002

akosiaris closed subtask T331126: Update wikikube eqiad to k8s 1.23 as Resolved.Wed, Mar 8, 11:57 AM
akosiaris closed subtask T326617: Decide on new Pod and Sevice IPv4 ranges for wikikube clusters as Resolved.Wed, Mar 8, 12:26 PM
JMeybohm reopened subtask T325292: Update cert-manager to 1.10.x as Open.Mon, Mar 13, 4:51 PM
elukey closed subtask T324542: Upgrade ML clusters to Kubernetes 1.23 as Resolved.Tue, Mar 14, 2:59 PM
JMeybohm closed subtask T325292: Update cert-manager to 1.10.x as Resolved.Wed, Mar 15, 11:15 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL