Page MenuHomePhabricator
Create Task
Maniphest T307943

Update Kubernetes clusters to v1.23
Closed, ResolvedPublic
Actions

Description

We need/want to upgrade our Kubernetes clusters to Kubernetes v1.23

Kubernetes v1.23 was selected as a target:

  • because it is the last version supporting dockershim (and we don't want to move away from that together with this update)
  • v1.24 was only released 2022-05-03 and we usually don't want to upgrade to a just released major/miner

Together with the Kubernetes update, we need to update the following other components (subtasks might be a good idea for each of them when more details available):

Preparation for the Kubernetes update:

  • Double check out docker API version is supported with Kubernetes v1.23 (minimum version is still 1.26.0)
  • Check if migration from command line flags to config files is required for some Kubernetes components: T300499
    • Looks to me like it is not yet strictly needed. It might be for some specific flags, though. That we will figure out during upgrade tests I guess.
  • Ensure all our charts are compatible with Kubernetes v1.23
    • Our current validation with kubeyaml does not support this and getting support for 1.23 into kubeyaml will require time that could be better invested into replacing kubeyaml in deployment-charts CI with kubeconform: T306165
  • Read Kubernetes changelogs (yellow/red flags just linked below each version. Tick the box if all action required items have been addressed), https://relnotes.k8s.io (relevant versions)
  • v1.18
    • Action Required
    • Note
      • The following features are unconditionally enabled and the corresponding --feature-gates flags have been removed: PodPriority, TaintNodesByCondition, ResourceQuotaScopeSelectors and ScheduleDaemonSetPods (#86210, @draveness)
      • Kube-proxy: Added dual-stack IPv4/IPv6 support to the iptables proxier. (#82462, @vllry)
      • Support server-side dry-run in kubectl with --dry-run=server for commands including apply, patch, create, run, annotate, label, set, autoscale, drain, rollout undo, and expose. (#87714, @julianvmodesto)
  • v1.19
    • Action Required
      • seccomp graduates to GA, check if we need to migrate PSPs off the annotations https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#seccomp-graduates-to-general-availability (this only affects pod spec not PSPs, nothing to do here)
      • Kube-apiserver: the componentstatus API is deprecated. This API provided status of etcd, kube-scheduler, and kube-controller-manager components, but only worked when those components were local to the API server, and when kube-scheduler and kube-controller-manager exposed unsecured health endpoints. Instead of this API, etcd health is included in the kube-apiserver health check and kube-scheduler/kube-controller-manager health checks can be made directly against those components' health endpoints. (#93570, @liggitt)
      • ✅ Kubeadm now includes CoreDNS version v1.7.0.
      • ✅ Kube-apiserver: The NodeRestriction admission plugin now restricts Node labels kubelets are permitted to set when creating a new Node to the --node-labels parameters accepted by kubelets in 1.16+. (#90307, @liggitt)
    • Note
      • EndpointSlices, be aware (during debugging) https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
      • Fix bug in reflector that couldn't recover from "Too large resource version" errors (#92537, @wojtek-t) [SIG API Machinery]
      • Kubelet: add '--logging-format' flag to support structured logging (#91532, @afrouzMashaykhi)
      • Add --logging-format flag for component-base. Defaults to "text" using unchanged klog. (#89683, @yuzhiquan)
      • Kube-controller-manager: add '--logging-format' flag to support structured logging (#91521, @SataQiu)
      • Kube-scheduler: add '--logging-format' flag to support structured logging (#91522, @SataQiu)
      • The DefaultIngressClass feature is now GA. The --feature-gate parameter will be removed in 1.20. (#91957, @cmluciano)
      • The kube-controller-manager managed signers can now have distinct signing certificates and keys. See the help about --cluster-signing-[signer-name]-{cert,key}-file. --cluster-signing-{cert,key}-file is still the default. (#90822, @deads2k)2
      • Kube-apiserver, kube-scheduler and kube-controller manager now use SO_REUSEPORT socket option when listening on address defined by --bind-address and --secure-port flags, when running on Unix systems (Windows is NOT supported). This allows to run multiple instances of those processes on a single host with the same configuration, which allows to update/restart them in a graceful way, without causing downtime. (#88893, @invidian)
  • v1.20
    • Action Required
      • ✅ TokenRequest and TokenRequestProjection are now GA features. The following flags are required by the API server:
        • ✅ --service-account-issuer, should be set to a URL identifying the API server that will be stable over the cluster lifetime.
        • ✅ --service-account-key-file, set to one or more files containing one or more public keys used to verify tokens.
        • ✅ --service-account-signing-key-file, set to a file containing a private key to use to sign service account tokens. Can be the same file given to kube-controller-manager with --service-account-private-key-file. (#95896, @zshihang)
      • ✅ Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences: https://github.com/kubernetes-sigs/kubectl-check-ownerreferences
      • ✅ In dual-stack bare-metal clusters, you can now pass dual-stack IPs to kubelet --node-ip. eg: kubelet --node-ip 10.1.0.5,fd01::0005
        • ✅ In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods will now get dual-stack PodIPs.
    • Note
      • A bug was fixed in kubelet where exec probe timeouts were not respected. This may result in unexpected behavior since the default timeout (if not specified) is 1s which may be too small for some exec probes. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts.
      • Kubernetes 1.20 now enables API Priority and Fairness (APF) by default.
      • IPv4/IPv6 dual-stack has been reimplemented for 1.20 to support dual-stack Services: https://docs.k8s.io/concepts/services-networking/dual-stack/
      • On-demand metrics calculation is now available through /metrics/resources
      • kubectl alpha debug graduates from alpha to beta in 1.20, becoming kubectl debug
      • Support the node label node.kubernetes.io/exclude-from-external-load-balancers (might be an idea to exclude ganeti VM nodes from LVS?)
  • v1.21
    • Action Required
      • ✅ Kubeadm now includes CoreDNS v1.8.0
      • ✅ New admission controller DenyServiceExternalIPs is available. Clusters which do not need the Service externalIPs feature should enable this controller and be more secure. (#97395, @thockin)
      • ✅ The pause image upgraded to v3.4.1 in kubelet and kubeadm for both Linux and Windows. (#98205, @pacoxu) T322920
      • ✅ Update the latest validated version of Docker to 20.10
      • ✅ Upgrades IPv6Dualstack to Beta and turns it on by default. New clusters or existing clusters are not be affected until an actor starts adding secondary Pods and service CIDRS CLI flags as described here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack (#98969, @khenidak)
    • Note
      • Pod with multiple containers can use kubectl.kubernetes.io/default-container annotation to have a container preselected for kubectl commands.
      • Immutable Secrets and ConfigMaps graduates to GA. This feature allows users to specify that the contents of a particular Secret or ConfigMap is immutable for its object lifetime. For such instances, Kubelet will not watch/poll for changes and therefore reducing apiserver load. (Probably true for almost all of our configmap/secret objects as we roll-restart deployments on configmap changes anyways).
      • ServiceNodeExclusion, NodeDisruptionExclusion and LegacyNodeRoleBehavior features have been promoted to GA. ServiceNodeExclusion and NodeDisruptionExclusion are now unconditionally enabled, while LegacyNodeRoleBehavior is unconditionally disabled.
      • ✅ TokenRequest and TokenRequestProjection feature gates have been removed and are unconditionally enabled
      • Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default.
      • Namespace API objects now have a kubernetes.io/metadata.name label matching their metadata.name field to allow selecting any namespace by its name using a label selector.
      • Kubectl: kubectl get will omit managed fields by default now. Users could set --show-managed-fields to true to show managedFields when the output format is either json or yaml. (#96878, @knight42)
  • v1.22
    • Action Required
      • ✅ Various beta API removals. We're not affected as kubeconform would have given notice
      • ✅ controller-manager changes:
        • ✅ controller-manager MUST start with --authorization-kubeconfig and --authentication-kubeconfig correctly set to get authentication/authorization working
        • ✅ Applications that fetch metrics from controller-manager should use a dedicated service account which is allowed to access nonResourceURLs /metrics. (#96216, @knight42)
        • ✅ (don't think we use that) liveness/readiness probes to controller-manager MUST use HTTPS now, and the default port has been changed to 10257
      • ✅ Updated pause image to version 3.5, which now runs per default as pseudo user and group 65535:65535. This does not have any effect on remote container runtimes like CRI-O and containerd, which setup the pod sandbox user and group on their own. (#100292, @saschagrunert) T322920
    • Note
      • As of now both system-node-critical and system-cluster-critical pods have -997 OOM score, making them one of the last processes to be OOMKilled. If the user wants to have the pod to be OOMKilled last and the pod has system-cluster-critical priority class, it has to be changed to system-node-critical priority class to preserve the existing behavior (#99729, @ravisantoshgudimetla)
      • Server-side Apply is GA https://kubernetes.io/docs/reference/using-api/server-side-apply/
      • Default/kubeadm etcd moves to version 3.5.0
        • Data ccorruption issues with etcd 3.5.[0-2], use >=3.5.3
      • Introducing Memory quality of service support with cgroups v2 (Alpha). The MemoryQoS feature is now in Alpha. This allows kubelet running with cgroups v2 to set memory QoS at container, pod and QoS level to protect and guarantee better memory quality. This feature can be enabled through feature gate Memory QoS. (#102970, @borgerli)
      • Kube-apiserver: the alpha PodSecurity feature can be enabled by passing --feature-gates=PodSecurity=true, and enables controlling allowed pods using namespace labels. See https://git.k8s.io/enhancements/keps/sig-auth/2579-psp-replacement for more details. (#103099, @liggitt)
      • The EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. (#101048, @dims)
      • The NamespaceDefaultLabelName is promoted to GA in this release. All Namespace API objects have a kubernetes.io/metadata.name label matching their metadata.name field to allow selecting any namespace by its name using a label selector. (#101342, @rosenhouse)
  • v1.23
    • Action Required
    • Note
      • IPv4/IPv6 Dual-stack Networking graduates to GA https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack
      • PodSecurity graduates to Beta (In 1.23, the PodSecurity feature gate is enabled by default.) https://kubernetes.io/docs/concepts/security/pod-security-admission/
      • Structured logging graduate to Beta
      • Log messages in JSON format are written to stderr by default now (same as text format) instead of stdout. Users who expected JSON output on stdout must now capture stderr instead or in addition to stdout. (#106146, @pohly) (we log zu stderr anyways, so probably no change here)
      • Support for the seccomp annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io/[name] has been deprecated since 1.19, will be dropped in 1.25. Transition to using the seccompProfile API field. (#104389, @saschagrunert)
      • Ephemeral containers graduated to beta and are now available by default. (#105405, @verb)
      • The TTLAfterFinished feature gate is now GA and enabled by default. (#105219, @sahilvv)
      • Introduce a feature gate DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers. (#102507, @ostrain)
  • Read Calico changelogs
    • v3.17 > 3.17.0
      • All components that use Typha now use the same logic to discover Typha’s address. They lookup the endpoints of the service directly and connect to one at random. This avoids a dependency on kube-proxy. typha #466 (@fasaxc)
      • kube-controllers runs a non-root by default kube-controllers #566 (@caseydavenport)
    • v3.18
    • v3.19
      • Update ipables version to 1.8.4-15
      • By default, limit each node to 20 IP address blocks. This value can be overridden through IPAM configuration.
    • v3.20
      • Service-based egress rules; Calico NetworkPolicy and GlobalNetworkPolicy now support egress rules which match on Kubernetes service names. Service matches in egress rules can be used to allow or deny access to in-cluster services, as well as services typically not backed by pods (for example, the Kubernetes API). Address and port information is learned from the individual endpoints within the service.
      • Configurable BGP graceful restart timer; See the maxRestartTime configuration option in the BGPPeer API.
      • calico/node marks nodes with NetworkUnavailable=true on shutdown node #993 (@song-jiang)
      • Add IP address garbage collection to kube-controllers kube-controllers #744 (@caseydavenport)
      • Calico will now release empty IPAM blocks from nodes that no longer need them so they can be used elsewhere. kube-controllers #799 (@caseydavenport)
    • v3.21
      • For users of BGP you can now view the status of your BGP routers, including session status, RIB / FIB contents, and agent health via the new CalicoNodeStatus API: https://docs.projectcalico.org/archive/v3.21/reference/resources/caliconodestatus
      • Service-based ingress rules; In v3.20, we introduced egress policy rules that can match on Kubernetes services. In v3.21, we improved upon that in two ways. First, you can now use service matches in Calico NetworkPolicy and GlobalNetworkPolicy ingress rules. Second, you can now use service-based network policy rules on Windows nodes.
      • Option to run Calico as non-privileged and non-root; https://docs.projectcalico.org/archive/v3.21/security/non-privileged
      • ACTION REQUIRED: calico/node logs write to /var/log/calico within the container by default, in addition to stdout node #1133 (@song-jiang)
    • v3.22
      • None
    • v3.23
      • Update to CNI plugins v1.1.1 calico #5944 (@caseydavenport)
      • New per-pool IPAM metrics added calico #5706 (@pasanw)

Additional things to do together with the re-init of clusters:

  • Move to bigger Pod and Service IP range (T326617)
  • Delete etcd v2 datastore of calico: $ etcdctl -C https://$(hostname -f):2379 rm -r /calico
  • Reimage etcd clusters to bullseye (so the above is probably not needed)
  • enable DenyServiceExternalIPs admission plugin

Details

SubjectRepoBranchLines +/-
Revert "k8s: Configure the IPv6 service ip range for apiserver"operations/puppetproduction+1 -6
k8s: Configure the IPv6 service ip range for apiserveroperations/puppetproduction+6 -1
profile::kubernetes::client: Switch to k8s 1.23operations/puppetproduction+4 -5
Add sre.k8s.wipe-cluster.pyoperations/cookbooksmaster+166 -0
Kubernetes masters: include profile::kubernetes::clientoperations/puppetproduction+41 -49
k8s::package: Ensure the apt component is registered firstoperations/puppetproduction+3 -1
KubernetesAPIErrorRate: make alert v1.23 compatibleoperations/alertsmaster+15 -2
kubernetes: Increase inotify limitsoperations/puppetproduction+10 -0
Add prometheus user to the system:monitoring grouplabs/privatemaster+4 -0
Pin coredns, eventrouter and helm-state-metrics for k8s 1.16operations/deployment-chartsmaster+15 -0
Update staging-codfw to k8s 1.23operations/deployment-chartsmaster+11 -1
k8s: Update staging-codfw to kubernetes 1.23operations/puppetproduction+47 -2
k8s: Remove default kubelet_cluster_domain definitionsoperations/puppetproduction+0 -6
k8s: Add the ClusterIP of kubernetes.default.cluster.local to certoperations/puppetproduction+4 -0
If-guard admin_ng objects no longer relevant for k8s 1.23operations/deployment-chartsmaster+20 -4
k8s: Remove authz_mode hiera keyoperations/puppetproduction+4 -22
k8s: Add support for PKI with k8s >= 1.23operations/puppetproduction+429 -130
pki: Add intermediate certifikates for wikikube and wikikube_stagingoperations/puppetproduction+89 -0
k8s: Fix duplicate definition of --service-account-key-fileoperations/puppetproduction+1 -2
k8s: Add a central ipv6dualstack flag to enable dual stackoperations/puppetproduction+40 -23
k8s: Refactor profile::kubernetes::master::service_cluster_ip_rangeoperations/puppetproduction+48 -19
k8s: make profile::kubernetes::cluster_cidr mandatoryoperations/puppetproduction+44 -4
k8s: Stop docker/runc spam from being written to syslogoperations/puppetproduction+8 -0
Update to v1.23.14operations/debs/kubernetesv1.23+6 -0
calico: Allow different versions, drop pre bullseye supportoperations/puppetproduction+26 -45
calico: Allow calico-cni access to ipreservationsoperations/deployment-chartsmaster+3 -2
calico: More calico 3.23.3 additions.operations/deployment-chartsmaster+29 -84
calico: Align formatting with k8s module and profilesoperations/puppetproduction+12 -14
Update calico to v3.23.3operations/deployment-chartsmaster+72 -11
calico: Allow alternative dnsConfig and disabling of IPv6operations/deployment-chartsmaster+30 -3
Add --service-account* flags for TokenRequestoperations/puppetproduction+29 -13
k8s: Use the K8s::Core::V1Taint typeoperations/puppetproduction+83 -41
k8s: Add version switching where neededoperations/puppetproduction+42 -16
Make Kubernetes version configurableoperations/puppetproduction+66 -91
kubernetes: Actually use the master_fqdn instead of the cert nameoperations/puppetproduction+3 -4
kubernetes::master fail if user tokens are not uniqueoperations/puppetproduction+10 -2
kubernetes::master remove apiserver_countoperations/puppetproduction+0 -12
Update to Kubernetes v1.23.12operations/debs/kubernetesv1.23+19 -18
k8s: Remove all debian version if-guardingoperations/puppetproduction+19 -94
aptrepo: Create versioned kubernetes components for busteroperations/puppetproduction+2 -0
k8s: Align formatting along k8s module and profilesoperations/puppetproduction+65 -75
aptrepo: Add bullseye components calico323 and kubernetes123operations/puppetproduction+2 -0
Update calico to v3.23.3operations/debs/calicov3.23+25 -6
Update calico-crds to v3.23.3operations/deployment-chartsmaster+921 -141
admin_ng: Allow to pin calico chart versions per environmentoperations/deployment-chartsmaster+15 -1
Update to v3.20.6operations/debs/calicov3.20+8 -2
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• JMeybohmT307943 Update Kubernetes clusters to v1.23
 Resolved• JMeybohmT270191 Add kubernetes 1.17+ topology annotations
 Resolved• JMeybohmT278329 Support multiple kubernetes versions with puppet
 Resolved• JMeybohmT270271 Target Sources (component/kubernetes-future/source/Sources) is configured multiple times
 Resolved• JMeybohmT300499 Migrate from command line flags to config files for kubernetes components
Restricted Task
 ResolvedayounsiT306649 Agree strategy for Kubernetes BGP peering to top-of-rack switches
 ResolvedelukeyT308418 Add missing failure domain labels to ml-serve-* clusters
 Resolved• JMeybohmT299236 Move away from system:node RBAC role
 Resolved• JMeybohmT306165 Replace kubeyaml in deployment-charts CI
 ResolvedClement_GoubertT316348 Remove kubeyaml from deployment-charts CI
 Resolved• JMeybohmT303279 Fix calico, cfssl-issuer and knative-serving Helm dependencies
 Resolved• JMeybohmT303184 High API server request latencies (LIST)
Resolved• JMeybohmT310486 Update cfssl-issuer to cert-manager 1.8.x
 Resolved• JMeybohmT310618 Define priorityClassName for istio and cert-manager deployments
 Resolved• JMeybohmT311251 Migrate kubernetes alerts away from icinga
Restricted Task
 Resolved• JMeybohmT313473 Switch to cgroup v2 and systemd as cgroup driver for docker and kubelet
 ResolvedClement_GoubertT315977 Clean up profile::docker::storage from puppet
Restricted Task
 ResolvedelukeyT321159 Import coredns 1.8.x (k8s 1.23 dependency)
 ResolvedelukeyT322193 Import istio 1.1x (k8s 1.23 dependency)
Restricted Task
 Resolved• JMeybohmT277876 Reserve resources for system daemons on kubernetes nodes
 ResolvedClement_GoubertT346422 Move 10% of mediawiki external requests to mw on k8s
 ResolvedelukeyT322920 Import pause container image >= 3.5 (k8s 1.23 dependency)
 OpenNoneT277677 Write a cookbook to set a k8s cluster in maintenance mode
 Resolved• JMeybohmT260663 Create a cookbook for depooling one or all services from one kubernetes cluster
 ResolvedjijikiT287491 Allow to address Kubernetes API servers from NetworkPolicy
 ResolvedklausmanT365479 Update kserve and knative-serving charts for new-style Calico network policies
 ResolvedCDanisT365855 Stop hardcoding k8s master (k8s API) endpoint IP addresses
 ResolvedelukeyT323793 Import new knative-serving version (k8s 1.23 dependency for ML)
 Resolved• JMeybohmT323993 Upgrade DSE K8 Cluster to 1.23
 Resolved• JMeybohmT324959 Scrape controller-manager and scheduler metrics
 Resolved• JMeybohmT296303 New Kubernetes nodes may end up with no Pod IPv4 block assigned
 Resolved• JMeybohmT325292 Update cert-manager to 1.10.x
 Resolved• JMeybohmT326340 Update staging-codfw to k8s 1.23
 ResolvedBTullisT327799 Datahub errors in staging-codfw
 Resolved• JMeybohmT326617 Decide on new Pod and Sevice IPv4 ranges for wikikube clusters
 ResolvedMarosteguiT331508 Remove deprecated toolhub wikikube grants
 Resolved• JMeybohmT327664 Update staging-eqiad to k8s 1.23
 Resolved• JMeybohmT327751 k8s-api for k8s-staging down in codfw
 Resolved• JMeybohmT328291 Post Kubernetes v1.23 cleanup
 Resolved• JMeybohmT326729 Remove the .Values.kubernetesApi hack
 Resolved• JMeybohmT290963 Drop the use of nonexisting groups in kubernetes infrastructure_users
 Resolved• JMeybohmT325066 Migrate charts away from deprecated topology annotations
 Resolved• JMeybohmT325620 Re-enable seccomProfile in cert-manager chart after k8s 1.23 migration completed
 Resolved• JMeybohmT325268 Migrate use of infrastructure_users tokens to client certificates
 Resolved• JMeybohmT322919 Metrics changes with Kubernetes v1.23
 OpenNoneT337826 Write a wrapper function combining pki::get_cert and k8s::kubeconfig
 Resolved• JMeybohmT329826 Kubernetes v1.23 use PKI for service-account signing (instead of cergen)
 Resolved• JMeybohmT341669 Allow for multiple confd instances in puppet
 Resolved• JMeybohmT380142 Reimaging a kubernetes control-plane invalidates service-account tokens issued by it
 Resolved• JMeybohmT335285 Selected IPv6 service-cluster-up ranges are to big
 ResolvedelukeyT324542 Upgrade ML clusters to Kubernetes 1.23
 ResolvedelukeyT327767 Upgrade the ml-staging-codfw cluster to k8s 1.23
 DeclinedNoneT330662 Upgrade the ml-etcd clusters to bullseye and PKI
 ResolvedNoneT330669 Upgrade the ml-serve-codfw cluster to k8s 1.23
 ResolvedelukeyT330758 Upgrade the ml-serve-eqiad cluster to k8s 1.23
 ResolvedelukeyT329556 K8s etcd on bullseye show TLS errors in logs
 Resolved• JMeybohmT329717 Switch wikikube-staging (codfw and eqiad) etcd clusters to use PKI
 Resolved• JMeybohmT329664 Update wikikube codfw to k8s 1.23
 ResolvedCDanisT329633 Upgrade the aux-eqiad cluster to k8s 1.23
 OpenNoneT330060 etcd cluster reimage strategies to use with the K8s upgrade cookbook
 ResolvedakosiarisT331126 Update wikikube eqiad to k8s 1.23

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Dec 12 2022, 2:49 PM

Change 867182 merged by JMeybohm:

[operations/puppet@production] pki: Add intermediate certifikates for wikikube and wikikube_staging

https://gerrit.wikimedia.org/r/867182

• JMeybohm added a subtask: T296303: New Kubernetes nodes may end up with no Pod IPv4 block assigned.Dec 13 2022, 8:42 AM
gerritbot added a comment.Dec 13 2022, 12:06 PM

Change 865592 merged by JMeybohm:

[operations/puppet@production] k8s: Add support for PKI with k8s >= 1.23

https://gerrit.wikimedia.org/r/865592

gerritbot added a comment.Dec 13 2022, 12:06 PM

Change 866444 merged by JMeybohm:

[operations/puppet@production] k8s: Remove authz_mode hiera key

https://gerrit.wikimedia.org/r/866444

• JMeybohm closed subtask T303279: Fix calico, cfssl-issuer and knative-serving Helm dependencies as Resolved.Dec 13 2022, 2:03 PM
• JMeybohm closed subtask T270191: Add kubernetes 1.17+ topology annotations as Resolved.Dec 14 2022, 3:04 PM
gerritbot added a comment.Dec 15 2022, 10:18 AM

Change 868232 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] If-guard admin_ng objects no longer relevant for k8s 1.23

https://gerrit.wikimedia.org/r/868232

gerritbot added a comment.Dec 15 2022, 12:25 PM

Change 868389 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] WIP: Update staging-codfw to k8s 1.23

https://gerrit.wikimedia.org/r/868389

• JMeybohm updated the task description. (Show Details)Dec 15 2022, 12:29 PM
gerritbot added a comment.Dec 15 2022, 1:20 PM

Change 868232 merged by jenkins-bot:

[operations/deployment-charts@master] If-guard admin_ng objects no longer relevant for k8s 1.23

https://gerrit.wikimedia.org/r/868232

• JMeybohm closed subtask T299236: Move away from system:node RBAC role as Resolved.Dec 15 2022, 1:44 PM
• JMeybohm updated the task description. (Show Details)Dec 15 2022, 1:47 PM
• JMeybohm added a subtask: T325620: Re-enable seccomProfile in cert-manager chart after k8s 1.23 migration completed.Dec 20 2022, 9:56 AM
gerritbot added a comment.Dec 22 2022, 10:57 AM

Change 870820 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Add the ClusterIP of kubernetes.default.cluster.local to cert

https://gerrit.wikimedia.org/r/870820

gerritbot added a comment.Jan 3 2023, 1:56 PM

Change 870820 merged by JMeybohm:

[operations/puppet@production] k8s: Add the ClusterIP of kubernetes.default.cluster.local to cert

https://gerrit.wikimedia.org/r/870820

• JMeybohm closed subtask T325292: Update cert-manager to 1.10.x as Resolved.Jan 5 2023, 2:30 PM
gerritbot added a comment.Jan 10 2023, 10:09 AM

Change 877963 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Remove default kubelet_cluster_domain definitions

https://gerrit.wikimedia.org/r/877963

gerritbot added a comment.Jan 10 2023, 10:42 AM

Change 877990 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Update staging-codfw to kubernetes 1.23

https://gerrit.wikimedia.org/r/877990

• JMeybohm updated the task description. (Show Details)Jan 10 2023, 10:50 AM
gerritbot added a comment.Jan 10 2023, 11:09 AM

Change 877963 merged by JMeybohm:

[operations/puppet@production] k8s: Remove default kubelet_cluster_domain definitions

https://gerrit.wikimedia.org/r/877963

gerritbot added a comment.Jan 10 2023, 5:10 PM

Change 877990 merged by JMeybohm:

[operations/puppet@production] k8s: Update staging-codfw to kubernetes 1.23

https://gerrit.wikimedia.org/r/877990

gerritbot added a comment.Jan 10 2023, 7:29 PM

Change 868389 merged by jenkins-bot:

[operations/deployment-charts@master] Update staging-codfw to k8s 1.23

https://gerrit.wikimedia.org/r/868389

gerritbot added a comment.Jan 11 2023, 1:47 PM

Change 878940 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Pin coredns, eventrouter and helm-state-metrics for k8s 1.16

https://gerrit.wikimedia.org/r/878940

gerritbot added a comment.Jan 11 2023, 4:57 PM

Change 878940 merged by jenkins-bot:

[operations/deployment-charts@master] Pin coredns, eventrouter and helm-state-metrics for k8s 1.16

https://gerrit.wikimedia.org/r/878940

elukey closed subtask T323793: Import new knative-serving version (k8s 1.23 dependency for ML) as Resolved.Jan 17 2023, 10:06 AM
• JMeybohm updated the task description. (Show Details)Jan 23 2023, 3:52 PM
gerritbot added a comment.Jan 24 2023, 10:42 AM

Change 883130 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[labs/private@master] Add prometheus user to the system:monitoring group

https://gerrit.wikimedia.org/r/883130

gerritbot added a comment.Jan 24 2023, 10:44 AM

Change 883130 merged by JMeybohm:

[labs/private@master] Add prometheus user to the system:monitoring group

https://gerrit.wikimedia.org/r/883130

• JMeybohm closed subtask T327751: k8s-api for k8s-staging down in codfw as Resolved.Jan 24 2023, 10:54 AM
• JMeybohm mentioned this in rLPRIb4bf985c254e: Add prometheus user to the system:monitoring group.Jan 24 2023, 10:54 AM
• JMeybohm updated the task description. (Show Details)Jan 24 2023, 5:55 PM
• JMeybohm added a subtask: T327786: Update toolhub helm chart to use the mcrouter helm chart module.Jan 24 2023, 6:07 PM
gerritbot added a comment.Jan 25 2023, 10:08 AM

Change 883539 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/alerts@master] KubernetesAPIErrorRate: make alert v1.23 compatible

https://gerrit.wikimedia.org/r/883539

gerritbot added a comment.Jan 27 2023, 1:34 PM

Change 884305 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes: Incease inotify limits

https://gerrit.wikimedia.org/r/884305

gerritbot added a comment.Jan 30 2023, 12:16 PM

Change 884305 merged by JMeybohm:

[operations/puppet@production] kubernetes: Increase inotify limits

https://gerrit.wikimedia.org/r/884305

• JMeybohm removed a subtask: T326729: Remove the .Values.kubernetesApi hack.Jan 30 2023, 1:15 PM
• JMeybohm removed a subtask: T290963: Drop the use of nonexisting groups in kubernetes infrastructure_users.
• JMeybohm removed a subtask: T325066: Migrate charts away from deprecated topology annotations.
• JMeybohm removed a subtask: T325620: Re-enable seccomProfile in cert-manager chart after k8s 1.23 migration completed.
• JMeybohm removed a subtask: T325268: Migrate use of infrastructure_users tokens to client certificates.
gerritbot added a comment.Jan 30 2023, 1:24 PM

Change 883539 merged by jenkins-bot:

[operations/alerts@master] KubernetesAPIErrorRate: make alert v1.23 compatible

https://gerrit.wikimedia.org/r/883539

• JMeybohm closed subtask T326340: Update staging-codfw to k8s 1.23 as Resolved.Jan 30 2023, 2:10 PM
• JMeybohm closed subtask T327664: Update staging-eqiad to k8s 1.23 as Resolved.Feb 1 2023, 8:41 AM
• JMeybohm closed subtask T321159: Import coredns 1.8.x (k8s 1.23 dependency) as Resolved.Feb 1 2023, 8:51 AM
• JMeybohm updated the task description. (Show Details)
• JMeybohm removed a subtask: T322919: Metrics changes with Kubernetes v1.23.Feb 2 2023, 2:01 PM
• JMeybohm updated the task description. (Show Details)Feb 2 2023, 2:26 PM
gerritbot added a comment.Feb 9 2023, 11:48 AM

Change 887981 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s::package: Ensure the apt component is registered first

https://gerrit.wikimedia.org/r/887981

Clement_Goubert mentioned this in T329193: March 2023 Datacenter Switchover Excluded services.Feb 9 2023, 2:45 PM
• JMeybohm added a subtask: T324542: Upgrade ML clusters to Kubernetes 1.23.Feb 13 2023, 2:58 PM
gerritbot added a comment.Feb 14 2023, 5:07 PM

Change 887981 merged by JMeybohm:

[operations/puppet@production] k8s::package: Ensure the apt component is registered first

https://gerrit.wikimedia.org/r/887981

gerritbot added a comment.Feb 15 2023, 9:33 AM

Change 889486 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Kubernets masters: include profile::kubernetes::client

https://gerrit.wikimedia.org/r/889486

gerritbot added a comment.Feb 15 2023, 9:56 AM

Change 889486 merged by JMeybohm:

[operations/puppet@production] Kubernetes masters: include profile::kubernetes::client

https://gerrit.wikimedia.org/r/889486

• JMeybohm updated the task description. (Show Details)Feb 15 2023, 10:17 AM
Jdforrester-WMF subscribed.Feb 15 2023, 4:35 PM
CDanis added a subtask: T329633: Upgrade the aux-eqiad cluster to k8s 1.23.Feb 16 2023, 11:47 AM
Clement_Goubert mentioned this in T329319: What should happen to Toolhub during the 2023 DC switch?.Feb 17 2023, 10:36 AM
gerritbot added a comment.Feb 17 2023, 10:51 AM

Change 889956 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] Add sre.k8s.wipe-cluster.py

https://gerrit.wikimedia.org/r/889956

gerritbot added a comment.Feb 17 2023, 11:52 AM

Change 889956 merged by jenkins-bot:

[operations/cookbooks@master] Add sre.k8s.wipe-cluster.py

https://gerrit.wikimedia.org/r/889956

• JMeybohm closed subtask T329633: Upgrade the aux-eqiad cluster to k8s 1.23 as Resolved.Feb 17 2023, 1:30 PM
elukey closed subtask T329556: K8s etcd on bullseye show TLS errors in logs as Resolved.Feb 22 2023, 8:57 AM
akosiaris closed subtask T329664: Update wikikube codfw to k8s 1.23 as Resolved.Feb 22 2023, 5:00 PM
• JMeybohm added a subtask: T331126: Update wikikube eqiad to k8s 1.23.Mar 3 2023, 1:54 PM
gerritbot added a comment.Mar 7 2023, 10:16 AM

Change 895138 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes__deployment_server: Switch to k8s 1.23

https://gerrit.wikimedia.org/r/895138

• JMeybohm closed subtask T323993: Upgrade DSE K8 Cluster to 1.23 as Resolved.Mar 7 2023, 10:20 AM
• JMeybohm closed subtask T296303: New Kubernetes nodes may end up with no Pod IPv4 block assigned as Resolved.
• JMeybohm removed a subtask: T327786: Update toolhub helm chart to use the mcrouter helm chart module.Mar 7 2023, 10:24 AM
Stashbot added a comment.Mar 7 2023, 10:51 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-07T10:51:07Z] <akosiaris> manually label kubemaster1001, kubemaster1002 giving them role master T307943

• JMeybohm closed subtask T322193: Import istio 1.1x (k8s 1.23 dependency) as Resolved.Mar 7 2023, 11:03 AM
• JMeybohm closed subtask T310618: Define priorityClassName for istio and cert-manager deployments as Resolved.Mar 7 2023, 12:24 PM
• JMeybohm updated the task description. (Show Details)Mar 7 2023, 12:28 PM
gerritbot added a comment.Mar 8 2023, 11:11 AM

Change 895138 merged by Alexandros Kosiaris:

[operations/puppet@production] profile::kubernetes::client: Switch to k8s 1.23

https://gerrit.wikimedia.org/r/895138

Stashbot added a comment.Mar 8 2023, 11:26 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-08T11:26:08Z] <akosiaris> T307943 upgrade kubernetes-client on deploy1002 deploy2002

akosiaris closed subtask T331126: Update wikikube eqiad to k8s 1.23 as Resolved.Mar 8 2023, 11:57 AM
akosiaris closed subtask T326617: Decide on new Pod and Sevice IPv4 ranges for wikikube clusters as Resolved.Mar 8 2023, 12:26 PM
• JMeybohm reopened subtask T325292: Update cert-manager to 1.10.x as Open.Mar 13 2023, 4:51 PM
elukey closed subtask T324542: Upgrade ML clusters to Kubernetes 1.23 as Resolved.Mar 14 2023, 2:59 PM
• JMeybohm closed subtask T325292: Update cert-manager to 1.10.x as Resolved.Mar 15 2023, 11:15 AM
gerritbot added a comment.Mar 28 2023, 2:24 PM

Change 903655 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] k8s: Configure the IPv6 service ip range for apiserver

https://gerrit.wikimedia.org/r/903655

ayounsi changed the status of subtask T306649: Agree strategy for Kubernetes BGP peering to top-of-rack switches from Open to Stalled.Apr 5 2023, 7:59 AM
gerritbot added a comment.Apr 24 2023, 12:37 PM

Change 903655 merged by JMeybohm:

[operations/puppet@production] k8s: Configure the IPv6 service ip range for apiserver

https://gerrit.wikimedia.org/r/903655

gerritbot added a comment.Apr 24 2023, 12:53 PM

Change 910873 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Revert "k8s: Configure the IPv6 service ip range for apiserver"

https://gerrit.wikimedia.org/r/910873

gerritbot added a comment.Apr 24 2023, 12:54 PM

Change 910873 merged by JMeybohm:

[operations/puppet@production] Revert "k8s: Configure the IPv6 service ip range for apiserver"

https://gerrit.wikimedia.org/r/910873

• JMeybohm closed this task as Resolved.May 31 2023, 2:34 PM
• JMeybohm updated the task description. (Show Details)

I'm going to resolve this as the update is done. There are a couple of tasks below T328291: Post Kubernetes v1.23 cleanup and some below this one which we will kick down the road.

• JMeybohm closed subtask T335285: Selected IPv6 service-cluster-up ranges are to big as Resolved.Jul 5 2023, 10:55 AM
• JMeybohm changed the status of subtask T277876: Reserve resources for system daemons on kubernetes nodes from Open to Stalled.Aug 21 2023, 8:32 AM
• JMeybohm closed subtask T329826: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) as Resolved.Sep 19 2023, 10:01 AM
• JMeybohm closed subtask T328291: Post Kubernetes v1.23 cleanup as Resolved.
• JMeybohm mentioned this in T275026: Use a separate key for service account token issuer.Sep 20 2023, 5:00 PM
• JMeybohm closed subtask T324959: Scrape controller-manager and scheduler metrics as Resolved.Sep 21 2023, 2:40 PM
• JMeybohm closed subtask T277876: Reserve resources for system daemons on kubernetes nodes as Resolved.Sep 26 2023, 9:14 AM
ayounsi closed subtask T306649: Agree strategy for Kubernetes BGP peering to top-of-rack switches as Resolved.Dec 12 2023, 9:54 AM
jijiki closed subtask T287491: Allow to address Kubernetes API servers from NetworkPolicy as Resolved.Jul 12 2024, 10:24 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits