Remove deprecated ingress objects from existing web services
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Bstorm
	Sep 29 2021, 6:52 PM

Description

Of the ~1000 Kubernetes web services, there are around 700 still using the old path-based ingress objects (tools.wmflabs.org/$tool). These are no longer required since a separate VM handles the redirection.

Scripting a removal of these would reduce the complexity of the nginx configuration in the ingress controllers and improve their health overall. This is especially needed to add routes for gridengine services, per T282975: Create Kubernetes ingress for tools running on the grid engine to remove dynamicproxy, but it also is likely to improve overall stability.

Related Objects

Mentioned Here: T282975: Create Kubernetes ingress for tools running on the grid engine to remove dynamicproxy

Event Timeline

• Bstorm created this task.Sep 29 2021, 6:52 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 29 2021, 6:52 PM

This sort of script is pretty simple if you have admin (kubectl sudo if you are using your local account, etc). I did this to fix a problem in the presets.

#!/bin/bash
# Run this script with your root/cluster admin account as appropriate.
# This will fix the dumps mounts for all existing tools.

set -Eeuo pipefail

declare -a namespaces
readarray -t namespaces < <(kubectl get ns -l tenancy=tool --no-headers=true -o custom-columns=:metadata.name)

for ns in "${namespaces[@]}"
do
    if [[ $(kubectl get podpreset -n "${ns}" --template='{{range .spec.volumeMounts}}{{ if eq .mountPath "/mnt/nfs/dumps-labstore1007.wikimedia.orgs" }}true{{end}}{{end}}' mount-toolforge-vols) == "true" ]]; then
        echo "Fixing ${ns}"
        kubectl -n "$ns" delete podpresets mount-toolforge-vols
        cat <<EOF | kubectl --namespace "$ns" apply -f -
apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
  name: mount-toolforge-vols
  namespace: ${ns}
spec:
  env:
  - name: HOME
    value: /data/project/${ns:5}
  selector:
    matchLabels:
      toolforge: tool
  volumeMounts:
  - mountPath: /public/dumps
    name: dumps
    readOnly: true
  - mountPath: /mnt/nfs/dumps-labstore1007.wikimedia.org
    name: dumpsrc1
    readOnly: true
  - mountPath: /mnt/nfs/dumps-labstore1006.wikimedia.org
    name: dumpsrc2
    readOnly: true
  - mountPath: /data/project
    name: home
  - mountPath: /etc/wmcs-project
    name: wmcs-project
    readOnly: true
  - mountPath: /data/scratch
    name: scratch
  - mountPath: /etc/ldap.conf
    name: etcldap-conf
    readOnly: true
  - mountPath: /etc/ldap.yaml
    name: etcldap-yaml
    readOnly: true
  - mountPath: /etc/novaobserver.yaml
    name: etcnovaobserver-yaml
    readOnly: true
  - mountPath: /var/lib/sss/pipes
    name: sssd-pipes
  volumes:
  - hostPath:
      path: /public/dumps
      type: Directory
    name: dumps
  - hostPath:
      path: /mnt/nfs/dumps-labstore1007.wikimedia.org
      type: Directory
    name: dumpsrc1
  - hostPath:
      path: /mnt/nfs/dumps-labstore1006.wikimedia.org
      type: Directory
    name: dumpsrc2
  - hostPath:
      path: /data/project
      type: Directory
    name: home
  - hostPath:
      path: /etc/wmcs-project
      type: File
    name: wmcs-project
  - hostPath:
      path: /data/scratch
      type: Directory
    name: scratch
  - hostPath:
      path: /etc/ldap.conf
      type: File
    name: etcldap-conf
  - hostPath:
      path: /etc/ldap.yaml
      type: File
    name: etcldap-yaml
  - hostPath:
      path: /etc/novaobserver.yaml
      type: File
    name: etcnovaobserver-yaml
  - hostPath:
      path: /var/lib/sss/pipes
      type: Directory
    name: sssd-pipes
EOF
        echo "created new preset for $ns"
        echo "Finished $ns"
    fi
done

echo "*********************"
echo "Done!"

Just deleting ingress objects with known names is much easier.

taavi claimed this task.Sep 30 2021, 1:24 PM

Restricted Application added a project: User-Majavah. · View Herald TranscriptSep 30 2021, 1:24 PM

Going to run this script:

#!/bin/bash
# Run this script with your root/cluster admin account as appropriate.
# Removes currently-unused ingress objects for old tools.wmflabs.org/$TOOL URLs.

set -Eeuo pipefail

declare -a namespaces
readarray -t namespaces < <(kubectl get ingress -A | grep legacy | awk '{print $1}')

for ns in "${namespaces[@]}"
do
    echo "Fixing ${ns}"
    kubectl delete ing -n ${ns} ${ns:5}-legacy
    sleep 1
done

echo "*********************"
echo "Done!"

Before:

taavi@tools-k8s-control-3:~ $ kubectl top pod -n ingress-nginx-gen2
NAME                                             CPU(cores)   MEMORY(bytes)
ingress-nginx-gen2-controller-786687f976-fj7wv   231m         652Mi
ingress-nginx-gen2-controller-786687f976-mhzkb   155m         553Mi
ingress-nginx-gen2-controller-786687f976-rccdk   303m         374Mi

Mentioned in SAL (#wikimedia-cloud) [2021-09-30T13:43:57Z] <majavah> cleaning up unused kubernetes ingress objects for tools.wmflabs.org urls T292105

After:

taavi@tools-k8s-control-3:~ $ kubectl top pod -n ingress-nginx-gen2
NAME                                             CPU(cores)   MEMORY(bytes)
ingress-nginx-gen2-controller-786687f976-fj7wv   88m          274Mi
ingress-nginx-gen2-controller-786687f976-mhzkb   108m         361Mi
ingress-nginx-gen2-controller-786687f976-rccdk   114m         318Mi

Sweet! Instantly better ingresses.

Remove deprecated ingress objects from existing web servicesClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Remove deprecated ingress objects from existing web services
Closed, ResolvedPublic
Actions