from openstack controller:
```
openstack coe cluster create paws --cluster-template paws-k8s21 --master-count 1 --node-count 3
# get kube config
openstack coe cluster config paws --dir /tmp/
cat /tmp/config
```
from tools bastion (tools-sgebastion-10.tools.eqiad1.wikimedia.cloud):
Put the output into .kube/config, or somewhere and `export KUBECONFIG=<location>`
```
helm upgrade --install ingress-nginx ingress-nginx \
--version v4.4.0 \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--set controller.service.type=NodePort \
--set controller.service.enableHttps=false \
--set controller.service.nodePorts.http=30001
git clone https://github.com/toolforge/paws.git # decrypt if necessary
git checkout T326268 # checkout the updated secrets file for the new db
kubectl config set-context --current --namespace=prod
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm dep up paws/
kubectl create namespace prod
helm install paws --namespace prod ./paws -f paws/secrets.yaml -f paws/production.yaml --timeout=50m
kubectl apply -f manifests/psp.yaml
```
Update haproxy nodes (paws-k8s-haproxy-2.paws.eqiad1.wikimedia.cloud and paws-k8s-haproxy-1.paws.eqiad1.wikimedia.cloud)
/etc/haproxy/conf.d/k8s-ingress.cfg:
```
server paws-k8s-ingress-3.paws.eqiad1.wikimedia.cloud paws-k8s-ingress-3.paws.eqiad1.wikimedia.cloud:30002 check
server paws-k8s-ingress-4.paws.eqiad1.wikimedia.cloud paws-k8s-ingress-4.paws.eqiad1.wikimedia.cloud:30002 check
```
to:
```
server <ip of worker1> <ip of worker1>:30001 check
server <ip of worker2> <ip of worker2>:30001 check
server <ip of worker3> <ip of worker3>:30001 check
```
`systemctl restart haproxy`