After the second round of mass restarts, 48 tools are still in CrashLoopBackOff state.

Here is an example of one that seems to be due to the missing mount:

$ sudo become superzerocool
$ kubectl get po
NAME                            READY     STATUS             RESTARTS   AGE
superzerocool-109243355-jz0iu   0/1       CrashLoopBackOff   45         92d
$ kubectl logs po/superzerocool-109243355-jz0iu
Traceback (most recent call last):
  File "/usr/bin/webservice-runner", line 4, in <module>
    from toollabs.common import Tool
  File "/usr/lib/python2.7/dist-packages/toollabs/common/__init__.py", line 1, in <module>
    from toollabs.common.tool import Tool
  File "/usr/lib/python2.7/dist-packages/toollabs/common/tool.py", line 6, in <module>
    with open('/etc/wmcs-project', 'r') as _projectfile:
IOError: [Errno 2] No such file or directory: '/etc/wmcs-project'

Mentioned in SAL (#wikimedia-cloud) [2018-06-06T22:00:41Z] <bd808> Scripting a restart of webservice for tools that are still in CrashLoopBackOff state after 2nd attempt (T196589)

On the third pass I used this helper script to only restart webservices that are logging the missing mounted file:

#!/usr/bin/env bash
kubectl logs po/$(kubectl get po|grep CrashLoopBackOff|awk '{print $1}') |
grep wmcs-project &&
webservice restart

After this ran there are 19 pods left in CraashLoopBackOff which seems like a much more reasonable number of broken webservices. The still broken webservices are:

anagrimes
android-maven-repo
articlerequest-dev
best-image
bldrwnsch
commonshelper
enwnbot
ft
himo
loltools
lyan
not-in-the-other-language
r96340-bot
readmore
sparqlblocks
toolschecker
verification-pages
wikidata-exports
wikisource-penguin-classics
wm-commons-emoji-bot

Mentioned in SAL (#wikimedia-cloud) [2018-06-07T00:06:16Z] <bd808> Restarted webservice. Stuck in CrashLoopBackOff due to T196589

It should not. Prior to the reboots there was one case when I deleted the pod but not deployment, and the deployment rebooted the pod using old deployment configuration, but with new docker image, causing the failure. /etc/wmcs-project is specified as a mount option in the webservice command code to configure the deployment afaict, so webservice restart should update the deployment configuration and fix them.

In T196589#4263382, @zhuyifei1999 wrote:

the deployment rebooted the pod using old deployment configuration, but with new docker image, causing the failure.

This actually makes a lot of sense about how we ended up with so many tools in this state. As the reboots rolled across the cluster the Kubernetes reconciliation loop would have replaced pods that disappeared with new ones based on the existing deployment. The new pods would use the latest Docker image, which in turn would contain the version of webservice-runner that required /etc/wmcs-project. Any deployment that predated the addition of that file mount would then cause the crashing problem.

What this does not clearly explain is why some tools required multiple webservice restart cycles in order to end up with a deployment that provided the needed mount. I guess I should look into the restart logic we use for the Kubernetes backend and see if it always or only conditionally tears down and recreates the entire deployment.

In T196589#4264779, @bd808 wrote:

I guess I should look into the restart logic we use for the Kubernetes backend and see if it always or only conditionally tears down and recreates the entire deployment.

webservice restart calls the request_stop() and request_start() methods on the backend (kubernetes in the case we are interested in here). The KubernetesBackend's implementation of request_stop() tries to delete the Service, Deployment, ReplicaSet, and Pod in that order which would clean up everything in the namespace that webservice start initially created. To me this says that only a single webservice restart should have been necessary for each tool.