Change Details

``` [17:25 UTC] krinkle at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ sudo -iu tools.nagf [18:06 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl get pods NAME READY STATUS RESTARTS AGE nagf-wu6jm 1/1 Running 0 4d [18:06 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl rolling-update nagf --image=yuvipanda/nagf Created nagf-6899d476598983c219c3beb256c8445f Scaling up nagf-6899d476598983c219c3beb256c8445f from 0 to 1, scaling down nagf from 1 to 0 (keep 1 pods available, don't exceed 2 pods) Scaling nagf-6899d476598983c219c3beb256c8445f up to 1 Scaling nagf down to 0 Update succeeded. Deleting old controller: nagf Renaming nagf-6899d476598983c219c3beb256c8445f to nagf replicationcontroller "nagf" rolling updated ``` About half-way through the above, <https://tools.wmflabs.org/nagf/> Nginx started to respond with HTTP 500 Internal Server Error, and hasn't recovered since.

``` [17:25 UTC] krinkle at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ sudo -iu tools.nagf [18:06 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl get pods NAME READY STATUS RESTARTS AGE nagf-wu6jm 1/1 Running 0 4d [18:06 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl rolling-update nagf --image=yuvipanda/nagf Created nagf-6899d476598983c219c3beb256c8445f Scaling up nagf-6899d476598983c219c3beb256c8445f from 0 to 1, scaling down nagf from 1 to 0 (keep 1 pods available, don't exceed 2 pods) Scaling nagf-6899d476598983c219c3beb256c8445f up to 1 Scaling nagf down to 0 Update succeeded. Deleting old controller: nagf Renaming nagf-6899d476598983c219c3beb256c8445f to nagf replicationcontroller "nagf" rolling updated [18:23 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl get pods NAME READY STATUS RESTARTS AGE nagf-6899d476598983c219c3beb256c8445f-hr235 1/1 Running 0 5m ``` About half-way through the above, <https://tools.wmflabs.org/nagf/> Nginx started to respond with HTTP 500 Internal Server Error, and hasn't recovered since.

``` [17:25 UTC] krinkle at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ sudo -iu tools.nagf [18:06 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl get pods NAME READY STATUS RESTARTS AGE nagf-wu6jm 1/1 Running 0 4d [18:06 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl rolling-update nagf --image=yuvipanda/nagf Created nagf-6899d476598983c219c3beb256c8445f Scaling up nagf-6899d476598983c219c3beb256c8445f from 0 to 1, scaling down nagf from 1 to 0 (keep 1 pods available, don't exceed 2 pods) Scaling nagf-6899d476598983c219c3beb256c8445f up to 1 Scaling nagf down to 0 Update succeeded. Deleting old controller: nagf Renaming nagf-6899d476598983c219c3beb256c8445f to nagf replicationcontroller "nagf" rolling updated [18:23 UTC] tools.nagf at tools-k8s-bastion-01.tools.eqiad.wmflabs in ~ $ kubectl get pods NAME READY STATUS RESTARTS AGE nagf-6899d476598983c219c3beb256c8445f-hr235 1/1 Running 0 5m ``` About half-way through the above, <https://tools.wmflabs.org/nagf/> Nginx started to respond with HTTP 500 Internal Server Error, and hasn't recovered since.