Page MenuHomePhabricator

Add config option in tools webservice debian package to write logs to /dev/null
Closed, ResolvedPublic

Description

For the upcoming tools migration to the secondary storage cluster, we will have the tools NFS share in Read-only mode for the duration of the migration. Since tools can't write to nfs anyway, we discussed black-holing webservice logs to /dev/null so they wouldn't keep attempting to hit the nfs servers and fail. I talked about this with @yuvipanda yesterday and he suggested pushing a change to the tools webservice package, that will have a configurable switch to write all logs to /dev/null. During the migration we can then toggle this switch through puppet.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2016, 8:09 PM

Let's do this rather than make it configurable, since doing config
via puppet in kubernetes land is kinda not the easiest. So our
workflow will be:

GridEngine:

  • Push new deb to aptly
  • Make sure new deb is in all webgrid hosts
  • Restart all webservices

Kubernetes

  • Build new kubernetes images
  • Pull on all worker nodes
  • Restart all webservices

To revert, we'll prepare another patch and another deb in advance,
and just do the same as above with the newer deb.

Change 319798 had a related patch set uploaded (by Yuvipanda):
Route all logs to /dev/null

https://gerrit.wikimedia.org/r/319798

Change 319798 merged by Yuvipanda:
Route all logs to /dev/null

https://gerrit.wikimedia.org/r/319798

I've reverted and built package and pushed new images. we need to:

  1. Install package on all webgrid nodes
  2. Restart all webservices (both k8s and gridengine)

for this to be complete

chasemp closed this task as Resolved.Nov 22 2016, 3:06 PM
chasemp claimed this task.

We reverted everything yesterday afternoon and so far no incident.