For the upcoming tools migration to the secondary storage cluster, we will have the tools NFS share in Read-only mode for the duration of the migration. Since tools can't write to nfs anyway, we discussed black-holing webservice logs to /dev/null so they wouldn't keep attempting to hit the nfs servers and fail. I talked about this with @yuvipanda yesterday and he suggested pushing a change to the tools webservice package, that will have a configurable switch to write all logs to /dev/null. During the migration we can then toggle this switch through puppet.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Route all logs to /dev/null | operations/software/tools-webservice | master | +20 -7 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Bstorm | T126083 overhaul labstore setup [tracking] | |||
| Resolved | • chasemp | T146154 Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] | |||
| Resolved | • chasemp | T149946 Add config option in tools webservice debian package to write logs to /dev/null |
Event Timeline
Comment Actions
Let's do this rather than make it configurable, since doing config
via puppet in kubernetes land is kinda not the easiest. So our
workflow will be:
GridEngine:
- Push new deb to aptly
- Make sure new deb is in all webgrid hosts
- Restart all webservices
Kubernetes
- Build new kubernetes images
- Pull on all worker nodes
- Restart all webservices
To revert, we'll prepare another patch and another deb in advance,
and just do the same as above with the newer deb.
Comment Actions
Change 319798 had a related patch set uploaded (by Yuvipanda):
Route all logs to /dev/null
Comment Actions
I've reverted and built package and pushed new images. we need to:
- Install package on all webgrid nodes
- Restart all webservices (both k8s and gridengine)
for this to be complete