This morning I noticed in icinga the following alarm:
ocg1003;OCG health;WARNING;HARD;3;WARNING: connection error: HTTPConnectionPool(host='localhost', port=8000): Read timed out. (read timeout=5)
which results in a warning if the read times out or any connection error happens.
This should not only be critical, but also we should get paged when any ocg server is unreachable until T120077 is solved, given right now any malfunction of a single ocg server results in user-noticeable downtime.
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/puppet | production | +11 -5 | ocg: send out an alarm when ocg doesn't respond to health checks |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T130591 Increase size of root partition on ocg* servers | |||
| Declined | None | T84723 reinstall OCG servers | |||
| Resolved | Dzahn | T90839 pybal issue? | |||
| Resolved | cscott | T120077 Implement flag to tell an OCG machine not to take new tasks from the redis task queue | |||
| Resolved | Joe | T120078 OCG checks should be CRITICAL when reading from the server times out | |||
| Resolved | cscott | T120079 The OCG cleanup cache script doesn't work properly | |||
| Declined | None | T135034 make ocg role work on labs instances (install deployment-pdf instance with jessie) |
Change 256412 had a related patch set uploaded (by Giuseppe Lavagetto):
ocg: send out an alarm when ocg doesn't respond to health checks
Change 256412 merged by Giuseppe Lavagetto:
ocg: send out an alarm when ocg doesn't respond to health checks