During the previous year we 've gone into great lengths to design and implement a standardized application environment. During 2017-2018 Q3 we want to use this environment during our Mathoid deployment T184919. This task is about documenting successes, failures and lessons to be learned from that standardized application environment
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | akosiaris | T184462 Serve one production service via Kubernetes | |||
| Resolved | akosiaris | T184923 Validate whether the (implemented) standardized application environment works as expected |
Event Timeline
Comment Actions
Change 420731 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] prometheus: Add kubernetes pods jobs
Comment Actions
Change 420731 merged by Alexandros Kosiaris:
[operations/puppet@production] prometheus: Add kubernetes pods jobs
Comment Actions
Change 421262 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Annotate pods for prometheus statsd scraping
Comment Actions
Change 421263 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Switch mathoid to use the local statsd_prometheus
Comment Actions
Change 421262 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Annotate pods for prometheus statsd scraping
Comment Actions
Change 421263 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Switch mathoid to use the local statsd_prometheus
Comment Actions
The metrics part has been validated. In fact https://grafana.wikimedia.org/dashboard/db/service-mathoid?orgId=1 currently has graphs being generated using prometheus and showing the same data that is being generated directly from statsd
Comment Actions
Change 421484 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Give tiller the right to manage network policies
Comment Actions
Change 421485 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Annotate namespace with a default deny policy
Comment Actions
Change 421486 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] WIP Add network policy objects to the helm charts
Comment Actions
Change 421484 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Give tiller the right to manage network policies
Comment Actions
Change 421485 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Annotate namespace with a default deny policy
Comment Actions
Change 421486 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Add network policy objects to the helm charts
Comment Actions
- Network policy has been validated
- statsd_prometheus_exporter has been validated and prometheus is scraping each pod and collecting data
- logging approach has not been validated, turns out we need to upgrade components for this to work and reevaluate the sidecar approach. However logging works just fine for the mathoid service with each pod sending logs directly to logstash in the gelf format as well as logging to stdout, making the logs accessible to kubectl logs.
Overall this is considered successful, I am resolving the task