Page MenuHomePhabricator
Create Task
Maniphest T130393

Create separate Kibana dashboards for production Cassandra clusters
Closed, ResolvedPublic
Actions

Description

It makes sense to separate the dashboards on a per-project/cluster basis.

Details

SubjectRepoBranchLines +/-
make logstash messages separable by clusteroperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Eevans created this task.Mar 18 2016, 6:38 PM
Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptMar 18 2016, 6:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot subscribed.Mar 18 2016, 6:44 PM

Change 278330 had a related patch set uploaded (by Eevans):
make logstash messages separable by cluster

https://gerrit.wikimedia.org/r/278330

gerritbot added a project: Patch-For-Review.Mar 18 2016, 6:44 PM
Eevans added a comment.Mar 18 2016, 7:14 PM

I've queued up https://gerrit.wikimedia.org/r/278330 for next Tuesdays Puppet SWAT (16:00 UTC).

NOTE: When merged this will go out to all clusters. Cassandra should automagically reload the logback config when the file is changed, but a rolling restart of Cassandra is still recommended.
gerritbot added a comment.Mar 22 2016, 4:10 PM

Change 278330 merged by Muehlenhoff:
make logstash messages separable by cluster

https://gerrit.wikimedia.org/r/278330

Stashbot subscribed.Mar 22 2016, 4:21 PM

Mentioned in SAL [2016-03-22T16:21:05Z] <urandom> Restarting Cassandra on restbase1007.eqiad.wmnet (canary) : T130393, T128787

Stashbot mentioned this in T128787: Reduce Cassandra logstash output.Mar 22 2016, 4:21 PM
Stashbot added a comment.Mar 22 2016, 4:54 PM

Mentioned in SAL [2016-03-22T16:54:55Z] <urandom> Performing rolling restart of AQS Cassandra cluster : T130393, T128787

Stashbot added a comment.Mar 22 2016, 5:03 PM

Mentioned in SAL [2016-03-22T17:03:18Z] <urandom> Rolling restart of AQS Cassandra cluster complete : : T130393, T128787

Stashbot added a comment.Mar 22 2016, 6:39 PM

Mentioned in SAL [2016-03-22T18:39:39Z] <urandom> Rolling restart of Cassandra on maps cluster : T130393, T128787

Stashbot added a comment.Mar 22 2016, 6:41 PM

Mentioned in SAL [2016-03-22T18:41:11Z] <urandom> Aborting restart of Cassandra on maps cluster : T130393, T128787

Stashbot added a comment.Mar 22 2016, 7:53 PM

Mentioned in SAL [2016-03-22T19:53:30Z] <urandom> Rolling restart of RESTBase Cassandra cluster : T130393, T128787

Stashbot added a comment.Mar 22 2016, 8:45 PM

Mentioned in SAL [2016-03-22T20:45:38Z] <urandom> Rolling restart of RESTBase Cassandra cluster complete : T130393, T128787

Eevans closed this task as Resolved.Mar 23 2016, 3:13 PM

Separate dashboards for Maps and AQS have been created using the new cluster log attribute:

https://logstash.wikimedia.org/#/dashboard/elasticsearch/analytics-cassandra
https://logstash.wikimedia.org/#/dashboard/elasticsearch/maps-cassandra

Eevans triaged this task as Medium priority.Mar 23 2016, 3:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits