Page MenuHomePhabricator

Build jessie based elasticsearch/logstash/kibana (ELK) host for beta testing
Closed, ResolvedPublic

Description

Build a new VM based on a jessie base image for use in the beta cluster. This is needed to verify that everything works well for a jessie based install. It will also be a good opportunity to introduce a new security group that can be used to allow external projects (eg WDQ) to feed log information into logstash-beta.wmflabs.org.

TODO:

  • Create instance deployment-logstash2
  • Configure deployment-logstash2 as a Logstash, Kibana and Elasticsearch host
  • Join deployment-logstash2 and deployment-logstash1 Elasticsearch nodes as a cluster
  • Migrate Elasticsearch data from deployment-logstash1 to deployment-logstash2
  • migrate role::logstash::ircbot from deployment-logstash1 to deployment-logstash2
  • point https://logstash-beta.wmflabs.org at deployment-logstash2
  • change all beta cluster config to forward logs to deployment-logstash2 instead of deployment-logstash1
  • delete deployment-logstash1 instance
  • cleanup trebuchet redis cache to remove deployment-logstash1 as a target

Related Objects

Event Timeline

bd808 created this task.Jun 5 2015, 6:49 PM
bd808 claimed this task.
bd808 raised the priority of this task from to Needs Triage.
bd808 updated the task description. (Show Details)
bd808 added a subscriber: bd808.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2015, 6:49 PM
bd808 triaged this task as High priority.Jun 5 2015, 9:39 PM

Change 216337 had a related patch set uploaded (by BryanDavis):
[WIP] logstash: jessie support and beta cluster cluster

https://gerrit.wikimedia.org/r/216337

bd808 added a comment.Jun 6 2015, 12:18 AM

Built host deployment-logstash2.deployment-prep.eqiad.wmflabs based on the labs jessie image.

The new host is attached to the logstash security group which allows syslog, gelf and irc inputs from any hosts inside or outside of the deployment-prep project. This will allow the

This config is enabled via wikitech:

  • role::kibana
  • role::logstash
  • role::logstash::apifeatureusage
  • role::logstash::puppetreports
  • role::labs::lvm::mnt
  • $::lvm_mount_point = /var/lib/elasticsearch

Forced all Elasticsearch shard allocations to the host via:

curl -XPUT localhost:9200/_cluster/settings -d '{
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "exclude": {
            "_host": "deployment-logstash1"
          }
        }
      }
    }
  }
}'
bd808 updated the task description. (Show Details)Jun 6 2015, 12:21 AM
bd808 set Security to None.
bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Jun 18 2015, 8:45 PM

Change 216337 merged by Ori.livneh:
logstash: jessie support and beta cluster cluster

https://gerrit.wikimedia.org/r/216337

bd808 updated the task description. (Show Details)Jun 29 2015, 11:25 PM
bd808 updated the task description. (Show Details)Jul 6 2015, 9:23 PM
Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 6 2015, 9:23 PM

Change 223184 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223184

Change 223185 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223185

Change 223185 merged by Alex Monk:
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223185

Change 223184 merged by Yuvipanda:
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223184

bd808 updated the task description. (Show Details)Jul 7 2015, 7:42 PM

Change 223443 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223443

Change 223443 merged by jenkins-bot:
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223443

bd808 updated the task description. (Show Details)Jul 7 2015, 9:25 PM
bd808 closed this task as Resolved.Jul 7 2015, 9:49 PM
bd808 updated the task description. (Show Details)
bd808 moved this task from In Dev/Progress to Done on the User-bd808 board.Jul 11 2015, 9:40 PM
bd808 moved this task from Done to Archive on the User-bd808 board.Jul 24 2015, 5:18 PM