kibana (ELK) host for beta testing
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Jun 5 2015, 6:49 PM

Description

Build a new VM based on a jessie base image for use in the beta cluster. This is needed to verify that everything works well for a jessie based install. It will also be a good opportunity to introduce a new security group that can be used to allow external projects (eg WDQ) to feed log information into logstash-beta.wmflabs.org.

TODO:

Create instance deployment-logstash2
Configure deployment-logstash2 as a Logstash, Kibana and Elasticsearch host
Join deployment-logstash2 and deployment-logstash1 Elasticsearch nodes as a cluster
Migrate Elasticsearch data from deployment-logstash1 to deployment-logstash2
migrate role::logstash::ircbot from deployment-logstash1 to deployment-logstash2
point https://logstash-beta.wmflabs.org at deployment-logstash2
change all beta cluster config to forward logs to deployment-logstash2 instead of deployment-logstash1
delete deployment-logstash1 instance
cleanup trebuchet redis cache to remove deployment-logstash1 as a target

Details

Subject	Repo	Branch	Lines +/-
beta: Replace deployment-logstash1 with deployment-logstash2	mediawiki/services/parsoid/deploy	master	+1 -1
beta: Replace deployment-logstash1 with deployment-logstash2	operations/puppet	production	+10 -10
beta: Replace deployment-logstash1 with deployment-logstash2	operations/mediawiki-config	master	+1 -1
logstash: jessie support and beta cluster cluster	operations/puppet	production	+23 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	bd808	T69817 Monitor for anomalies/spikes in read failures of memcached
Resolved	bd808	T100735 Have Logstash report per-channel log message rate to Graphite
Resolved	bd808	T99735 Upgrade Logstash to 1.5.3
Resolved	bd808	T97545 reinstall logstash1001-1003
Resolved	bd808	T96692 Rack and Setup (3) Logstash Servers
Resolved	RobH	T84958 eqiad: (3) servers for logstash service
Declined	bd808	T87078 Upgrade RAM for logstash100[123] to 64G
Resolved	RobH	T89402 purchase 3 additional logstash nodes
Declined	RobH	T87460 Allocate temporary Elasticsearch nodes from spares pool for Logstash
Resolved	faidon	T97481 jessie installs fail - mirror issue due to jessie release?
Resolved	bd808	T97645 Elasticsearch not starting on Jessie hosts
Resolved	bd808	T101541 Build jessie based elasticsearch/logstash/kibana (ELK) host for beta testing
Resolved	MoritzMuehlenhoff	T98042 Update Wikimedia apt repo to include debs for Elasticsearch & Logstash on jessie
Resolved	bd808	T105101 Upgrade Logstash Elasticsearch cluster to 1.6.0
Resolved	fgiunchedi	T104035 logstash partman recipe huge root partition
Resolved	bd808	T107083 Java class org.bouncycastle.jcajce.provider.digest.MD5$Digest not found for Logstash on logstash1001 (jessie)

Event Timeline

bd808 created this task.Jun 5 2015, 6:49 PM

bd808 claimed this task.

bd808 raised the priority of this task from to Needs Triage.

bd808 updated the task description. (Show Details)

bd808 added projects: User-bd808, Beta-Cluster-Infrastructure, Wikimedia-Logstash.

bd808 subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2015, 6:49 PM

Smalyshev subscribed.Jun 5 2015, 6:51 PM

bd808 added a parent task: T97545: reinstall logstash1001-1003.Jun 5 2015, 6:52 PM

bd808 moved this task from Backlog to In Dev/Progress on the Wikimedia-Logstash board.

bd808 triaged this task as High priority.Jun 5 2015, 9:39 PM

bd808 added a parent task: T99735: Upgrade Logstash to 1.5.3.

Change 216337 had a related patch set uploaded (by BryanDavis):
[WIP] logstash: jessie support and beta cluster cluster

https://gerrit.wikimedia.org/r/216337

gerritbot added a project: Patch-For-Review.Jun 5 2015, 11:05 PM

Built host deployment-logstash2.deployment-prep.eqiad.wmflabs based on the labs jessie image.

The new host is attached to the logstash security group which allows syslog, gelf and irc inputs from any hosts inside or outside of the deployment-prep project. This will allow the

This config is enabled via wikitech:

role::kibana
role::logstash
role::logstash::apifeatureusage
role::logstash::puppetreports
role::labs::lvm::mnt
$::lvm_mount_point = /var/lib/elasticsearch

Forced all Elasticsearch shard allocations to the host via:

curl -XPUT localhost:9200/_cluster/settings -d '{
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "exclude": {
            "_host": "deployment-logstash1"
          }
        }
      }
    }
  }
}'

bd808 updated the task description. (Show Details)Jun 6 2015, 12:21 AM

bd808 set Security to None.

hashar moved this task from To Triage to In-progress on the Beta-Cluster-Infrastructure board.Jun 15 2015, 7:30 PM

bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Jun 18 2015, 8:45 PM

Change 216337 merged by Ori.livneh:
logstash: jessie support and beta cluster cluster

https://gerrit.wikimedia.org/r/216337

ori mentioned this in rOPUP36175d73f409: logstash: jessie support and beta cluster cluster.Jun 25 2015, 11:34 PM

bd808 updated the task description. (Show Details)Jun 29 2015, 11:25 PM

bd808 updated the task description. (Show Details)Jul 6 2015, 9:23 PM

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 6 2015, 9:23 PM

Change 223184 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223184

Change 223185 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223185

Change 223185 merged by Alex Monk:
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223185

Krenair mentioned this in rOMWCd1609c13b7a7: beta: Replace deployment-logstash1 with deployment-logstash2.Jul 6 2015, 11:16 PM

Change 223184 merged by Yuvipanda:
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223184

yuvipanda mentioned this in rOPUP6db1a6bbc2f5: beta: Replace deployment-logstash1 with deployment-logstash2.Jul 7 2015, 7:40 PM

bd808 updated the task description. (Show Details)Jul 7 2015, 7:42 PM

Change 223443 had a related patch set uploaded (by BryanDavis):
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223443

Change 223443 merged by jenkins-bot:
beta: Replace deployment-logstash1 with deployment-logstash2

https://gerrit.wikimedia.org/r/223443

bd808 mentioned this in rGPAD6a31ee0fcf23: beta: Replace deployment-logstash1 with deployment-logstash2.Jul 7 2015, 9:12 PM

bd808 updated the task description. (Show Details)Jul 7 2015, 9:25 PM

bd808 closed this task as Resolved.Jul 7 2015, 9:49 PM

bd808 updated the task description. (Show Details)

bd808 moved this task from In Dev/Progress to Done on the User-bd808 board.Jul 11 2015, 9:40 PM

bd808 moved this task from Done to Archive on the User-bd808 board.Jul 24 2015, 5:18 PM

bd808 moved this task from In Dev/Progress to Archive on the Wikimedia-Logstash board.Jul 28 2015, 4:49 PM

Luke081515 moved this task from In-progress to Done on the Beta-Cluster-Infrastructure board.Sep 7 2015, 3:12 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:29 PM

Build jessie based elasticsearch/logstash/kibana (ELK) host for beta testingClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Build jessie based elasticsearch/logstash/kibana (ELK) host for beta testing
Closed, ResolvedPublic
Actions

Related Objects
Search...