⚓ T99735 Upgrade Logstash to 1.5.3

Subject	Repo	Branch	Lines +/-
Update configuration for logstash 1.5.3	operations/puppet	production	+156 -146
Add logstash-filter-prune 0.1.5	operations/software/logstash/plugins	master	+180 -5
Initialise repo	operations/software/logstash/plugins	master	+10 -0

Status	Assigned	Task
Resolved	bd808	T69817 Monitor for anomalies/spikes in read failures of memcached
Resolved	bd808	T100735 Have Logstash report per-channel log message rate to Graphite
Resolved	bd808	T99735 Upgrade Logstash to 1.5.3
Resolved	bd808	T97545 reinstall logstash1001-1003
Resolved	bd808	T96692 Rack and Setup (3) Logstash Servers
Resolved	RobH	T84958 eqiad: (3) servers for logstash service
Declined	bd808	T87078 Upgrade RAM for logstash100[123] to 64G
Resolved	RobH	T89402 purchase 3 additional logstash nodes
Declined	RobH	T87460 Allocate temporary Elasticsearch nodes from spares pool for Logstash
Resolved	faidon	T97481 jessie installs fail - mirror issue due to jessie release?
Resolved	bd808	T97645 Elasticsearch not starting on Jessie hosts
Resolved	bd808	T101541 Build jessie based elasticsearch/logstash/kibana (ELK) host for beta testing
Resolved	MoritzMuehlenhoff	T98042 Update Wikimedia apt repo to include debs for Elasticsearch & Logstash on jessie
Resolved	bd808	T105101 Upgrade Logstash Elasticsearch cluster to 1.6.0
Resolved	fgiunchedi	T104035 logstash partman recipe huge root partition
Resolved	bd808	T107083 Java class org.bouncycastle.jcajce.provider.digest.MD5$Digest not found for Logstash on logstash1001 (jessie)
Declined	bd808	T107121 Setup rsyncable git fat store to host Logstash plugins
Resolved	MoritzMuehlenhoff	T107916 Import logstash 1.5.3 into apt.wm.o

bd808 created this task.May 19 2015, 9:54 PM

bd808 raised the priority of this task from to Medium.

bd808 updated the task description. (Show Details)

bd808 added projects: Wikimedia-Logstash, User-bd808.

bd808 subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2015, 9:54 PM

The upstream deb is at http://packages.elasticsearch.org/logstash/1.5/debian/pool/main/l/logstash/logstash_1.5.0-1_all.deb

bd808 added a subtask: T101541: Build jessie based elasticsearch/logstash/kibana (ELK) host for beta testing.Jun 5 2015, 9:40 PM

bd808 added a parent task: T100735: Have Logstash report per-channel log message rate to Graphite.

bd808 closed subtask T101541: Build jessie based elasticsearch/logstash/kibana (ELK) host for beta testing as Resolved.Jul 7 2015, 9:50 PM

bd808 claimed this task.Jul 7 2015, 11:24 PM

bd808 added a subtask: T97545: reinstall logstash1001-1003.Jul 9 2015, 3:33 PM

Change 226991 had a related patch set uploaded (by BryanDavis):
[WIP] Update configuration for logstash 1.5.3

https://gerrit.wikimedia.org/r/226991

gerritbot added a project: Patch-For-Review.Jul 25 2015, 7:50 PM

Change 227375 had a related patch set uploaded (by BryanDavis):
Initialise repo

https://gerrit.wikimedia.org/r/227375

Change 227375 merged by BryanDavis:
Initialise repo

https://gerrit.wikimedia.org/r/227375

bd808 moved this task from Backlog to In Dev/Progress on the Wikimedia-Logstash board.Jul 28 2015, 4:50 PM

bd808 closed subtask T97545: reinstall logstash1001-1003 as Resolved.Jul 28 2015, 4:52 PM

Change 229073 had a related patch set uploaded (by BryanDavis):
Add logstash-filter-prune 0.1.5

https://gerrit.wikimedia.org/r/229073

bd808 closed subtask T107121: Setup rsyncable git fat store to host Logstash plugins as Declined.Aug 4 2015, 4:48 AM

Change 229073 merged by BryanDavis:
Add logstash-filter-prune 0.1.5

https://gerrit.wikimedia.org/r/229073

bd808 renamed this task from Upgrade Logstash to 1.5.0 to Upgrade Logstash to 1.5.3.Aug 4 2015, 6:23 PM

bd808 updated the task description. (Show Details)

bd808 set Security to None.

These code changes are now deployed in the beta cluster. Assuming we don't see anything horrible happen there in the new 2-3 days this will be ready to go to production next week.

Pinged SRE about support needs during production deploy window. Tentatively planning to deploy on 2015-08-11. (edited to correct off-by-one error in month)

MoritzMuehlenhoff closed subtask T107916: Import logstash 1.5.3 into apt.wm.o as Resolved.Aug 5 2015, 7:13 AM

bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Aug 5 2015, 4:02 PM

Deployment plan:

Disable puppet on logstash100[1-3]:
1. sudo puppet agent --disable "bd808 upgrading logstash to 1.5.3"
Merge needed ops/puppet patches (needs SRE help):
Force puppet run on tin to setup new logstash/plugins Trebuchet repo (needs SRE help)
Tag initial logstash/plugins state
1. cd /srv/deployment/logstash/plugins
2. git deploy start
3. git deploy sync
Update each of logstash100[1-3] sequentially:
1. sudo service logstash stop
2. sudo apt-get remove logstash-contrib
3. sudo apt-get install logstash (1.5.3 is in apt.wm.o now)
4. sudo puppet agent --enable
5. sudo puppet agent --test --verbose
6. Verify:
  1. /etc/logstash/* config updated
  2. /srv/deployment/logstash/plugins present
  3. logstash started
  4. no errors in /var/log/logstash.{err,log}