Page MenuHomePhabricator
Create Task
Maniphest T213898

Replace and expand Elasticsearch/Kafka storage in eqiad and upgrade the cluster from Debian jessie to stretch
Closed, ResolvedPublic
Actions

Description

Proposed plan to complete this task:

  • Put new storage systems in production with stretch
    • Install new systems with stretch (shipment and racking pending, T210498)
    • Add hosts to elasticsearch cluster
    • Add hosts to kafka cluster
  • Take old storage hosts out of service for elasticsearch
    • Migrate all ES indices onto new hosts
    • Retire ES on old hosts
  • Take old storage hosts out of service for kafka
    • migrate logstash1004 to logstash1010
    • migrate logstash1005 to logstash1011
    • migrate logstash1006 to logstash1012
    • All producers/consumers pointed to new kafka brokers
  • Replace logstash jessie VMs (logstash100[789]) with stretch VMs

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
logstash: shrink es cluster back to 3 nodes, remove retired hostsoperations/puppetproduction+2 -14
rsyslog: replace logstash1006 with logstash1012 in kafka_shipperoperations/puppetproduction+1 -1
logstash: disable notifications on logstash1006 and logstash1012operations/puppetproduction+2 -0
kafka-logging: replace logstash1006 with logstash1012operations/puppetproduction+5 -9
rsyslog: replace logstash1005 with logstash1011 in kafka_shipperoperations/puppetproduction+1 -1
logstash: disable notifications on logstash1005 and logstash1011operations/puppetproduction+2 -0
kafka-logging: replace logstash1005 with logstash1011operations/puppetproduction+6 -6
rsyslog: replace logstash1004 with logstash1010 in kafka_shipperoperations/puppetproduction+1 -1
logstash101[12]: add ipv6 mapped addressoperations/puppetproduction+1 -0
logstash1010: add ipv6 mapped addressoperations/puppetproduction+1 -0
kafka-logging: replace logstash1004 with logstash1010operations/puppetproduction+12 -3
logstash: add kafka logging role to logstash101[012]operations/puppetproduction+10 -0
logstash: remove elasticsearch role from logstash100[456]operations/puppetproduction+41 -3
logstash: add logstash101[012] to unicast hostsoperations/puppetproduction+6 -0
Revert "scap: use logstash1008 for logstash_host"operations/puppetproduction+1 -1
logstash: force use elasticsearch-curator 5operations/puppetproduction+4 -1
logstash: move role::ls::eventlogging to profile::logstash::collectoroperations/puppetproduction+18 -31
scap: use logstash1008 for logstash_hostoperations/puppetproduction+1 -1
install_server: use default distribution for logstash100[789]operations/puppetproduction+0 -6
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jan 16 2019, 11:12 AM
fgiunchedi triaged this task as Medium priority.
fgiunchedi mentioned this in T203169: Logstash hardware expansion.Jan 16 2019, 11:33 AM
herron moved this task from Backlog to Working on on the User-herron board.Jan 22 2019, 9:24 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 11:05 PM
gerritbot subscribed.Feb 13 2019, 8:35 PM

Change 490401 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: move role::ls::eventlogging to profile::logstash::collector

https://gerrit.wikimedia.org/r/490401

gerritbot added a project: Patch-For-Review.Feb 13 2019, 8:35 PM
gerritbot added a comment.Feb 14 2019, 1:40 PM

Change 490601 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: use default distribution for logstash100[789]

https://gerrit.wikimedia.org/r/490601

gerritbot added a comment.Feb 14 2019, 1:40 PM

Change 490602 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] scap: use logstash service name for logstash_host

https://gerrit.wikimedia.org/r/490602

gerritbot added a comment.Feb 14 2019, 1:43 PM

Change 490601 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: use default distribution for logstash100[789]

https://gerrit.wikimedia.org/r/490601

gerritbot added a comment.Feb 14 2019, 2:20 PM

Change 490602 merged by Filippo Giunchedi:
[operations/puppet@production] scap: use logstash1008 for logstash_host

https://gerrit.wikimedia.org/r/490602

Stashbot subscribed.Feb 14 2019, 2:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-14T14:45:34Z] <godog> depool and stop logstash1009 for stretch reimage - T213898

gerritbot added a comment.Feb 14 2019, 6:11 PM

Change 490401 merged by Herron:
[operations/puppet@production] logstash: move role::ls::eventlogging to profile::logstash::collector

https://gerrit.wikimedia.org/r/490401

fgiunchedi added a comment.Feb 15 2019, 8:05 AM

On stretch by default we're installing elasticsearch-curator from stretch which is at version 4.2, instead the package needs to be at version >= 5 and thus come from stretch-wikimedia.

gerritbot added a comment.Feb 15 2019, 8:30 AM

Change 490809 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: force use elasticsearch-curator 5

https://gerrit.wikimedia.org/r/490809

gerritbot added a comment.Feb 20 2019, 3:46 PM

Change 490809 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: force use elasticsearch-curator 5

https://gerrit.wikimedia.org/r/490809

gerritbot added a comment.Feb 20 2019, 4:26 PM

Change 491794 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Revert "scap: use logstash1008 for logstash_host"

https://gerrit.wikimedia.org/r/491794

gerritbot added a comment.Feb 20 2019, 4:28 PM

Change 491794 merged by Filippo Giunchedi:
[operations/puppet@production] Revert "scap: use logstash1008 for logstash_host"

https://gerrit.wikimedia.org/r/491794

Stashbot added a comment.Feb 20 2019, 4:36 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-20T16:36:10Z] <godog> depool and reimage logstash1008 with stretch - T213898

Stashbot added a comment.Feb 21 2019, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T13:57:21Z] <godog> depool and reimage logstash1007 - T213898

fgiunchedi updated the task description. (Show Details)Feb 21 2019, 2:38 PM
Paladox subscribed.Feb 21 2019, 2:39 PM
herron updated the task description. (Show Details)Feb 25 2019, 3:43 PM
gerritbot added a comment.Feb 25 2019, 3:44 PM

Change 492695 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: remove elasticsearch role from logstash100[456]

https://gerrit.wikimedia.org/r/492695

herron mentioned this in T214608: rack/setup/install logstash101[012].eqiad.wmnet.Feb 25 2019, 3:45 PM
Stashbot added a comment.Feb 25 2019, 9:11 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-25T21:11:03Z] <herron> turning down elasticsearch service on logstash100[456] (data has been migrated to logstash101[012]) T213898

gerritbot added a comment.Feb 25 2019, 9:23 PM

Change 492769 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add logstash101[012] to unicast hosts

https://gerrit.wikimedia.org/r/492769

gerritbot added a comment.Feb 25 2019, 9:31 PM

Change 492769 merged by Herron:
[operations/puppet@production] logstash: add logstash101[012] to unicast hosts

https://gerrit.wikimedia.org/r/492769

Stashbot added a comment.Feb 25 2019, 11:15 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-25T23:15:15Z] <herron> service restarts to make logstash101[012] master eligible are taking longer than expected, leaving elasticsearch on logstash100[456] enabled overnight T213898

Stashbot added a comment.Feb 26 2019, 4:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-26T16:09:46Z] <herron> elasticsearch stopped on logstash100[456] T213898

gerritbot added a comment.Feb 26 2019, 4:12 PM

Change 492695 merged by Herron:
[operations/puppet@production] logstash: remove elasticsearch role from logstash100[456]

https://gerrit.wikimedia.org/r/492695

gerritbot added a comment.Feb 26 2019, 7:29 PM

Change 493098 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: shrink es cluster back to 3 nodes, remove retired hosts

https://gerrit.wikimedia.org/r/493098

gerritbot added a comment.Feb 26 2019, 7:50 PM

Change 493102 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add kafka logging role to logstash101[012]

https://gerrit.wikimedia.org/r/493102

herron updated the task description. (Show Details)Feb 27 2019, 3:23 PM
gerritbot added a comment.Feb 27 2019, 6:45 PM

Change 493102 abandoned by Herron:
logstash: add kafka logging role to logstash101[012]

Reason:
in favor of replacing hosts in rolling fashion

https://gerrit.wikimedia.org/r/493102

gerritbot added a comment.Feb 27 2019, 6:54 PM

Change 493290 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka-logging: replace logstash1004 with logstash1010

https://gerrit.wikimedia.org/r/493290

Stashbot added a comment.Feb 27 2019, 7:26 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-27T19:26:51Z] <herron> replacing kafka on logstash1004 with logstash1010 T213898

gerritbot added a comment.Feb 27 2019, 7:46 PM

Change 493290 merged by Herron:
[operations/puppet@production] kafka-logging: replace logstash1004 with logstash1010

https://gerrit.wikimedia.org/r/493290

gerritbot added a comment.Feb 27 2019, 8:01 PM

Change 493303 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash1010: add ipv6 mapped address

https://gerrit.wikimedia.org/r/493303

gerritbot added a comment.Feb 27 2019, 8:03 PM

Change 493303 merged by Herron:
[operations/puppet@production] logstash1010: add ipv6 mapped address

https://gerrit.wikimedia.org/r/493303

gerritbot added a comment.Feb 27 2019, 8:07 PM

Change 493306 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash100[12]: add ipv6 mapped address

https://gerrit.wikimedia.org/r/493306

gerritbot added a comment.Feb 27 2019, 8:10 PM

Change 493306 merged by Herron:
[operations/puppet@production] logstash101[12]: add ipv6 mapped address

https://gerrit.wikimedia.org/r/493306

herron added a comment.Feb 27 2019, 10:47 PM

kafka service from logstash1004 has been migrated to logstash1010, and logstash1004 is now transitioned to spare::system.

herron updated the task description. (Show Details)Feb 27 2019, 10:49 PM
herron updated the task description. (Show Details)
gerritbot added a comment.Feb 28 2019, 3:35 PM

Change 493429 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka-logging: replace logstash1005 with logstash1011

https://gerrit.wikimedia.org/r/493429

gerritbot added a comment.Feb 28 2019, 3:37 PM

Change 493430 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: disable notifications on logstash1005 and logstash1011

https://gerrit.wikimedia.org/r/493430

gerritbot added a comment.Feb 28 2019, 3:47 PM

Change 493430 merged by Herron:
[operations/puppet@production] logstash: disable notifications on logstash1005 and logstash1011

https://gerrit.wikimedia.org/r/493430

gerritbot added a comment.Feb 28 2019, 4:06 PM

Change 493440 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: replace logstash1004 with logstash1010 in kafka_shipper

https://gerrit.wikimedia.org/r/493440

gerritbot added a comment.Feb 28 2019, 4:10 PM

Change 493440 merged by Herron:
[operations/puppet@production] rsyslog: replace logstash1004 with logstash1010 in kafka_shipper

https://gerrit.wikimedia.org/r/493440

Stashbot added a comment.Feb 28 2019, 4:27 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-28T16:27:56Z] <herron> migrating kafka on logstash1005 to logstash1011 T213898

Stashbot added a comment.Feb 28 2019, 4:28 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-28T16:28:00Z] <herron> migrating kafka on logstash1005 to logstash1011 T213898

gerritbot added a comment.Feb 28 2019, 4:30 PM

Change 493429 merged by Herron:
[operations/puppet@production] kafka-logging: replace logstash1005 with logstash1011

https://gerrit.wikimedia.org/r/493429

Stashbot added a comment.Feb 28 2019, 5:51 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-28T17:51:46Z] <herron> logstash1011 kafka now in sync. transitioning logstash1005 to spare system T213898

herron renamed this task from Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch to Replace and expand Elasticsearch/Kafka storage in eqiad and upgrade the cluster from Debian jessie to stretch.Feb 28 2019, 5:52 PM
herron updated the task description. (Show Details)
gerritbot added a comment.Feb 28 2019, 6:10 PM

Change 493471 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka-logging: replace logstash1006 with logstash1012

https://gerrit.wikimedia.org/r/493471

gerritbot added a comment.Feb 28 2019, 6:26 PM

Change 493476 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: disable notifications on logstash1006 and logstash1012

https://gerrit.wikimedia.org/r/493476

gerritbot added a comment.Feb 28 2019, 6:27 PM

Change 493476 merged by Herron:
[operations/puppet@production] logstash: disable notifications on logstash1006 and logstash1012

https://gerrit.wikimedia.org/r/493476

gerritbot added a comment.Feb 28 2019, 6:33 PM

Change 493477 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: replace logstash1005 with logstash1011 in kafka_shipper

https://gerrit.wikimedia.org/r/493477

gerritbot added a comment.Feb 28 2019, 6:39 PM

Change 493477 merged by Herron:
[operations/puppet@production] rsyslog: replace logstash1005 with logstash1011 in kafka_shipper

https://gerrit.wikimedia.org/r/493477

Stashbot added a comment.Feb 28 2019, 6:52 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-28T18:52:06Z] <herron> migrating logstash1006 kafka to logstash1012 T213898

gerritbot added a comment.Feb 28 2019, 6:55 PM

Change 493471 merged by Herron:
[operations/puppet@production] kafka-logging: replace logstash1006 with logstash1012

https://gerrit.wikimedia.org/r/493471

herron updated the task description. (Show Details)Mar 1 2019, 10:37 PM
herron updated the task description. (Show Details)
gerritbot added a comment.Mar 4 2019, 2:24 PM

Change 494224 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: replace logstash1006 with logstash1012 in kafka_shipper

https://gerrit.wikimedia.org/r/494224

gerritbot added a comment.Mar 4 2019, 2:51 PM

Change 494224 merged by Herron:
[operations/puppet@production] rsyslog: replace logstash1006 with logstash1012 in kafka_shipper

https://gerrit.wikimedia.org/r/494224

herron updated the task description. (Show Details)Mar 4 2019, 2:52 PM
herron closed this task as Resolved.Mar 4 2019, 2:59 PM
herron claimed this task.

Service migration and OS upgrade work is complete with ES and Kafka services running from logstash101[012], and frontend VMs logstash100[789] upgraded to stretch.

Tracking hardware retirement in task T217556

fgiunchedi added a project: observability.Aug 19 2019, 2:30 PM
fgiunchedi mentioned this in T250133: production-logstash elastic cluster is yellow state.Apr 16 2020, 9:11 AM
Papaul closed subtask T217556: Decommission old eqiad logstash hardware hosts logstash100[456] as Resolved.Apr 21 2020, 6:32 PM
gerritbot added a comment.Apr 2 2024, 1:58 PM

Change #493098 abandoned by Herron:

[operations/puppet@production] logstash: shrink es cluster back to 3 nodes, remove retired hosts

Reason:

spring cleaning -- stale patch

https://gerrit.wikimedia.org/r/493098

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2024, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits