⚓ T220103 TEC6: Logging infrastructure (Q4 2018/19 goal)

Subject	Repo	Branch	Lines +/-
logstash: set retention to 90 days	operations/puppet	production	+4 -4
logstash: add deprecated-input tag to syslog input	operations/puppet	production	+3 -1
logstash: add deprecated-input tag to deprecated inputs	operations/puppet	production	+15 -122
logstash: ramp up logs retention	operations/puppet	production	+4 -4

Status	Assigned	Task
Resolved	fgiunchedi	T220103 TEC6: Logging infrastructure (Q4 2018/19 goal)
Open	colewhite	T213902 Implement sensitive logstash access control
Open	None	T193766 Ship host syslogs to ELK
Resolved	None	T193664 Knock down puppet 4 deprecation warnings
Open	None	T197173 Ship MX logs to ELK
Open	None	T292682 Develop tooling for quickly parsing 5xx and sampled-1000 logs
Open	None	T288619 Improve the process to consume and use API.LOG to filter out bad performing queries either by extra tooling within o11y or analytics
Open	None	T301110 Ingest webrequest sampled 1000 into logstash
Open	None	T309186 create a sampled log of POST data
Resolved	colewhite	T313868 OpenSearch Security plugin packaging
Resolved	colewhite	T333732 Upgrade OpenSearch using upstream deb packages
Open	None	T350516 Enable OpenSearch security plugin - Beta Logs

fgiunchedi created this task.Apr 4 2019, 1:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2019, 1:03 PM

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Apr 4 2019, 1:07 PM

fgiunchedi renamed this task from TEC6: logging infrastructure (Q4 2018/19 goal) to TEC6: Logging infrastructure (Q4 2018/19 goal).Apr 5 2019, 8:27 AM

fgiunchedi triaged this task as Medium priority.Apr 9 2019, 8:37 AM

colewhite subscribed.Apr 12 2019, 8:01 PM

fgiunchedi edited projects, added Wikimedia-Logstash; removed observability.Apr 15 2019, 2:54 PM

fgiunchedi moved this task from Backlog to In Dev/Progress on the Wikimedia-Logstash board.

Change 505741 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: ramp up logs retention

https://gerrit.wikimedia.org/r/505741

gerritbot added a project: Patch-For-Review.Apr 23 2019, 10:31 AM

Change 505741 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: ramp up logs retention

https://gerrit.wikimedia.org/r/505741

herron updated the task description. (Show Details)May 21 2019, 8:37 PM

herron added a subtask: T213902: Implement sensitive logstash access control.

Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:33 PM

Change 512193 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: add deprecated-input tag to deprecated inputs

https://gerrit.wikimedia.org/r/512193

gerritbot added a project: Patch-For-Review.May 23 2019, 5:12 PM

Change 512193 merged by Cwhite:
[operations/puppet@production] logstash: add deprecated-input tag to deprecated inputs

https://gerrit.wikimedia.org/r/512193

Maintenance_bot removed a project: Patch-For-Review.May 28 2019, 3:10 PM

Change 513612 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: add deprecated-input tag to syslog input

https://gerrit.wikimedia.org/r/513612

gerritbot added a project: Patch-For-Review.May 31 2019, 2:26 PM

Change 513612 merged by Cwhite:
[operations/puppet@production] logstash: add deprecated-input tag to syslog input

https://gerrit.wikimedia.org/r/513612

Maintenance_bot removed a project: Patch-For-Review.May 31 2019, 6:11 PM

Gehel mentioned this in T224911: [Epic] Migrate log transport to kafka for Search Platform applications.Jun 3 2019, 4:42 PM

fgiunchedi removed a subtask: T227080: Deprecate all non-Kafka logstash inputs.Jul 2 2019, 12:51 PM

Change 520409 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: set retention to 90 days

https://gerrit.wikimedia.org/r/520409

gerritbot added a project: Patch-For-Review.Jul 3 2019, 10:43 AM

Change 520409 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: set retention to 90 days

https://gerrit.wikimedia.org/r/520409

Maintenance_bot removed a project: Patch-For-Review.Jul 4 2019, 10:10 AM