Increase utilization of application logging pipeline (FY2018-2019 Q3 TEC6)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Jan 8 2019, 10:45 AM

Description

Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch
Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs
Retire udp2log: onboard its producers and consumers to the logging pipeline
[stretch] Implement sensitive log access control, onboard 3 sensitive log producers

Status	Assigned	Task
Resolved	fgiunchedi	T213157 Increase utilization of application logging pipeline (FY2018-2019 Q3 TEC6)
Resolved	herron	T213898 Replace and expand Elasticsearch/Kafka storage in eqiad and upgrade the cluster from Debian jessie to stretch
Resolved	Jclark-ctr	T217556 Decommission old eqiad logstash hardware hosts logstash100[456]
		Unknown Object (Task)
Resolved	herron	T214608 rack/setup/install logstash101[012].eqiad.wmnet
Open	colewhite	T213902 Implement sensitive logstash access control
Open	None	T193766 Ship host syslogs to ELK
Resolved	None	T193664 Knock down puppet 4 deprecation warnings
Open	None	T197173 Ship MX logs to ELK
Open	None	T292682 Develop tooling for quickly parsing 5xx and sampled-1000 logs
Open	None	T288619 Improve the process to consume and use API.LOG to filter out bad performing queries either by extra tooling within o11y or analytics
Open	None	T301110 Ingest webrequest sampled 1000 into logstash
Open	None	T309186 create a sampled log of POST data
Resolved	colewhite	T313868 OpenSearch Security plugin packaging
Resolved	colewhite	T333732 Upgrade OpenSearch using upstream deb packages
Open	None	T350516 Enable OpenSearch security plugin - Beta Logs

herron triaged this task as Medium priority.Jan 8 2019, 7:21 PM

herron added a project: User-herron.

herron subscribed.

• Cmjohnson closed subtask Unknown Object (Task) as Resolved.Jan 17 2019, 10:31 PM

herron reopened subtask Unknown Object (Task) as Open.Jan 22 2019, 3:03 PM

RobH closed subtask Unknown Object (Task) as Resolved.Feb 20 2019, 10:47 PM

herron updated the task description. (Show Details)

Resolving, related tasks have been reparented.