Page MenuHomePhabricator

Ship peopleweb apache2 error logs to ELK
Closed, ResolvedPublic

Description

Ship peopleweb apache2 logs to logstash.

This is a low-volume apache2 log producer which makes it a good early candidate for log shipping via the logging pipeline built in Q2 2018.

Event Timeline

herron triaged this task as Medium priority.Nov 19 2018, 4:37 PM
herron created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 19 2018, 4:37 PM

Change 474760 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog::input::file add rsyslog imfile wrapper for file ingestion

https://gerrit.wikimedia.org/r/474760

Change 474813 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add type "apache-error" and use logstash core patterns

https://gerrit.wikimedia.org/r/474813

Change 474760 merged by Herron:
[operations/puppet@production] rsyslog::input::file add rsyslog imfile wrapper for file ingestion

https://gerrit.wikimedia.org/r/474760

Change 474813 merged by Herron:
[operations/puppet@production] logstash: add type "apache2-error" and use logstash core patterns

https://gerrit.wikimedia.org/r/474813

Change 474973 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] peopleweb: ingest apache error logs with rsyslog for shipping

https://gerrit.wikimedia.org/r/474973

Change 474974 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: output logs tagged 'input-file-apache2-error' to kafka

https://gerrit.wikimedia.org/r/474974

Change 474974 merged by Herron:
[operations/puppet@production] rsyslog: output logs tagged 'input-file-apache2-error' to kafka

https://gerrit.wikimedia.org/r/474974

Change 474973 merged by Herron:
[operations/puppet@production] peopleweb: ingest apache error logs with rsyslog for shipping

https://gerrit.wikimedia.org/r/474973

Apache error logs for peopleweb (rutherfordium) are now present in ELK. These can be found by searching in Kibana for host:rutherfordium, or program.raw:input-file-apache2-error for example. The volume of errors is low so it may help to search across a wide time range.

herron renamed this task from Ship peopleweb apache2 logs to ELK to Ship peopleweb apache2 error logs to ELK.Nov 20 2018, 9:22 PM
herron closed this task as Resolved.Nov 26 2018, 5:54 PM
Dzahn added a subscriber: Dzahn.EditedNov 26 2018, 5:57 PM

@herron please note that rutherfordium is being replaced by people1001 as we speak (jessie -> stretch). Currently in progress. The new host already has the role and i will switch Varnish backend this week. If there are any manual steps involved or the hostname rutherfordium is hardcoded, that would need changes. [T210036]

@herron please note that rutherfordium is being replaced by people1001 as we speak (jessie -> stretch). Currently in progress. The new host already has the role and i will switch Varnish backend this week. If there are any manual steps involved or the hostname rutherfordium is hardcoded, that would been changes. [T210036]

Ah! Good to know. The log shipping configs were applied at the role level so no host specific changes would be needed, but do let me know if any related issues arise during the stretch upgrade!