We need to move AQS to the new logging pipeline. The config should be updated similar to https://gerrit.wikimedia.org/r/#/c/mediawiki/services/change-propagation/deploy/+/500813 and newest node dependencies should be used (newest version of service-runner)
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| aqs: replace logstash host/port with rsyslog localhost/port | operations/puppet | production | +11 -13 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | herron | T227080 Deprecate all non-Kafka logstash inputs | |||
| Invalid | None | T225122 Migrate services using deprecated Gelf logstash input to Kafka enabled logging pipeline | |||
| Resolved | herron | T225125 Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline | |||
| Resolved | • Pchelolo | T211125 Move service-runner to new logging infrastructure | |||
| Resolved | elukey | T219928 Move AQS logging to new logging pipeline | |||
| Resolved | Milimetric | T236757 Upgrade service-runner on AQS to unblock rsyslog logging |
Event Timeline
Comment Actions
@Nuria, do you think that we could work on this during the next couple of months? Seems to be an easy enough change to be ready in a day or so (including testing).
Comment Actions
Change 543278 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] aqs: replace logstash host/port with rsyslog localhost/port
Comment Actions
@mobrovac I just noticed in the description the newest version of service-runner part. We currently run 2.6.7, is it enough?
Comment Actions
Eh no, unfortunately not. Support for prefixed syslog logging came in v2.6.15. The [diff between v2.6.7 and v2.6.15](https://github.com/wikimedia/service-runner/compare/v2.6.7...v2.6.15) suggests it's safe for you to update to that version. Moreover, it would beneficial if you could update to the latest version, v2.7.3 as it brings many improvements and bug fixes over v2.6.15. Again, the diff suggests there are no backwards-incompatible interface changes, so you should fine in doing so.
Comment Actions
Dan deployed the new version of service-runner for aqs, I applied the puppet patch and verified that the new settings works in logstash:
2020-01-15T08:36:15 level:WARNING pid:4 type:aqs message:startup finished normalized_message:startup finished levelPath:warn/service-runner tags:input-kafka-rsyslog-udp-localhost, rsyslog-udp-localhost, kafka, syslog, es, normalized_message_untrimmed @timestamp:2020-01-15T08:36:15 host:aqs1004 @version:1 _type:aqs _index:logstash-2020.01.15 _score: -
(see tags:input-kafka-rsyslog-udp-localhost ..)