Page MenuHomePhabricator

Move AQS logging to new logging pipeline
Closed, ResolvedPublic

Description

We need to move AQS to the new logging pipeline. The config should be updated similar to https://gerrit.wikimedia.org/r/#/c/mediawiki/services/change-propagation/deploy/+/500813 and newest node dependencies should be used (newest version of service-runner)

Event Timeline

Pchelolo created this task.Apr 2 2019, 7:31 PM
fdans triaged this task as Medium priority.Apr 4 2019, 5:07 PM
fdans moved this task from Incoming to Operational Excellence on the Analytics board.

@Nuria, do you think that we could work on this during the next couple of months? Seems to be an easy enough change to be ready in a day or so (including testing).

Nuria added a comment.Jul 1 2019, 2:50 PM

Let's plan this for this quarter then? (q2?)

Change 543278 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] aqs: replace logstash host/port with rsyslog localhost/port

https://gerrit.wikimedia.org/r/543278

elukey claimed this task.Oct 16 2019, 8:02 AM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey moved this task from Backlog to In Progress on the User-Elukey board.Oct 16 2019, 8:14 AM

@mobrovac I just noticed in the description the newest version of service-runner part. We currently run 2.6.7, is it enough?

@mobrovac I just noticed in the description the newest version of service-runner part. We currently run 2.6.7, is it enough?

Eh no, unfortunately not. Support for prefixed syslog logging came in v2.6.15. The [diff between v2.6.7 and v2.6.15](https://github.com/wikimedia/service-runner/compare/v2.6.7...v2.6.15) suggests it's safe for you to update to that version. Moreover, it would beneficial if you could update to the latest version, v2.7.3 as it brings many improvements and bug fixes over v2.6.15. Again, the diff suggests there are no backwards-incompatible interface changes, so you should fine in doing so.

Fine for me, will have to coordinate with my team on upgrading service-runner first!

elukey moved this task from In Progress to Stalled on the User-Elukey board.Jan 3 2020, 10:59 AM
elukey changed the task status from Stalled to Open.Jan 15 2020, 8:41 AM
elukey moved this task from Paused to Done on the Analytics-Kanban board.
elukey set Final Story Points to 5.

Dan deployed the new version of service-runner for aqs, I applied the puppet patch and verified that the new settings works in logstash:

2020-01-15T08:36:15	level:WARNING pid:4 type:aqs message:startup finished normalized_message:startup finished levelPath:warn/service-runner tags:input-kafka-rsyslog-udp-localhost, rsyslog-udp-localhost, kafka, syslog, es, normalized_message_untrimmed @timestamp:2020-01-15T08:36:15 host:aqs1004 @version:1 _type:aqs _index:logstash-2020.01.15 _score: -

(see tags:input-kafka-rsyslog-udp-localhost ..)

elukey moved this task from Stalled to Done on the User-Elukey board.Jan 17 2020, 3:39 PM