Page MenuHomePhabricator

Move citoid logging to new logging pipeline
Closed, ResolvedPublic

Description

We need to move Citoid to the new logging pipeline. The config should be updated similar to https://gerrit.wikimedia.org/r/#/c/mediawiki/services/change-propagation/deploy/+/500813 and newest node dependencies should be used (newest version of service-runner.

Event Timeline

Pchelolo created this task.Apr 2 2019, 7:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2019, 7:19 PM
Pchelolo reassigned this task from Pchelolo to Mvolz.Apr 2 2019, 7:19 PM
mobrovac triaged this task as Medium priority.Apr 3 2019, 3:23 PM
Mvolz moved this task from Backlog to Production on the Citoid board.Aug 8 2019, 1:57 PM

Reminder/ping as we (SRE Observability) would like to deprecate all non-kafka inputs by end of Q4 FY19/20. If the service is moving (or has moved) to k8s then what's left to do is disable gelf log output and keep on stdout/stderr. If the service isn't moving to k8s then we'll also need to perform puppet-level changes. Thanks!

Mvolz raised the priority of this task from Medium to High.Mar 12 2020, 2:12 PM

Change 602304 had a related patch set uploaded (by Mvolz; owner: Mvolz):
[mediawiki/services/citoid@master] Switch from gelf to rsyslog

https://gerrit.wikimedia.org/r/602304

The patch above doesn't change anything in production. In general, having 'config.prod.yaml' in citoid source repo is misleading - that config is not what is used in production, and AFAIK that one is not used anywhere. I think it should be removed completely.

Citoid is running in k8s, so the logs are delivered automatically. It only needs to be logging to stdout. Change should be made to deployment-charts repo.

Mvolz added a comment.EditedJun 4 2020, 5:12 PM

The patch above doesn't change anything in production. In general, having 'config.prod.yaml' in citoid source repo is misleading - that config is not what is used in production, and AFAIK that one is not used anywhere. I think it should be removed completely.

Citoid is running in k8s, so the logs are delivered automatically. It only needs to be logging to stdout. Change should be made to deployment-charts repo.

There are some things in config.prod.yaml that aren't in its chart in deployment-prep (i.e. the template to talk to mw-api) - is it inheriting some things from there at present? Would this have to be moved from citoid to deployment-prep then?

Mvolz added a comment.Jun 5 2020, 8:40 AM

The patch above doesn't change anything in production. In general, having 'config.prod.yaml' in citoid source repo is misleading - that config is not what is used in production, and AFAIK that one is not used anywhere. I think it should be removed completely.

Citoid is running in k8s, so the logs are delivered automatically. It only needs to be logging to stdout. Change should be made to deployment-charts repo.

There are some things in config.prod.yaml that aren't in its chart in deployment-prep (i.e. the template to talk to mw-api) - is it inheriting some things from there at present? Would this have to be moved from citoid to deployment-prep then?

Also I think config.prod.yaml is used in beta? But don't quote me on that :).

Change 602683 had a related patch set uploaded (by Mvolz; owner: Mvolz):
[operations/deployment-charts@master] Switch from gelf to rsyslog

https://gerrit.wikimedia.org/r/602683

Change 602683 abandoned by Mvolz:
Switch from gelf to rsyslog in citoid chart

https://gerrit.wikimedia.org/r/602683

Change 602304 merged by jenkins-bot:
[mediawiki/services/citoid@master] Switch from gelf to rsyslog

https://gerrit.wikimedia.org/r/602304

It looks like citoid is now on k8s but still using gelf for logging, possibly the easiest at this point is switching to stdout logging like other node services on k8s. What do you think @Mvolz ?

lmata moved this task from Externally blocked to Radar on the observability board.Sep 21 2020, 8:28 PM

It looks like citoid is now on k8s but still using gelf for logging, possibly the easiest at this point is switching to stdout logging like other node services on k8s. What do you think @Mvolz ?

I think that's waiting on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/594492 - which from the citoid standpoint is ready to merge, but is now quite stale. I don't know about eventgate and eventstreams.

But yeah if you want to do a pr to switch, as long as we pass the named-levels config in it, that's fine with me!

Change 631437 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/deployment-charts@master] citoid: stop using gelf for logging

https://gerrit.wikimedia.org/r/631437

It looks like citoid is now on k8s but still using gelf for logging, possibly the easiest at this point is switching to stdout logging like other node services on k8s. What do you think @Mvolz ?

I think that's waiting on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/594492 - which from the citoid standpoint is ready to merge, but is now quite stale. I don't know about eventgate and eventstreams.

But yeah if you want to do a pr to switch, as long as we pass the named-levels config in it, that's fine with me!

Sounds good, see https://gerrit.wikimedia.org/r/631437

Change 631437 merged by Filippo Giunchedi:
[operations/deployment-charts@master] citoid: stop using gelf for logging

https://gerrit.wikimedia.org/r/631437

fgiunchedi closed this task as Resolved.Oct 5 2020, 2:13 PM

Citoid is using only k8s-native logging now, resolving.