Page MenuHomePhabricator
Create Task
Maniphest T219919

Move citoid logging to new logging pipeline
Closed, ResolvedPublic
Actions

Description

We need to move Citoid to the new logging pipeline. The config should be updated similar to https://gerrit.wikimedia.org/r/#/c/mediawiki/services/change-propagation/deploy/+/500813 and newest node dependencies should be used (newest version of service-runner.

Details

SubjectRepoBranchLines +/-
citoid: stop using gelf for loggingoperations/deployment-chartsmaster+2 -5
Switch from gelf to rsyslogmediawiki/services/citoidmaster+8 -9
Switch from gelf to rsyslog in citoid chartoperations/deployment-chartsmaster+5 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Pchelolo created this task.Apr 2 2019, 7:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2019, 7:19 PM
• Pchelolo reassigned this task from • Pchelolo to Mvolz.Apr 2 2019, 7:19 PM
• mobrovac triaged this task as Medium priority.Apr 3 2019, 3:23 PM
Mvolz moved this task from Backlog to Production on the Citoid board.Aug 8 2019, 1:57 PM
fgiunchedi added a project: observability.Aug 19 2019, 2:30 PM
fgiunchedi added a comment.Mar 2 2020, 10:36 AM

Reminder/ping as we (SRE Observability) would like to deprecate all non-kafka inputs by end of Q4 FY19/20. If the service is moving (or has moved) to k8s then what's left to do is disable gelf log output and keep on stdout/stderr. If the service isn't moving to k8s then we'll also need to perform puppet-level changes. Thanks!

Mvolz raised the priority of this task from Medium to High.Mar 12 2020, 2:12 PM
fgiunchedi moved this task from Inbox to Externally blocked on the observability board.Mar 16 2020, 2:23 PM
akosiaris added a subtask: T239459: service-runner apps running on kubernetes emit logs with log level 50 .Apr 15 2020, 4:58 PM
gerritbot added a comment.Jun 4 2020, 9:49 AM

Change 602304 had a related patch set uploaded (by Mvolz; owner: Mvolz):
[mediawiki/services/citoid@master] Switch from gelf to rsyslog

https://gerrit.wikimedia.org/r/602304

gerritbot added a project: Patch-For-Review.Jun 4 2020, 9:49 AM
• Pchelolo added a comment.Jun 4 2020, 2:01 PM

The patch above doesn't change anything in production. In general, having 'config.prod.yaml' in citoid source repo is misleading - that config is not what is used in production, and AFAIK that one is not used anywhere. I think it should be removed completely.

Citoid is running in k8s, so the logs are delivered automatically. It only needs to be logging to stdout. Change should be made to deployment-charts repo.

Mvolz added a comment.EditedJun 4 2020, 5:12 PM
In T219919#6192799, @Pchelolo wrote:

The patch above doesn't change anything in production. In general, having 'config.prod.yaml' in citoid source repo is misleading - that config is not what is used in production, and AFAIK that one is not used anywhere. I think it should be removed completely.

Citoid is running in k8s, so the logs are delivered automatically. It only needs to be logging to stdout. Change should be made to deployment-charts repo.

There are some things in config.prod.yaml that aren't in its chart in deployment-prep (i.e. the template to talk to mw-api) - is it inheriting some things from there at present? Would this have to be moved from citoid to deployment-prep then?

Mvolz added a comment.Jun 5 2020, 8:40 AM
In T219919#6193561, @Mvolz wrote:
In T219919#6192799, @Pchelolo wrote:

The patch above doesn't change anything in production. In general, having 'config.prod.yaml' in citoid source repo is misleading - that config is not what is used in production, and AFAIK that one is not used anywhere. I think it should be removed completely.

Citoid is running in k8s, so the logs are delivered automatically. It only needs to be logging to stdout. Change should be made to deployment-charts repo.

There are some things in config.prod.yaml that aren't in its chart in deployment-prep (i.e. the template to talk to mw-api) - is it inheriting some things from there at present? Would this have to be moved from citoid to deployment-prep then?

Also I think config.prod.yaml is used in beta? But don't quote me on that :).

gerritbot added a comment.Jun 5 2020, 1:22 PM

Change 602683 had a related patch set uploaded (by Mvolz; owner: Mvolz):
[operations/deployment-charts@master] Switch from gelf to rsyslog

https://gerrit.wikimedia.org/r/602683

gerritbot added a comment.Jun 5 2020, 1:29 PM

Change 602683 abandoned by Mvolz:
Switch from gelf to rsyslog in citoid chart

https://gerrit.wikimedia.org/r/602683

gerritbot added a comment.Jun 5 2020, 9:38 PM

Change 602304 merged by jenkins-bot:
[mediawiki/services/citoid@master] Switch from gelf to rsyslog

https://gerrit.wikimedia.org/r/602304

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2020, 10:10 PM
fgiunchedi added a comment.Sep 21 2020, 9:24 AM

It looks like citoid is now on k8s but still using gelf for logging, possibly the easiest at this point is switching to stdout logging like other node services on k8s. What do you think @Mvolz ?

lmata moved this task from Externally blocked to Radar on the observability board.Sep 21 2020, 8:28 PM
Mvolz added a comment.Sep 28 2020, 12:42 PM
In T219919#6478632, @fgiunchedi wrote:

It looks like citoid is now on k8s but still using gelf for logging, possibly the easiest at this point is switching to stdout logging like other node services on k8s. What do you think @Mvolz ?

I think that's waiting on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/594492 - which from the citoid standpoint is ready to merge, but is now quite stale. I don't know about eventgate and eventstreams.

But yeah if you want to do a pr to switch, as long as we pass the named-levels config in it, that's fine with me!

gerritbot added a comment.Oct 1 2020, 12:49 PM

Change 631437 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/deployment-charts@master] citoid: stop using gelf for logging

https://gerrit.wikimedia.org/r/631437

gerritbot added a project: Patch-For-Review.Oct 1 2020, 12:49 PM
fgiunchedi added a comment.Oct 1 2020, 12:49 PM
In T219919#6498391, @Mvolz wrote:
In T219919#6478632, @fgiunchedi wrote:

It looks like citoid is now on k8s but still using gelf for logging, possibly the easiest at this point is switching to stdout logging like other node services on k8s. What do you think @Mvolz ?

I think that's waiting on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/594492 - which from the citoid standpoint is ready to merge, but is now quite stale. I don't know about eventgate and eventstreams.

But yeah if you want to do a pr to switch, as long as we pass the named-levels config in it, that's fine with me!

Sounds good, see https://gerrit.wikimedia.org/r/631437

gerritbot added a comment.Oct 5 2020, 1:50 PM

Change 631437 merged by Filippo Giunchedi:
[operations/deployment-charts@master] citoid: stop using gelf for logging

https://gerrit.wikimedia.org/r/631437

Maintenance_bot removed a project: Patch-For-Review.Oct 5 2020, 2:10 PM
fgiunchedi closed this task as Resolved.Oct 5 2020, 2:13 PM

Citoid is using only k8s-native logging now, resolving.

akosiaris closed subtask T239459: service-runner apps running on kubernetes emit logs with log level 50 as Resolved.Jan 13 2021, 3:49 PM
• nnikkhoui mentioned this in T288132: Configure logging.Aug 31 2021, 1:59 PM
Maintenance_bot added a project: SRE Observability.Aug 31 2021, 2:45 PM
• nnikkhoui mentioned this in T290156: Config.dev.yaml and config.prod.yaml need updating to remove references to logstash.Sep 1 2021, 1:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL