Page MenuHomePhabricator

Send celery logs and events to logstash
Closed, ResolvedPublic

Description

I can't find examples of our celery logs anywhere in logstash. Maybe we should configure this? The events seem particularly interesting, but I guess they would be redundant with the events we collect directly from the app?

Event Timeline

Halfak added a parent task: Restricted Task.Jul 6 2017, 3:01 PM

How do logs get sent to logstash anyway? I'm not even sure how to get started on this task.

[05:13]<mutante> celery seems to mean it's an analytics thing
[05:13]<mutante> is this about getting into stats?
[05:13]<mutante> +https://wikitech.wikimedia.org/wiki/Wikimetrics, ?
[05:13]<mutante> eh, gettin data OUT of stats?
[05:14]<mutante> maybe it's a question for #wikimedia-analytics if they run celerey ?
[05:14]<mutante> the word "celery" doesnt appear in site.pp either

It seems that @Dzahn isn't familiar with our use of celery. I think analytics used celery for something at some point in the past. We use celery for ORES' distributed processing now. Celery's daemon works just like uwsgi. We should be able to direct the logs and roll them in any way we want.

That's right. When i was asked about it on IRC there was no context, basically just the word "celery" and i am not familiar with it.

1:53 PM <ebernhardson> depending on which thing is sending to logstash ... for example amir started sending ores logs to logstash with https://gerrit.wikimedia.org/r/#/c/321096/11/modules/service/manifests/uwsgi.pp
1:53 PM <ebernhardson> basically just stuffs python logging on a socket that logstash reads

Halfak triaged this task as High priority.Jul 20 2017, 3:09 PM

Change 377553 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] [WIP] Send logs to logstash

https://gerrit.wikimedia.org/r/377553

Looks like we already have messages going to logstash, per T175736#3604901. Confirm that celery logs are included, before closing this task.

Change 377553 abandoned by Awight:
[WIP] Send logs to logstash

https://gerrit.wikimedia.org/r/377553

Looks like we already have messages going to logstash, per T175736#3604901. Confirm that celery logs are included, before closing this task.

Celery is in another service (and in case of labs, it's even in another node) by checking ores-worker-01 node in labs I can confirm we are not sending any logs :/

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Change 468314 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/services/ores/deploy@master] Add uwsgi logging

https://gerrit.wikimedia.org/r/468314

Change 468314 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Add uwsgi logging

https://gerrit.wikimedia.org/r/468314

Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:09:14Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)