Page MenuHomePhabricator
Create Task
Maniphest T169586

Send celery logs and events to logstash
Closed, ResolvedPublic
Actions

Description

I can't find examples of our celery logs anywhere in logstash. Maybe we should configure this? The events seem particularly interesting, but I guess they would be redundant with the events we collect directly from the app?

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1ores: Change logstash port from GELF to json lines
mediawiki/services/ores/deploymaster+2 -0Add uwsgi logging
mediawiki/services/ores/deploymaster+3 -0Start using logstash
operations/puppetproduction+14 -0ores: Add logstash config
mediawiki/services/ores/deploymaster+11 -4[WIP] Send logs to logstash
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 Resolved• LadsgroupT169586 Send celery logs and events to logstash

Event Timeline

awight created this task.Jul 3 2017, 10:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2017, 10:31 PM
Halfak added a parent task: Restricted Task.Jul 6 2017, 3:01 PM
Halfak added a subscriber: Halfak.Jul 7 2017, 8:43 PM

How do logs get sent to logstash anyway? I'm not even sure how to get started on this task.

Halfak added a project: ORES.Jul 7 2017, 8:43 PM
Halfak added a project: Wikimedia-Logstash.
Zppix moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Jul 7 2017, 9:21 PM
Zppix added a subscriber: Zppix.Jul 8 2017, 6:13 PM

[05:13]<mutante> celery seems to mean it's an analytics thing
[05:13]<mutante> is this about getting into stats?
[05:13]<mutante> ＋https://wikitech.wikimedia.org/wiki/Wikimetrics， ?
[05:13]<mutante> eh, gettin data OUT of stats?
[05:14]<mutante> maybe it's a question for #wikimedia-analytics if they run celerey ?
[05:14]<mutante> the word "celery" doesnt appear in site.pp either

Halfak added a subscriber: Dzahn.Jul 9 2017, 3:29 PM

It seems that @Dzahn isn't familiar with our use of celery. I think analytics used celery for something at some point in the past. We use celery for ORES' distributed processing now. Celery's daemon works just like uwsgi. We should be able to direct the logs and roll them in any way we want.

Dzahn added a comment.Jul 9 2017, 4:32 PM

That's right. When i was asked about it on IRC there was no context, basically just the word "celery" and i am not familiar with it.

Dzahn removed a subscriber: Dzahn.Jul 9 2017, 4:33 PM
Zppix added a comment.Jul 14 2017, 6:59 PM

1:53 PM <ebernhardson> depending on which thing is sending to logstash ... for example amir started sending ores logs to logstash with https://gerrit.wikimedia.org/r/#/c/321096/11/modules/service/manifests/uwsgi.pp
1:53 PM <ebernhardson> basically just stuffs python logging on a socket that logstash reads

Halfak triaged this task as High priority.Jul 20 2017, 3:09 PM
Halfak merged a task: T175653: ORES log messages should be sent to logstash.Sep 12 2017, 9:18 PM
Halfak added a subscriber: gerritbot.
gerritbot added a comment.Sep 12 2017, 9:20 PM

Change 377553 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] [WIP] Send logs to logstash

https://gerrit.wikimedia.org/r/377553

gerritbot added a project: Patch-For-Review.Sep 12 2017, 9:20 PM
awight mentioned this in rORESDEPLOY06b0b1e2fff9: [WIP] Send logs to logstash.Sep 12 2017, 9:20 PM
awight mentioned this in T175736: Give ores admins read access to /srv/log/ores/main.log*.Sep 13 2017, 3:56 PM
awight added a comment.Sep 13 2017, 4:38 PM

Looks like we already have messages going to logstash, per T175736#3604901. Confirm that celery logs are included, before closing this task.

awight added a project: TestMe.Nov 1 2017, 5:12 PM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptNov 1 2017, 5:12 PM
gerritbot added a comment.May 9 2018, 9:14 PM

Change 377553 abandoned by Awight:
[WIP] Send logs to logstash

https://gerrit.wikimedia.org/r/377553

fgiunchedi moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Aug 6 2018, 1:12 PM
• Ladsgroup added a subscriber: • Ladsgroup.Sep 7 2018, 11:49 AM
In T169586#3604923, @awight wrote:

Looks like we already have messages going to logstash, per T175736#3604901. Confirm that celery logs are included, before closing this task.

Celery is in another service (and in case of labs, it's even in another node) by checking ores-worker-01 node in labs I can confirm we are not sending any logs :/

• Ladsgroup claimed this task.Sep 26 2018, 3:56 PM
• Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed TestMe, Patch-For-Review, Machine-Learning-Team.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 26 2018, 3:56 PM
• Ladsgroup mentioned this in rORES38d385020d07: Add LogstashHandler and LogstashFormatter.Sep 28 2018, 12:09 PM
• Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Sep 28 2018, 2:51 PM
• Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:20 PM
awight moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Oct 10 2018, 11:23 PM
awight mentioned this in rORES9c10f5f581d0: Add LogstashHandler and LogstashFormatter (#268).Oct 10 2018, 11:25 PM
gerritbot added a comment.Oct 11 2018, 7:10 PM

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a project: Patch-For-Review.Oct 11 2018, 7:10 PM
gerritbot added a comment.Oct 12 2018, 10:07 AM

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a comment.Oct 12 2018, 11:37 AM

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

• Ladsgroup mentioned this in rORESDEPLOYba4ea4056c86: Start using logstash.Oct 12 2018, 11:40 AM
gerritbot added a comment.Oct 12 2018, 11:42 AM

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

gerritbot added a comment.Oct 18 2018, 1:24 PM

Change 468314 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/services/ores/deploy@master] Add uwsgi logging

https://gerrit.wikimedia.org/r/468314

gerritbot added a comment.Oct 18 2018, 1:25 PM

Change 468314 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Add uwsgi logging

https://gerrit.wikimedia.org/r/468314

• Ladsgroup mentioned this in rORESDEPLOYd724d2030d9e: Add uwsgi logging.Oct 18 2018, 1:33 PM
Stashbot mentioned this in T181546: Let the ORES application set log severity, not uWSGI.Oct 18 2018, 5:09 PM
Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:09:14Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256

Stashbot mentioned this in T168921: Send error logs to logstash.Oct 18 2018, 5:09 PM
Stashbot mentioned this in T181630: Send celery and wsgi service logs to logstash.
Stashbot mentioned this in T205256: ORES uwsgi logs in logstash are useless.
Stashbot added a comment.Oct 18 2018, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

• Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Oct 18 2018, 9:42 PM
• Ladsgroup mentioned this in rORES51ec07c96237: Use json-lines instead of GELF for logstash.Oct 31 2018, 12:37 PM
• Ladsgroup mentioned this in rORES58947e3ba1f6: Use json-lines instead of GELF for logstash.Oct 31 2018, 1:03 PM
gerritbot added a comment.Oct 31 2018, 1:09 PM

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

gerritbot added a comment.Oct 31 2018, 7:31 PM

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Stashbot added a comment.Oct 31 2018, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Stashbot mentioned this in T178441: Upgrade celery to 4.1.0 for ORES.Oct 31 2018, 8:06 PM
Stashbot added a comment.Oct 31 2018, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)

• Ladsgroup closed this task as Resolved.Nov 1 2018, 8:00 PM
fgiunchedi added a project: observability.Aug 19 2019, 2:28 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL