Page MenuHomePhabricator

Send celery and wsgi service logs to logstash
Closed, ResolvedPublic

Description

There's good stuff in service celery-ores-worker status -l, we want that made visible.

Event Timeline

awight created this task.Nov 29 2017, 2:34 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 29 2017, 2:34 PM

A slightly related request--it looks like /srv/log/ores/main.log is created by modules/service/manifests/uwsgi.pp, it would be great if the logfile were readable by regular users.

awight added a comment.Dec 5 2017, 3:51 PM

Celery is now logging verbosely to /srv/log/ores/app.log, please wire that into logstash.

Celery is now logging verbosely to /srv/log/ores/app.log, please wire that into logstash.

We will need celery to actually do that. A quick look at the celery code says it is using the standand logging python module. Which makes it easier to send stuff to logstash but we will need https://pypi.python.org/pypi/python-logstash. So we need to add it in the venv and then configure it. I can do the latter, but the former needs someone else. @awight ?

Ottomata triaged this task as Medium priority.Jan 16 2018, 8:07 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 26 2018, 3:56 PM
Dzahn removed a subscriber: Dzahn.Sep 28 2018, 5:13 PM
Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:20 PM

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

It needs some changes. I will make them and when you do them it just works:


cc @fgiunchedi

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)

Ladsgroup closed this task as Resolved.Nov 1 2018, 8:01 PM