⚓ T181630 Send celery and wsgi service logs to logstash

Subject	Repo	Branch	Lines +/-
ores: Change logstash port from GELF to json lines	operations/puppet	production	+1 -1
Start using logstash	mediawiki/services/ores/deploy	master	+3 -0
ores: Add logstash config	operations/puppet	production	+14 -0

		Status	Subtype	Assigned	Task
		Resolved		None	T181538 ORES overload incident, 2017-11-28
		Resolved		Ladsgroup	T181630 Send celery and wsgi service logs to logstash

awight created this task.Nov 29 2017, 2:34 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 29 2017, 2:34 PM

A slightly related request--it looks like /srv/log/ores/main.log is created by modules/service/manifests/uwsgi.pp, it would be great if the logfile were readable by regular users.

Dzahn added a project: Wikimedia-Logstash.Nov 30 2017, 12:39 AM

Celery is now logging verbosely to /srv/log/ores/app.log, please wire that into logstash.

In T181630#3813187, @awight wrote:

Celery is now logging verbosely to /srv/log/ores/app.log, please wire that into logstash.

We will need celery to actually do that. A quick look at the celery code says it is using the standand logging python module. Which makes it easier to send stuff to logstash but we will need https://pypi.python.org/pypi/python-logstash. So we need to add it in the venv and then configure it. I can do the latter, but the former needs someone else. @awight ?

greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Dec 5 2017, 8:13 PM

Ottomata triaged this task as Medium priority.Jan 16 2018, 8:07 PM

awight moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Jun 21 2018, 11:38 AM

fgiunchedi moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Aug 6 2018, 1:06 PM

Ladsgroup claimed this task.Sep 26 2018, 3:56 PM

Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 26 2018, 3:56 PM

Ladsgroup mentioned this in rORES38d385020d07: Add LogstashHandler and LogstashFormatter.Sep 28 2018, 12:09 PM

Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Sep 28 2018, 2:51 PM

Dzahn unsubscribed.Sep 28 2018, 5:13 PM

Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:20 PM

awight moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Oct 10 2018, 11:23 PM

awight mentioned this in rORES9c10f5f581d0: Add LogstashHandler and LogstashFormatter (#268).Oct 10 2018, 11:25 PM

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a project: Patch-For-Review.Oct 11 2018, 7:10 PM

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Ladsgroup mentioned this in rORESDEPLOYba4ea4056c86: Start using logstash.Oct 12 2018, 11:40 AM

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

It works: https://logstash-beta.wmflabs.org/goto/f9a3fea8c95c02724813fb7cbebb6471
Shall we go prod? let's do it on Monday.

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:09:14Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256

Stashbot mentioned this in T205256: ORES uwsgi logs in logstash are useless.Oct 18 2018, 5:09 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Oct 18 2018, 9:42 PM

It needs some changes. I will make them and when you do them it just works:

cc @fgiunchedi

Ladsgroup mentioned this in rORES51ec07c96237: Use json-lines instead of GELF for logstash.Oct 31 2018, 12:37 PM

Ladsgroup mentioned this in rORES58947e3ba1f6: Use json-lines instead of GELF for logstash.Oct 31 2018, 1:03 PM

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Stashbot mentioned this in T178441: Upgrade celery to 4.1.0 for ORES.Oct 31 2018, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)