Page MenuHomePhabricator
Create Task
Maniphest T205256

ORES uwsgi logs in logstash are useless
Closed, ResolvedPublic
Actions

Description

So if you go to ORES dashboard on logstash, it's basically list of requests coming there. It took me so long to come with the idea of adding '-200' to remove 200 responses and then added '-404' and some other to get basic meaning of the logs. Still there is lots of nonsense regarding respawning web workers, etc. and you have to filter out so many words until you get some meaning out of it.
This task is done when we have a "Fatal monitor" dashboard for ores

Details

SubjectRepoBranchLines +/-
ores: Change logstash port from GELF to json linesoperations/puppetproduction+1 -1
Start using logstashmediawiki/services/ores/deploymaster+3 -0
ores: Add logstash configoperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects

Event Timeline

Ladsgroup created this task.Sep 24 2018, 9:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 24 2018, 9:10 AM
Ladsgroup claimed this task.Sep 26 2018, 12:42 PM
Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 26 2018, 12:42 PM
Ladsgroup added a subscriber: akosiaris.Sep 26 2018, 2:51 PM

The reason that everything is INFO is that our uwsgi encoder is designed to do so: https://github.com/wikimedia/puppet/blob/production/modules/service/manifests/uwsgi.pp#L166

It's not easily fixable as uwsgi doesn't pass around log level: https://github.com/unbit/uwsgi/blob/3149df02ed443131c54ea6afb29fcbb0ed4d1139/core/logging.c#L1860

I'm basically thinking of writing a shiny support for logstash, that'll be fun. Only thing before moving forward: Does k8s solve this issue? if k8s can send everything in stderr to logstash, all work here would be useless. @akosiaris do you know more about this?

Ladsgroup mentioned this in rORES38d385020d07: Add LogstashHandler and LogstashFormatter.Sep 28 2018, 12:09 PM
Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Sep 28 2018, 2:51 PM
akosiaris added a comment.Oct 2 2018, 11:33 AM
In T205256#4619239, @Ladsgroup wrote:

The reason that everything is INFO is that our uwsgi encoder is designed to do so: https://github.com/wikimedia/puppet/blob/production/modules/service/manifests/uwsgi.pp#L166

Yes, given how we did it it's expected. We should be deprecating this approach and not rely on uwsgi for logging or at the very least find a way to inform it of the log level (see below for an approach)

It's not easily fixable as uwsgi doesn't pass around log level: https://github.com/unbit/uwsgi/blob/3149df02ed443131c54ea6afb29fcbb0ed4d1139/core/logging.c#L1860

It's not up to uwsgi to pass around loglevel but per https://uwsgi-docs.readthedocs.io/en/latest/LogFormat.html#user-defined-logvars it might be doable

I'm basically thinking of writing a shiny support for logstash, that'll be fun. Only thing before moving forward: Does k8s solve this issue? if k8s can send everything in stderr to logstash, all work here would be useless. @akosiaris do you know more about this?

Partly (the application needs to cooperate). So the work we will be doing this quarter is about forwarding logs from applications running on kubernetes to logstash. The applications will be logging to stdout/stderr, in order to not break commonly expected working kubernetes patterns like kubectl logs <pod_name> and WMF infrastructure will be collecting those logs and forwarding them. I expect this to be declared ready for use at the end of the quarter, possibly a bit earlier.

However, note that kubernetes can not know the log level of an entry. It is something that only the application knows and it's the only one that should be setting that.

Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:19 PM
awight moved this task from Review to Parked on the Machine-Learning-Team (Active Tasks) board.Oct 4 2018, 5:48 PM
awight subscribed.

Moving from review->active, apologies in advance if this is incorrect.

awight moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Oct 4 2018, 5:56 PM

Oops, I see this is tagged in a patch for review.

awight moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Oct 10 2018, 11:23 PM
awight mentioned this in rORES9c10f5f581d0: Add LogstashHandler and LogstashFormatter (#268).Oct 10 2018, 11:25 PM
gerritbot subscribed.Oct 11 2018, 7:10 PM

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a project: Patch-For-Review.Oct 11 2018, 7:10 PM
gerritbot added a comment.Oct 12 2018, 10:07 AM

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a comment.Oct 12 2018, 11:37 AM

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Ladsgroup mentioned this in rORESDEPLOYba4ea4056c86: Start using logstash.Oct 12 2018, 11:40 AM
gerritbot added a comment.Oct 12 2018, 11:42 AM

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Stashbot mentioned this in T181546: Let the ORES application set log severity, not uWSGI.Oct 18 2018, 5:09 PM
Stashbot mentioned this in T169586: Send celery logs and events to logstash.
Stashbot mentioned this in T168921: Send error logs to logstash.
Stashbot mentioned this in T181630: Send celery and wsgi service logs to logstash.
Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:09:14Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256

Stashbot added a comment.Oct 18 2018, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Oct 18 2018, 9:42 PM
Ladsgroup mentioned this in rORES51ec07c96237: Use json-lines instead of GELF for logstash.Oct 31 2018, 12:37 PM
Ladsgroup mentioned this in rORES58947e3ba1f6: Use json-lines instead of GELF for logstash.Oct 31 2018, 1:03 PM
gerritbot added a comment.Oct 31 2018, 1:09 PM

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

gerritbot added a comment.Oct 31 2018, 7:31 PM

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Stashbot added a comment.Oct 31 2018, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Stashbot mentioned this in T178441: Upgrade celery to 4.1.0 for ORES.Oct 31 2018, 8:06 PM
Stashbot added a comment.Oct 31 2018, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)

Ladsgroup closed this task as Resolved.Nov 1 2018, 7:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL