Page MenuHomePhabricator
Create Task
Maniphest T168921

Send error logs to logstash
Closed, ResolvedPublic
Actions

Description

I noticed that a recent server error didn't seem to appear in logstash. I'm not sure if we're failing to send errors, or if the level is not being recorded.

See also T149010#3136402.

Details

SubjectRepoBranchLines +/-
ores: Change logstash port from GELF to json linesoperations/puppetproduction+1 -1
Start using logstashmediawiki/services/ores/deploymaster+3 -0
ores: Add logstash configoperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects

Event Timeline

awight created this task.Jun 26 2017, 11:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 26 2017, 11:26 PM
Halfak triaged this task as Low priority.Jun 29 2017, 2:50 PM
Halfak raised the priority of this task from Low to High.
Halfak moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.
awight added a comment.Jul 3 2017, 10:53 PM

When returning error responses, ores.wsgi.util.format_error summarizes as type=error class name, message=str cast. This response error handling code might be a good place to log the complete traceback as well.

awight awarded a token.Jul 7 2017, 4:23 AM
Ladsgroup subscribed.Jul 23 2018, 2:10 PM

Today I realized this is very important, we don't report anything outside of uwsgi logs to the logstash and I don't have access to syslog or deamon.log, Basically I can't see any errors of ORES

Ladsgroup added a project: ORES.Aug 3 2018, 11:21 AM
hoo claimed this task.Aug 27 2018, 4:02 PM
hoo added a comment.Sep 1 2018, 6:49 PM

Hm, it seems to me the way forward here would be to include python-logstash and then add it as logging handler via the deployed [[https://phabricator.wikimedia.org/source/ores-deploy/browse/master/logging_config.yaml|logging_config.yaml]].

Ladsgroup added a comment.Sep 3 2018, 11:27 AM

The library seems unmaintained :/ but beside that it's a good idea to use it, worst case, we fork and maintain it.

Ladsgroup claimed this task.Sep 26 2018, 3:35 PM
Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.
Ladsgroup added a subscriber: hoo.

The library is basically four files, the last commit on it was two years ago and it doesn't support python3, we basically can write it from scratch. I will do it.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 26 2018, 3:35 PM
Ladsgroup mentioned this in rORES38d385020d07: Add LogstashHandler and LogstashFormatter.Sep 28 2018, 12:09 PM
Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Sep 28 2018, 2:51 PM
Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:20 PM
awight moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Oct 10 2018, 11:23 PM
awight mentioned this in rORES9c10f5f581d0: Add LogstashHandler and LogstashFormatter (#268).Oct 10 2018, 11:25 PM
gerritbot subscribed.Oct 11 2018, 7:10 PM

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a project: Patch-For-Review.Oct 11 2018, 7:10 PM
gerritbot added a comment.Oct 12 2018, 10:07 AM

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a comment.Oct 12 2018, 11:37 AM

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Ladsgroup mentioned this in rORESDEPLOYba4ea4056c86: Start using logstash.Oct 12 2018, 11:40 AM
gerritbot added a comment.Oct 12 2018, 11:42 AM

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Stashbot mentioned this in T181546: Let the ORES application set log severity, not uWSGI.Oct 18 2018, 5:09 PM
Stashbot mentioned this in T169586: Send celery logs and events to logstash.
Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:09:14Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256

Stashbot mentioned this in T181630: Send celery and wsgi service logs to logstash.Oct 18 2018, 5:09 PM
Stashbot mentioned this in T205256: ORES uwsgi logs in logstash are useless.
Stashbot added a comment.Oct 18 2018, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Oct 18 2018, 9:42 PM
Ladsgroup mentioned this in rORES51ec07c96237: Use json-lines instead of GELF for logstash.Oct 31 2018, 12:37 PM
Ladsgroup mentioned this in rORES58947e3ba1f6: Use json-lines instead of GELF for logstash.Oct 31 2018, 1:03 PM
gerritbot added a comment.Oct 31 2018, 1:09 PM

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

gerritbot added a comment.Oct 31 2018, 7:31 PM

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Stashbot added a comment.Oct 31 2018, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Stashbot mentioned this in T178441: Upgrade celery to 4.1.0 for ORES.Oct 31 2018, 8:06 PM
Stashbot added a comment.Oct 31 2018, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)

Ladsgroup closed this task as Resolved.Nov 1 2018, 8:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits