Page MenuHomePhabricator

Send error logs to logstash
Closed, ResolvedPublic

Description

I noticed that a recent server error didn't seem to appear in logstash. I'm not sure if we're failing to send errors, or if the level is not being recorded.

See also T149010#3136402.

Event Timeline

awight created this task.Jun 26 2017, 11:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 26 2017, 11:26 PM
Halfak triaged this task as Low priority.Jun 29 2017, 2:50 PM
Halfak raised the priority of this task from Low to High.
Halfak moved this task from Untriaged to Maintenance/cleanup on the Scoring-platform-team board.

When returning error responses, ores.wsgi.util.format_error summarizes as type=error class name, message=str cast. This response error handling code might be a good place to log the complete traceback as well.

awight awarded a token.Jul 7 2017, 4:23 AM

Today I realized this is very important, we don't report anything outside of uwsgi logs to the logstash and I don't have access to syslog or deamon.log, Basically I can't see any errors of ORES

hoo claimed this task.Aug 27 2018, 4:02 PM
hoo added a comment.Sep 1 2018, 6:49 PM

Hm, it seems to me the way forward here would be to include python-logstash and then add it as logging handler via the deployed logging_config.yaml.

The library seems unmaintained :/ but beside that it's a good idea to use it, worst case, we fork and maintain it.

Ladsgroup added a subscriber: hoo.

The library is basically four files, the last commit on it was two years ago and it doesn't support python3, we basically can write it from scratch. I will do it.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 26 2018, 3:35 PM
Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:20 PM

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:09:14Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)

Ladsgroup closed this task as Resolved.Nov 1 2018, 8:00 PM