Page MenuHomePhabricator
Create Task
Maniphest T181546

Let the ORES application set log severity, not uWSGI
Closed, ResolvedPublic
Actions

Description

Seems that uwsgi is masking our log levels, so we get things like:

fork(): Cannot allocate memory [core/master_utils.c line 724]

at INFO level rather than OFAK as it should be.

Details

SubjectRepoBranchLines +/-
ores: Change logstash port from GELF to json linesoperations/puppetproduction+1 -1
Start using logstashmediawiki/services/ores/deploymaster+3 -0
ores: Add logstash configoperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

awight created this task.Nov 28 2017, 6:49 PM
Ottomata triaged this task as Medium priority.Jan 16 2018, 8:08 PM
awight moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Jun 21 2018, 11:38 AM
awight added a subscriber: Ladsgroup.Sep 26 2018, 6:29 PM

@Ladsgroup This might be of interest for your logging spree.

Ladsgroup claimed this task.Sep 27 2018, 5:25 AM
Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

It will be also handled by changes to the logging system of ours.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 27 2018, 5:25 AM
Ladsgroup mentioned this in rORES38d385020d07: Add LogstashHandler and LogstashFormatter.Sep 28 2018, 12:09 PM
Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Sep 28 2018, 2:51 PM
Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:20 PM
awight moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Oct 10 2018, 11:23 PM
awight mentioned this in rORES9c10f5f581d0: Add LogstashHandler and LogstashFormatter (#268).Oct 10 2018, 11:25 PM
gerritbot added a comment.Oct 11 2018, 7:10 PM

Change 466716 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a project: Patch-For-Review.Oct 11 2018, 7:10 PM
gerritbot added a comment.Oct 12 2018, 10:07 AM

Change 466716 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Add logstash config

https://gerrit.wikimedia.org/r/466716

gerritbot added a comment.Oct 12 2018, 11:37 AM

Change 466857 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Ladsgroup mentioned this in rORESDEPLOYba4ea4056c86: Start using logstash.Oct 12 2018, 11:40 AM
gerritbot added a comment.Oct 12 2018, 11:42 AM

Change 466857 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Start using logstash

https://gerrit.wikimedia.org/r/466857

Stashbot added a comment.Oct 18 2018, 5:09 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:09:14Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256

Stashbot mentioned this in T169586: Send celery logs and events to logstash.Oct 18 2018, 5:09 PM
Stashbot mentioned this in T168921: Send error logs to logstash.
Stashbot mentioned this in T181630: Send celery and wsgi service logs to logstash.
Stashbot mentioned this in T205256: ORES uwsgi logs in logstash are useless.
Stashbot added a comment.Oct 18 2018, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T17:33:00Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s)

Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Oct 18 2018, 9:42 PM
Ladsgroup mentioned this in rORES51ec07c96237: Use json-lines instead of GELF for logstash.Oct 31 2018, 12:37 PM
Ladsgroup mentioned this in rORES58947e3ba1f6: Use json-lines instead of GELF for logstash.Oct 31 2018, 1:03 PM
gerritbot added a comment.Oct 31 2018, 1:09 PM

Change 470827 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

gerritbot added a comment.Oct 31 2018, 7:31 PM

Change 470827 merged by Dzahn:
[operations/puppet@production] ores: Change logstash port from GELF to json lines

https://gerrit.wikimedia.org/r/470827

Stashbot added a comment.Oct 31 2018, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:06:27Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441

Stashbot mentioned this in T178441: Upgrade celery to 4.1.0 for ORES.Oct 31 2018, 8:06 PM
Stashbot added a comment.Oct 31 2018, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-31T20:27:56Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s)

Ladsgroup closed this task as Resolved.Nov 1 2018, 8:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits