Page MenuHomePhabricator

Investigate error logs for failing geoline maps
Open, Needs TriagePublic

Description

Let's explore what our much more verbose error logging left in the logs, to debug recent geoline failures.

Outcome: our logging caught nothing! Needs some more tweaks...

Event Timeline

awight renamed this task from Investigate error logging for failing geoline maps to Investigate error logs for failing geoline maps.Oct 19 2022, 11:05 AM
awight claimed this task.
awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2022-10-12 board.
awight updated the task description. (Show Details)

I haven't found anything useful in the logs so far. Even more puzzling, the only message that might possibly be associated with these snapshot requests is "Calling 'Metrics.endTiming' directly is deprecated.", which is not emitted for the other requests. Looking at all usages of endTiming, they seem to be identical so I don't understand why the message isn't emitted much more often.

Patch https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/844494 should stop these deprecation warnings.

Perhaps we need to increase the severity level of our errors. For example, the message for a missing geoline is at sent "info" level:

{"name":"kartotherian","hostname":"68d5677a61db","pid":438,"level":30,"levelPath":"info","msg":"No results from geoline local database request","time":"2022-10-19T15:35:06.714Z","v":0}

Change 844494 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/services/kartotherian@master] Update deprecated metrics usages

https://gerrit.wikimedia.org/r/844494

Change 844498 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/services/kartotherian@master] [WIP] Log errors at higher severity

https://gerrit.wikimedia.org/r/844498

Change 852880 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/services/kartotherian@mapnik-3.1] Update deprecated metrics usages; speak prometheus

https://gerrit.wikimedia.org/r/852880

Change 844494 merged by jenkins-bot:

[mediawiki/services/kartotherian@master] Update deprecated metrics usages; speak prometheus

https://gerrit.wikimedia.org/r/844494

Change 852880 merged by jenkins-bot:

[mediawiki/services/kartotherian@mapnik-3.1] Update deprecated metrics usages; speak prometheus

https://gerrit.wikimedia.org/r/852880