Page MenuHomePhabricator

Consider logging 4xx action API errors in RESTBase
Closed, ResolvedPublic

Description

It seems RESTBase only logs failed action API calls when they are not mapped to a 4xx code in action.js, which makes sense from an operational perspective, but seeing the 4xx errors would be nice when debugging clients. Maybe those could be logged to a separate channel?

Event Timeline

mobrovac subscribed.

We don't generally log client errors to reduce noise and have the general policy of one channel per service in logstash, so this would be quite an undertaking.

If you don't feel strongly about this, I'm inclined to decline the task.

I don't feel strongly about it since reading lists just proxy the requests so it's not any harder to do the logging on the action API side. In general though I think providing information about client errors will be important when there is an API which is complex enough that clients can easily get it wrong (ie. not read-only) and the API owner is working closely with another team doing client development.

As a middle ground, we can log them with DEBUG level and we have a feature to probabilistically elevate the logged level per channel, so we can start logging 5% of them to logstash - that will allow having some idea about most common client errors and avoid unnecessary noise in the logs. How does that sound?

@Pchelolo sounds like a good compromise to me.

Change 430087 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Add 1% sampling for no-index and action api 400 error logs.

https://gerrit.wikimedia.org/r/430087

Change 430087 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Add 1% sampling for no-index and action api 400 error logs.

https://gerrit.wikimedia.org/r/430087

Pchelolo claimed this task.

All the 4xx from MW Action API except 404 are logged with 1% probability now. Log entry example: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.05.02/restbase?id=AWMiRbXNpesmgM3lqi_i&_g=h@b74aee6

I'm resolving the ticket, please reopen if you want to add more data to the log or if you feel like we need to increase the sampling probability.