during a recent incident it was observed that it would be nice to have a sampling of the POST body contents logged somewhere, at least the first N bytes 1/1K times in short-term somewhere
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T213157 Increase utilization of application logging pipeline (FY2018-2019 Q3 TEC6) | |||
| Resolved | fgiunchedi | T220103 TEC6: Logging infrastructure (Q4 2018/19 goal) | |||
| Open | colewhite | T213902 Implement sensitive logstash access control | |||
| Resolved | jbond | T309178 text-https:443 has failed probes (retrospective task) | |||
| Restricted Task | |||||
| Open | None | T309186 create a sampled log of POST data |
Event Timeline
Comment Actions
We have something like this for POST requests to api.php on appservers, which we log (unsampled) to api.log on the mwlog host (excluded from Logstash). I've updated the documentation about this and other log files at https://wikitech.wikimedia.org/wiki/Logs.
I'm not suggesting that we take the same approach per-se. But, it's an option. Though regardless of approach, we'll probably want to sample these and/or limit to a particular appserver host. If we approach it from within MW (e.g. WikimediaEvents extension), we'd do good to do it from a post-send deferred update so as to take away any runtime overhead.
We also have something similar for GET requests for any kind, with (sampled, from small set of apaches), which we send to apache2.log and to Logstash.
Comment Actions
Apparently SRE Observability team has it in their board, I am tagging with that and removing SRE