In our documentation for thanos-query failing probes, we recommend running thanos-query-log-explore against the apache log. This command currently returns without printing anything in many cases - are long-running/expiring queries ending up in the relevant log, and if they do are we parsing them correctly? A few outages have shown no output.

The grafana logs rather than the apache logs seem to show these failing queries more readily, perhaps we need to adjust the tool or write a new one to parse these logs.

Longer-term, we need to centralise these logs to be able to dashboard them and filter them much better.