We want to update MediaWiki DB query comments to include the request ID, so we can link slow/problematic queries with the request that triggered them.
Tendril will need to be updated to strip out the request ID when aggregating/grouping queries together per discussion on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/721926
The proposed syntax is r:{request_id} at the end of the comment, so a regex should be able to remove it.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T201409 Harmonise the identification of requests across our stack | |||
| Resolved | Marostegui | T277417 14 March 2021 Wikimedia API Outage | |||
| Declined | Krinkle | T193050 Include request id (if present) in a comment in DB queries | |||
| Duplicate | None | T291419 Update tendril or its replacement to support having request IDs in DB query comments |
Upgrading Tendril is pretty much impossible.
A few months ago we decided to remove Tendril from our infra.
It has served us nicely but unfortunately we have reached a point where it doesn't escale anymore, it is an in-house tool, and it has been giving lots of headaches T281486 T231185 T249085 T252331 T262782 T231182 T231165 T252324 - it has been unmaintained since 2015 or so (we have only committed changes that remove features).
We use Tendril for mainly two things:
We have replaced the first part already with Orchestrator (https://phabricator.wikimedia.org/tag/orchestrator/), we are evaluating options for the second feature, and we are going to test PMM (T273054). We are hoping to have Tendril removed by this fiscal year if things go well.
Even if we wanted, this change isn't easy at all and we wouldn't even know where to start from. Tendril is very complex and sort of a blackbox at the moment. For its slow query reports it uses lots of materialized tables and events (all based on show processlist - it doesn't use performance_schema or anything similar).
Upgrading tendril isn't technically a blocker for adding request ID to the queries, unfortunately, based on the way Tendril works, I think that'd make its slow query report pretty much unusable as, as far as I know, all the queries would have a different string and hence Tendril will fail to group them.
My suggestion would be to wait to see which solution we go for to replace this Tendril's feature before pushing the above change.