PDF rendering was impacted as part of T372470. We should have monitoring that will proactively detect when an individual format has a surge in errors. The easiest path here is getting better metrics and monitoring spikes - as far as I know we don't have per-format failure counters, which would be vital here.
Having per-format canaries is an option, but is both expensive and complicated to implement as far our interaction with Swift goes.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hnowlan | T372470 Elevated 429 responses from Thumbor on codfw starting 2024-08-14 00:00 UTC | |||
| Open | None | T376538 Per-format monitoring for Thumbor |
Would per-format canaries allow to allocate different hardware resources per file format? TIFF files would benefit to get more memory, for example.
Setting the priority to high as this was marked as an incident follow-up.
@hnowlan shall we unassign you from this one, to allow someone else to pick it up?
Looking at current metrics, we have timing metrics split per format but not success/failure. I *think* this is something we can add within thumbor-plugins (from a cursory look at the code), but is contingent on what was pointed out by @MoritzMuehlenhoff in T411076#11544976, namely that we can rebuild thumbor-plugins freely.