Something we should do in general, and specifically to get native trace sampling (cfr T378190).
The upgrade will also fix the following spurious message from thanos-query cc @CDanis
level=error ts=2025-01-17T09:59:26.017154668Z caller=bridge.go:30 msg="OpenTelemetry ErrorHandler" err="OTLP partial success: empty message (0 spans rejected)"
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T383966 Upgrade Thanos to 0.38.0 | |||
| Resolved | fgiunchedi | T392994 Move Thanos trace sampling to native and off otlp coll |
Mentioned in SAL (#wikimedia-operations) [2025-04-14T12:50:45Z] <godog> upgrade prometheus2005 to thanos 0.38.0 - T383966
Mentioned in SAL (#wikimedia-operations) [2025-04-14T12:51:34Z] <godog> upgrade prometheus2007 to thanos 0.38.0 - T383966
Mentioned in SAL (#wikimedia-operations) [2025-04-15T07:47:52Z] <godog> upgrade thanos to 0.38.0 on prometheus100[57] - T383966
Mentioned in SAL (#wikimedia-operations) [2025-04-15T12:21:05Z] <godog> upgrade thanos to 0.38.0 on O:prometheus::pop - T383966
Mentioned in SAL (#wikimedia-operations) [2025-04-16T13:24:40Z] <godog> finish rollout of thanos 0.38 to prometheus* - T383966
Mentioned in SAL (#wikimedia-operations) [2025-04-28T07:29:17Z] <godog> upgrade thanos to 0.38 on titan1* - T383966
Mentioned in SAL (#wikimedia-operations) [2025-04-28T07:34:47Z] <godog> upgrade thanos to 0.38 on titan2* - T383966
Ran into thanos-rule log spam, reported as https://github.com/thanos-io/thanos/issues/8229
Change #1139414 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] thanos: raise rule log level to avoid log spam
Change #1139414 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: raise rule log level to avoid log spam
Change #1139852 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] thanos: enable auto memlimit
Mentioned in SAL (#wikimedia-cloud) [2025-04-29T14:10:22Z] <taavi> upgrading thanos to 0.38 T383966
Change #1139852 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: enable auto memlimit