Page MenuHomePhabricator
Create Task
Maniphest T384841

Upgrade to Grafana 11
Closed, ResolvedPublic
Actions

Description

The task tracks Grafana 11 upgrade, rough steps off the top of my head:

  • Validate the new deb and puppet keeps working as expected in a testing environment (e.g. Pontoon)
  • Temp disable sync between grafana hosts in production and upgrade (dpkg -i) only grafana-next
  • Verify dashboards work as expected on grafana-next, and invite users to try it out too. Make sure T384831 is fixed as well.
  • Change reprepro to import grafana 11 debs
  • After verification period is over, upgrade grafana.w.o too
  • Re enable sync between grafana hosts

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
grafana: Enable dashboard sync between hostsoperations/puppetproduction+1 -1
grafana: Toggle data sync using feature flagoperations/puppetproduction+1 -0
grafana: Add enable_dashboard_sync feature flag in hieraoperations/puppetproduction+9 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jan 27 2025, 3:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2025, 3:08 PM
fgiunchedi mentioned this in T384831: Repeated library panels in Grafana showing only after refresh, not on first load.Jan 27 2025, 3:08 PM
• lmata edited projects, added SRE Observability (FY2024/2025-Q4); removed SRE Observability.Jan 29 2025, 3:09 PM
andrea.denisse subscribed.Apr 23 2025, 7:27 PM
andrea.denisse changed the task status from Open to In Progress.Apr 23 2025, 8:11 PM
andrea.denisse claimed this task.

I'll test Grafana v11.6.1 (the latest release as of today).

andrea.denisse updated the task description. (Show Details)Apr 29 2025, 7:45 PM
Stashbot added a comment.Apr 29 2025, 8:02 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-29T20:02:25Z] <denisse> disabling Puppet on grafana2001 - T384841

andrea.denisse updated the task description. (Show Details)Apr 29 2025, 9:20 PM
gerritbot added a comment.May 1 2025, 6:03 PM

Change #1140523 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] grafana: Add conditional data sync via enable_sync hiera variable

https://gerrit.wikimedia.org/r/1140523

gerritbot added a project: Patch-For-Review.May 1 2025, 6:03 PM
gerritbot added a comment.May 2 2025, 4:47 PM

Change #1140760 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] grafana: Toggle data sync using feature flag

https://gerrit.wikimedia.org/r/1140760

gerritbot added a comment.May 6 2025, 4:10 PM

Change #1140760 merged by Andrea Denisse:

[operations/puppet@production] grafana: Add enable_dashboard_sync feature flag in hiera

https://gerrit.wikimedia.org/r/1140760

andrea.denisse changed the status of subtask T393439: Graphite data sources broken on grafana-next from Open to In Progress.May 6 2025, 4:11 PM
gerritbot added a comment.May 6 2025, 4:28 PM

Change #1140523 merged by Andrea Denisse:

[operations/puppet@production] grafana: Toggle data sync using feature flag

https://gerrit.wikimedia.org/r/1140523

Maintenance_bot removed a project: Patch-For-Review.May 6 2025, 4:30 PM
Stashbot added a comment.May 6 2025, 4:34 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T16:34:15Z] <denisse> enable Puppet on Grafana2001 - T384841

andrea.denisse added a comment.EditedMay 6 2025, 6:38 PM

I found an issue where after disabling dashboard sync the stunnel4 unit fails due to the ports still being bounded on both hosts to rsync but but there was no configuration in place which caused the unit to fail.
Manually restarting the unit didn't work as stunnel4 was still running even if the unit was stopped or restarted. This required me to gracefully kill the process and start the unit again.
I'll work on a patch for this.

andrea.denisse closed subtask T393439: Graphite data sources broken on grafana-next as Resolved.May 7 2025, 2:53 PM
andrea.denisse updated the task description. (Show Details)May 8 2025, 6:51 PM
gerritbot added a comment.May 8 2025, 7:28 PM

Change #1143660 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] grafana: Enable dashboard sync between hosts

https://gerrit.wikimedia.org/r/1143660

gerritbot added a project: Patch-For-Review.May 8 2025, 7:28 PM
gerritbot added a comment.May 8 2025, 7:56 PM

Change #1143660 merged by Andrea Denisse:

[operations/puppet@production] grafana: Enable dashboard sync between hosts

https://gerrit.wikimedia.org/r/1143660

Maintenance_bot removed a project: Patch-For-Review.May 8 2025, 8:30 PM
andrea.denisse closed this task as Resolved.May 8 2025, 8:34 PM
andrea.denisse updated the task description. (Show Details)
In T384841#10797493, @andrea.denisse wrote:

I found an issue where after disabling dashboard sync the stunnel4 unit fails due to the ports still being bounded on both hosts to rsync but but there was no configuration in place which caused the unit to fail.
Manually restarting the unit didn't work as stunnel4 was still running even if the unit was stopped or restarted. This required me to gracefully kill the process and start the unit again.
I'll work on a patch for this.

I'll create a separate task to track the stunnel4 issue.
The upgrade is complete now.

andrea.denisse mentioned this in T393738: The stunnel4 service requires a manual restart after disabling/enabling sync in Grafana hosts.May 8 2025, 8:38 PM
Peter mentioned this in T393883: Grafana 11.6.1 changed how it shows images in annotations.May 12 2025, 11:51 AM
Vgutierrez subscribed.May 13 2025, 9:11 AM

I've noticed some weird behavior on the LVS dashboard:

Clement_Goubert subscribed.May 13 2025, 10:08 AM

Rendering a graph as an image seems broken, the generation request ends up returning a 504 after 65289 ms.

andrea.denisse mentioned this in T394045: When selecting a DC some Grafana panels show instances for other DC.May 13 2025, 2:19 PM
In T384841#10815402, @Vgutierrez wrote:

I've noticed some weird behavior on the LVS dashboard:

Hi @Vgutierrez , thanks for the report. I can reproduce the issue (attaching a screenshot), I opened T394045 to track this.

img-2025-05-13-08-16-34.png (637×1 px, 124 KB)

andrea.denisse added a comment.May 13 2025, 2:21 PM
In T384841#10815526, @Clement_Goubert wrote:

Rendering a graph as an image seems broken, the generation request ends up returning a 504 after 65289 ms.

Hi @Clement_Goubert , thanks for the bug report. Could you please share a dashboard where you're seeing this behavior along with the steps you took for it to happen? I'd like to reproduce it to investigate it further.

Clement_Goubert added a comment.May 13 2025, 3:31 PM
In T384841#10816646, @andrea.denisse wrote:
In T384841#10815526, @Clement_Goubert wrote:

Rendering a graph as an image seems broken, the generation request ends up returning a 504 after 65289 ms.

Hi @Clement_Goubert , thanks for the bug report. Could you please share a dashboard where you're seeing this behavior along with the steps you took for it to happen? I'd like to reproduce it to investigate it further.

Sure thing.

It's not a guaranteed failure, and I think any dashboard would work, but let's take https://grafana.wikimedia.org/goto/FQCY_I-Hg?orgId=1 and https://grafana.wikimedia.org/goto/XyNV9S-HR?orgId=1 for example. Both of them have failed, and worked, in the past ten minutes.

  • Click on the graph hamburger menu
  • Select Share -> Share Link
  • Click Generate Image

Expected: An image of the graph for the selected time range is generated and displayed in the browser
Actual: Fails due to timeout. It does not happen every time, but the operation is always slow.

image.png (589×1 px, 52 KB)

Clicking "Shorten Link" doesn't change anything. The "Download Image" button is always greyed out.

andrea.denisse mentioned this in T394069: Rendering Graph's as images times out on Grafana 11.May 13 2025, 4:21 PM
In T384841#10817102, @Clement_Goubert wrote:
In T384841#10816646, @andrea.denisse wrote:
In T384841#10815526, @Clement_Goubert wrote:

Rendering a graph as an image seems broken, the generation request ends up returning a 504 after 65289 ms.

Hi @Clement_Goubert , thanks for the bug report. Could you please share a dashboard where you're seeing this behavior along with the steps you took for it to happen? I'd like to reproduce it to investigate it further.

Sure thing.

It's not a guaranteed failure, and I think any dashboard would work, but let's take https://grafana.wikimedia.org/goto/FQCY_I-Hg?orgId=1 and https://grafana.wikimedia.org/goto/XyNV9S-HR?orgId=1 for example. Both of them have failed, and worked, in the past ten minutes.

  • Click on the graph hamburger menu
  • Select Share -> Share Link
  • Click Generate Image

Expected: An image of the graph for the selected time range is generated and displayed in the browser
Actual: Fails due to timeout. It does not happen every time, but the operation is always slow.

image.png (589×1 px, 52 KB)

Clicking "Shorten Link" doesn't change anything. The "Download Image" button is always greyed out.

Thank you @Clement_Goubert , I was able to reproduce the issue and created T394069 to track this.

• lmata moved this task from Inbox to Done on the SRE Observability (FY2024/2025-Q4) board.Jun 30 2025, 4:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits