Page MenuHomePhabricator

Upgrade to Grafana 7
Closed, ResolvedPublic

Description

As per title, we shall upgrade to Grafana 7.

Grafana 7.1.3 is available at https://grafana-next.wikimedia.org for testing.

Please note that dashboards are currently not synced from https://grafana.wikimedia.org to not overwrite the Grafana 7 DB schema.

Upgrade steps

  • Test (with database sync disabled) Grafana 7 on https://grafana-next.wikimedia.org
  • Manually sync the database once more before transition
  • Point grafana.w.o to the same place as grafana-next.w.o and announce the change, wait for feedback. Note that ATM this also means failing over Grafana to codfw
  • Once things are working ok, upgrade the (now standby) host that was running Grafana 6 to Grafana 7
  • Turn back on database sync
  • Fail back to eqiad

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 29 2020, 3:37 PM
fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 31 2020, 12:33 PM

Change 618069 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] grafana: temp disable grafana db sync ahead of upgrade

https://gerrit.wikimedia.org/r/618069

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Aug 3 2020, 4:20 PM

I started looking into this. The upgrade itself seems to work fine, however we'll have to ship the https://github.com/grafana/grafana-image-renderer (GIR) if we want to be able to export/share static images. GIR installed via grafana-cli plugins install grafana-image-renderer means having an embedded copy of chromium (leaving aside the fact that we'll have to ship the plugin offline, packaged somehow) which I don't think we really want to do.

Other options as I see them:

  1. Use GIR releases without chromium (but already built), e.g. at https://github.com/grafana/grafana-image-renderer/releases/download/v2.0.0/plugin-linux-x64-glibc-no-chromium.zip (and packaged)
  2. Build GIR without chromium ourselves (haven't tried yet) (and packaged)

Soon we'll have Chromium embedded in bootloaders as well...

What is the difference of GIR releases without chromium compared to what grafana-cli would fetch/install? Does it lose some functionablity?

One option for deploying the Grafana plugin(s) (in case more are expected) is a similar approach to the Elastic plugins maybe?

AFAICT there's no difference in the plugin's functionality, GIR has support for using the system's chromium instead of the embedded one (assuming the chromium's version are compatible)

Yes we could ship the plugin offline similarly to what we do with Elastic; we have one more plugin ATM that we ship via scap (operations/software/grafana/simple-json-datasource) although I think switching to Debian packages and ship all plugins would be more convenient

Using the chromium package as packaged in Debian sounds good to me, the only risk is that new versions potentially break the headless setup (but in that case Grafana would also need to update their packages, so seems manageable).

I gave the Debian-shipped Chromium a try but to not avail, will be using the plugin-provided Chromium instead

Change 618953 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/grafana-plugins@master] Debian packaging for Grafana plugins

https://gerrit.wikimedia.org/r/618953

Change 618069 merged by Filippo Giunchedi:
[operations/puppet@production] grafana: temp disable grafana db sync ahead of upgrade

https://gerrit.wikimedia.org/r/618069

fgiunchedi updated the task description. (Show Details)Aug 11 2020, 10:34 AM
fgiunchedi updated the task description. (Show Details)

Change 619451 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: switch Grafana plugins to Debian package

https://gerrit.wikimedia.org/r/619451

Ladsgroup added a subscriber: Ladsgroup.

Change 618953 merged by Filippo Giunchedi:
[operations/debs/grafana-plugins@master] Debian packaging for Grafana plugins

https://gerrit.wikimedia.org/r/618953

fgiunchedi updated the task description. (Show Details)Aug 17 2020, 9:30 AM

Change 620663 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: switch grafana.w.o to Grafana 7

https://gerrit.wikimedia.org/r/620663

Peter added a subscriber: Peter.Aug 18 2020, 9:08 AM

cc cloud-services-team, as heads up I'd like to upgrade https://grafana-labs.wikimedia.org/ (i.e. cloudmetrics) to Grafana 7 ASAP.

Unless there are objections? There's a production instance available at https://grafana-next.wikimedia.org which so far hasn't yield any complaints

Change 619451 merged by Filippo Giunchedi:
[operations/puppet@production] profile: switch Grafana plugins to Debian package

https://gerrit.wikimedia.org/r/619451

Mentioned in SAL (#wikimedia-operations) [2020-08-19T07:46:17Z] <godog> upgrade to grafana 7 on cloudmetrics hosts - T259143

Change 621197 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: disable panel html sanitization for grafana-labs

https://gerrit.wikimedia.org/r/621197

Change 620663 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: switch grafana.w.o to Grafana 7

https://gerrit.wikimedia.org/r/620663

Mentioned in SAL (#wikimedia-operations) [2020-08-19T08:20:29Z] <godog> switch grafana.w.o to grafana 7 in codfw - T259143

fgiunchedi updated the task description. (Show Details)Aug 19 2020, 8:28 AM
Peter added a comment.Aug 19 2020, 9:36 AM

Woho, the new 7 looks great. One thing: before the plugin grafana-piechart-panel was installed by default, that seems to be missing now. You can see that here (scroll down): https://grafana.wikimedia.org/d/cZgMg49Wz/performance-summary?orgId=1

Woho, the new 7 looks great. One thing: before the plugin grafana-piechart-panel was installed by default, that seems to be missing now. You can see that here (scroll down): https://grafana.wikimedia.org/d/cZgMg49Wz/performance-summary?orgId=1

Interesting, I'll add the plugin to our grafana-plugins package. I can't find traces of that plugin in puppet, I'm guessing it was installed manually

Change 621255 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/grafana-plugins@master] Add grafana-piechart-panel

https://gerrit.wikimedia.org/r/621255

Change 621255 merged by Filippo Giunchedi:
[operations/debs/grafana-plugins@master] Add grafana-piechart-panel

https://gerrit.wikimedia.org/r/621255

Woho, the new 7 looks great. One thing: before the plugin grafana-piechart-panel was installed by default, that seems to be missing now. You can see that here (scroll down): https://grafana.wikimedia.org/d/cZgMg49Wz/performance-summary?orgId=1

Interesting, I'll add the plugin to our grafana-plugins package. I can't find traces of that plugin in puppet, I'm guessing it was installed manually

Package upgraded, I can see the piecharts now

Change 621472 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] grafana: remove deprecated settings

https://gerrit.wikimedia.org/r/621472

Change 621197 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: disable panel html sanitization for grafana-labs

https://gerrit.wikimedia.org/r/621197

Change 621472 merged by Filippo Giunchedi:
[operations/puppet@production] grafana: remove deprecated settings

https://gerrit.wikimedia.org/r/621472

Change 622610 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: re-enable grafana rsync codfw->eqiad

https://gerrit.wikimedia.org/r/622610

Change 622610 merged by Filippo Giunchedi:
[operations/puppet@production] profile: re-enable grafana rsync codfw->eqiad

https://gerrit.wikimedia.org/r/622610

fgiunchedi updated the task description. (Show Details)Aug 27 2020, 9:03 AM
fgiunchedi changed the task status from Open to Stalled.Aug 27 2020, 1:32 PM

Stalling until we're switching back to eqiad

fgiunchedi moved this task from Doing to Other on the User-fgiunchedi board.Aug 27 2020, 1:33 PM
fgiunchedi moved this task from Other to Doing on the User-fgiunchedi board.Sep 15 2020, 8:49 AM

Change 627515 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] grafana: switch to eqiad

https://gerrit.wikimedia.org/r/627515

Change 627515 merged by Filippo Giunchedi:
[operations/puppet@production] grafana: switch to eqiad

https://gerrit.wikimedia.org/r/627515

Mentioned in SAL (#wikimedia-operations) [2020-09-15T14:53:41Z] <godog> switch grafana to eqiad - T259143

fgiunchedi closed this task as Resolved.Sep 15 2020, 2:58 PM
fgiunchedi claimed this task.

grafana.w.o is back in eqiad, resolving