Page MenuHomePhabricator

Upgrade grafana to 6.4.4
Closed, ResolvedPublic

Description

Switchover to Grafana 6 planned for Monday Nov 25.

Grafana 6 adds a bunch of cool new stuff:

  • "Explore", a UI for playing around with monitoring data, and for exploring what metrics are in Prometheus. It looks ideal for either constructing queries for a new console, or for doing incident response across multiple systems (and then linking in your postmortem). https://grafana.com/docs/features/explore/
  • "ad hoc filtering for Prometheus", which looks like a big improvement on the template variable stuff we do on many consoles right now. https://grafana.com/docs/guides/whats-new-in-v6-1/
  • better panel editor UI

Things to watch out for per https://grafana.com/docs/installation/upgrading/#upgrading-to-v6-0

  • check for any text panels with embedded <script> tags
    • figure out what to do re: the frontpage panel that has this; also the puppet panel that iframes to puppetdb is broken -- maybe we should just disable sanitization? https://github.com/grafana/grafana/issues/15392
    • yes, let's just disable sanitization

Plan is similar to last time:

  • create new Ganeti VM grafana1002.eqiad.wmnet -- will attempt using the new automation for such, and probably also try using buster
  • point grafana-next.wikimedia.org to that host
  • copy a snapshot of the database and ask the same groups as in T210416 to test
  • pick a time to make -next the new normal (should be just a few minutes of readonly)

Fundraising firewall should not be a concern this time as they have their own grafana now.

It's also worth looking at testing the upgrade on grafana-labs if they're interested

Details

Event Timeline

CDanis created this task.Apr 12 2019, 4:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2019, 4:55 PM
CDanis updated the task description. (Show Details)Apr 15 2019, 3:15 PM
CDanis moved this task from Inbox to Up next on the observability board.Apr 29 2019, 3:16 PM
Peter added a subscriber: Peter.Jun 14 2019, 5:05 AM
Peter added a comment.Jun 14 2019, 5:10 AM

If we could upgrade to 6.2.x that would be great. I've been using it for my projects for a while and the lazy loading of panels out of view really makes large dashboards working better (and faster). We have a couple of dashboards in the performance team that are really slow.

CDanis renamed this task from Upgrade grafana to 6.1 to Upgrade grafana to 6.x.Jun 24 2019, 3:12 PM
CDanis triaged this task as Low priority.Jul 22 2019, 3:15 PM

so far work on dbctl has taken priority but I should have time for this in another few weeks.

Change 539894 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] grafana1002: new VM for grafana 6.x

https://gerrit.wikimedia.org/r/539894

Change 539894 merged by CDanis:
[operations/dns@master] grafana1002: new VM for grafana 6.x

https://gerrit.wikimedia.org/r/539894

Change 539913 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana1002: autoinstall/netboot/role data

https://gerrit.wikimedia.org/r/539913

Change 539913 merged by CDanis:
[operations/puppet@production] grafana1002: autoinstall/netboot/role data

https://gerrit.wikimedia.org/r/539913

Change 543861 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana1002: answer for grafana-beta.wikimedia.org

https://gerrit.wikimedia.org/r/543861

Change 543861 merged by CDanis:
[operations/puppet@production] grafana1002: answer for grafana-beta.wikimedia.org

https://gerrit.wikimedia.org/r/543861

Change 543862 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana-beta.wikimedia.org: point to grafana1002

https://gerrit.wikimedia.org/r/543862

Change 543862 merged by CDanis:
[operations/puppet@production] grafana-beta.wikimedia.org: point to grafana1002

https://gerrit.wikimedia.org/r/543862

Change 544023 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] trafficserver: actually map grafana-beta to grafana1002

https://gerrit.wikimedia.org/r/544023

Change 544023 merged by CDanis:
[operations/puppet@production] trafficserver: actually map grafana-beta to grafana1002

https://gerrit.wikimedia.org/r/544023

Change 544029 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] trafficserver: add never-cache for grafana1002

https://gerrit.wikimedia.org/r/544029

Change 544029 merged by CDanis:
[operations/puppet@production] trafficserver: add never-cache for grafana1002

https://gerrit.wikimedia.org/r/544029

CDanis updated the task description. (Show Details)Oct 21 2019, 2:01 PM
fgiunchedi moved this task from Up next to In progress on the observability board.Oct 28 2019, 2:31 PM
CDanis updated the task description. (Show Details)Oct 28 2019, 3:26 PM

Change 551572 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana1002: disable HTML sanitization in panels

https://gerrit.wikimedia.org/r/551572

Change 551572 merged by CDanis:
[operations/puppet@production] grafana1002: disable HTML sanitization in panels

https://gerrit.wikimedia.org/r/551572

Change 551616 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana1002: security.cookie_secure per release notes

https://gerrit.wikimedia.org/r/551616

Change 551616 merged by CDanis:
[operations/puppet@production] grafana1002: security.cookie_secure per release notes

https://gerrit.wikimedia.org/r/551616

CDanis renamed this task from Upgrade grafana to 6.x to Upgrade grafana to 6.4.4.Mon, Nov 18, 7:38 PM
CDanis updated the task description. (Show Details)

Change 551624 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] graphite: add grafana-beta to cors_origins

https://gerrit.wikimedia.org/r/551624

Change 551624 merged by CDanis:
[operations/puppet@production] graphite: add grafana-beta to cors_origins

https://gerrit.wikimedia.org/r/551624

Change 551633 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] grafana-beta: rename to grafana-next

https://gerrit.wikimedia.org/r/551633

Change 551633 merged by CDanis:
[operations/dns@master] grafana-beta: rename to grafana-next

https://gerrit.wikimedia.org/r/551633

Change 551636 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana-beta: rename to grafana-next

https://gerrit.wikimedia.org/r/551636

Change 551636 merged by CDanis:
[operations/puppet@production] grafana-beta: rename to grafana-next

https://gerrit.wikimedia.org/r/551636

Change 551653 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana: cert: add SANs grafana-next and grafana1002

https://gerrit.wikimedia.org/r/551653

Change 551653 merged by CDanis:
[operations/puppet@production] grafana: cert: add SANs grafana-next and grafana1002

https://gerrit.wikimedia.org/r/551653

Change 551660 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana: quickdatacopy 1001->1002 for plugins and for pngs

https://gerrit.wikimedia.org/r/551660

Change 551660 merged by CDanis:
[operations/puppet@production] grafana: quickdatacopy 1001->1002 for plugins and for pngs

https://gerrit.wikimedia.org/r/551660

upgraded the pie chart plugin to a recent version that actually works with 6.x:

❌cdanis@grafana1002.eqiad.wmnet ~ 🕕🍺 sudo http_proxy=http://webproxy.eqiad.wmnet:8080 grafana-cli plugins install grafana-piechart-panel

now on version 1.3.9

CDanis updated the task description. (Show Details)Mon, Nov 18, 11:17 PM

I've heard no complaints, and can verify from the logs that it's seen at least some testing by others. Planning to do a final snapshot and move traffic over on Monday afternoon my time.

CDanis updated the task description. (Show Details)Thu, Nov 21, 6:33 PM

Change 552876 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana1002: is just grafana.wm.o now

https://gerrit.wikimedia.org/r/552876

Change 552876 merged by CDanis:
[operations/puppet@production] grafana1002: is just grafana.wm.o now

https://gerrit.wikimedia.org/r/552876

Change 552879 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana1002: is now the server for grafana.wm.o

https://gerrit.wikimedia.org/r/552879

Mentioned in SAL (#wikimedia-operations) [2019-11-25T19:11:36Z] <cdanis> copied snapshot of database from grafana1001 to grafana1002 T220838

Mentioned in SAL (#wikimedia-operations) [2019-11-25T19:13:41Z] <cdanis> restarted grafana-server on grafana1002 T220838

Change 552879 merged by CDanis:
[operations/puppet@production] grafana1002: is now the server for grafana.wm.o

https://gerrit.wikimedia.org/r/552879

CDanis closed this task as Resolved.Mon, Nov 25, 8:01 PM

Grafana 6.4.4 is now in use at https://grafana.wikimedia.org.

Peter awarded a token.Mon, Nov 25, 8:34 PM

Change 552933 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana: remove rsyncs after complete migration

https://gerrit.wikimedia.org/r/552933

Change 552933 merged by CDanis:
[operations/puppet@production] grafana: remove rsyncs after complete migration

https://gerrit.wikimedia.org/r/552933

Change 552934 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana: first, ensure=>absent the obsolete rsyncs

https://gerrit.wikimedia.org/r/552934

Change 552934 merged by CDanis:
[operations/puppet@production] grafana: first, ensure=>absent the obsolete rsyncs

https://gerrit.wikimedia.org/r/552934

Change 552936 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] grafana: okay, *now* remove the obsolete rsyncs

https://gerrit.wikimedia.org/r/552936

Change 552936 merged by CDanis:
[operations/puppet@production] grafana: okay, *now* remove the obsolete rsyncs

https://gerrit.wikimedia.org/r/552936