Page MenuHomePhabricator
Create Task
Maniphest T436045

Grafana no longer includes many datasources but adds them as plugins
Closed, ResolvedPublic
Actions

Description

As of version 13.1.1 the Debian Grafana package no longer includes Prometheus as built-in data source but adds it as a separate plugin [1] [2], this means that it requires to be installed as part of the grafana-plugins repository.

Logs:

Aug 26 03:48:27 grafana2001 grafana[1211199]: logger=context userId=709 orgId=1 uname=denisse t=2026-08-26T03:48:27.842393423Z level=error msg="Could not find plugin definition for data source" datasource_type=prometheus

Looking deeper at error logs it seems to be affecting other datasources that are now installed as plugins:

PluginRequiredEnabledURL
elasticsearchNoYeshttps://grafana.com/grafana/plugins/elasticsearch/
grafana-advisor-appNoNohttps://grafana.com/grafana/plugins/grafana-advisor-app/
grafana-exploretraces-appNoNohttps://grafana.com/grafana/plugins/grafana-exploretraces-app/
grafana-lokiexplore-appNoNohttps://grafana.com/grafana/plugins/grafana-lokiexplore-app/
grafana-metricsdrilldown-appYesNohttps://grafana.com/grafana/plugins/grafana-metricsdrilldown-app/
grafana-postgresql-datasourceNoNohttps://grafana.com/grafana/plugins/grafana-postgresql-datasource/
grafana-pyroscope-appNoNohttps://grafana.com/grafana/plugins/grafana-pyroscope-app/
grafana-pyroscope-datasourceNoNohttps://grafana.com/grafana/plugins/grafana-pyroscope-datasource/
influxdbNoNohttps://grafana.com/grafana/plugins/influxdb/
jaegerNoYeshttps://grafana.com/grafana/plugins/jaeger/
lokiYesYeshttps://grafana.com/grafana/plugins/loki/
mssqlNoNohttps://grafana.com/grafana/plugins/mssql/
mysqlNoNohttps://grafana.com/grafana/plugins/mysql/
opentsdbNoNohttps://grafana.com/grafana/plugins/opentsdb/
prometheusYesYeshttps://grafana.com/grafana/plugins/prometheus/
stackdriverNoNohttps://grafana.com/grafana/plugins/stackdriver/
tempoNoNohttps://grafana.com/grafana/plugins/tempo/
zipkinNoNohttps://grafana.com/grafana/plugins/zipkin/

The plugins that we don't need and are not installed should be added to the disable_plugins section so Grafana doesn't try to load them to prevent it from generating useless logs.

[1] https://github.com/grafana/grafana/pull/123035
[2] https://grafana.com/grafana/plugins/prometheus/changelog/
[3] https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#disable_plugins

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
grafana: Disable loading of not installed core pluginsAndrea Denisseoperations/puppetproduction+32 -2
grafana: Disable automatic updates for pluginsAndrea Denisseoperations/puppetproduction+1 -0
Add grafana-prometheus-datasourceAndrea Denisseoperations/debs/grafana-pluginsmaster+16 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Add grafana-metricsdrilldown-app 2.5.1repos/sre/grafana-plugins!4denisseT436045-2main
Customize query in GitLab

Related Objects

Event Timeline

andrea.denisse created this task.Aug 26 2026, 4:07 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2026, 4:07 AM
andrea.denisse changed the task status from Open to In Progress.Aug 26 2026, 4:08 AM
andrea.denisse triaged this task as Unbreak Now! priority.
andrea.denisse renamed this task from Grafana no longer includes the Prometheus datasource but adds it as a plugin to Grafana no longer includes many datasources but adds them as plugins.Aug 26 2026, 4:17 AM
andrea.denisse updated the task description. (Show Details)
gerritbot added a comment.Aug 26 2026, 5:32 AM

Change #1329416 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] grafana: Disable automatic updates for plugins

https://gerrit.wikimedia.org/r/1329416

gerritbot added a project: Patch-For-Review.Aug 26 2026, 5:32 AM
andrea.denisse updated the task description. (Show Details)Aug 26 2026, 5:38 AM
andrea.denisse updated the task description. (Show Details)
MoritzMuehlenhoff subscribed.Aug 26 2026, 6:20 AM

Wow, that's a breaking change they make from 13.1.0 to 13.1.1? What's even the purpose of their release branches, I have to wonder..

gerritbot added a comment.Aug 26 2026, 7:22 AM

Change #1329426 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/debs/grafana-plugins@master] Add grafana-prometheus-datasource

https://gerrit.wikimedia.org/r/1329426

gerritbot added a comment.Aug 26 2026, 7:49 AM

Change #1329426 merged by Andrea Denisse:

[operations/debs/grafana-plugins@master] Add grafana-prometheus-datasource

https://gerrit.wikimedia.org/r/1329426

andrea.denisse mentioned this in rOBGP97d813fb9264: Add grafana-prometheus-datasource.Aug 26 2026, 7:59 AM
andrea.denisse mentioned this in T436056: Move grafana-plugins from Gerrit to Gitlab to ease building the package.Aug 26 2026, 8:00 AM
andrea.denisse added a comment.Aug 26 2026, 8:18 AM

We currently have the following data sources enabled on Grafana: Prometheus, Elastic, Graphite, Jaeger, JSON API, Loki, Simple JSON. Among these, the Elastic one is broken as it was also moved to a plugin [1] but it doesn't seem like we're using it anyway so we might disable it. There's an OpenSearch datasource plugin [2] that we might use for T422499.

[1] https://grafana.com/grafana/plugins/elasticsearch/
[2] https://grafana.com/grafana/plugins/grafana-opensearch-datasource/

andrea.denisse updated the task description. (Show Details)Aug 26 2026, 8:38 AM
gerritbot added a comment.Aug 26 2026, 9:37 AM

Change #1329517 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] grafana: Disable loading of not installed core plugins

https://gerrit.wikimedia.org/r/1329517

andrea.denisse updated the task description. (Show Details)Aug 26 2026, 9:49 AM
colewhite updated the task description. (Show Details)Aug 26 2026, 2:25 PM
colewhite updated the task description. (Show Details)Aug 26 2026, 2:30 PM
hnowlan moved this task from Inbox to FY2026/2027-Q1 on the SRE Observability board.Aug 26 2026, 5:22 PM
andrea.denisse lowered the priority of this task from Unbreak Now! to Low.Aug 26 2026, 5:34 PM

Changing priority to low as the critical datasource is already in the grafana-plugins package.

andrea.denisse changed the task status from In Progress to Open.Aug 26 2026, 5:34 PM
andrea.denisse updated the task description. (Show Details)Aug 26 2026, 5:37 PM
andrea.denisse added a comment.Aug 26 2026, 5:40 PM

Among the listed plugins the only ones that we need to add to the grafana-plugins package are loki and grafana-metricsdrilldown-app. I'll add them once T436056 is complete as that would also allow us to test the GitLab CI pipeline to build the Debian package.

hnowlan subscribed.Aug 27 2026, 10:32 AM
gerritbot added a comment.Aug 27 2026, 3:32 PM

Change #1329416 merged by Andrea Denisse:

[operations/puppet@production] grafana: Disable automatic updates for plugins

https://gerrit.wikimedia.org/r/1329416

gerritbot added a comment.Aug 27 2026, 3:33 PM

Change #1329517 merged by Andrea Denisse:

[operations/puppet@production] grafana: Disable loading of not installed core plugins

https://gerrit.wikimedia.org/r/1329517

andrea.denisse added a comment.Sep 1 2026, 3:22 AM

The Network Weathermap NG [1] plugin was suggested on IRC as a potentially useful plugin.

[1] https://github.com/allamiro/grafana-network-weathermap-ng

hnowlan raised the priority of this task from Low to Medium.Mon, Sep 7, 5:22 PM
hnowlan mentioned this in T406478: Scap logs on Grafana dashboards are broken.

Increasing the priority as the loki plugin's removal has broken annotations on Grafana.

andrea.denisse added a comment.Mon, Sep 7, 7:18 PM
In T436045#12295665, @hnowlan wrote:

Increasing the priority as the loki plugin's removal has broken annotations on Grafana.

I'll add it.

CodeReviewBot added a comment.Tue, Sep 8, 4:34 AM

denisse opened https://gitlab.wikimedia.org/repos/sre/grafana-plugins/-/merge_requests/3

Add loki plugin v13.1.0

CodeReviewBot added a comment.Tue, Sep 8, 4:47 AM

denisse merged https://gitlab.wikimedia.org/repos/sre/grafana-plugins/-/merge_requests/3

Add loki plugin v13.1.0

andrea.denisse lowered the priority of this task from Medium to Low.Tue, Sep 8, 5:32 AM

Added loki, had to build it manually as it's not yet part of the trusted runners repo [1].
Will add grafana-metricsdrilldown-app through the GitLab CI pipeline to test building the package from it as part of T436056.

[1] https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/185

CodeReviewBot added a comment.Tue, Sep 8, 5:46 AM

denisse opened https://gitlab.wikimedia.org/repos/sre/grafana-plugins/-/merge_requests/4

Add grafana-metricsdrilldown-app 2.5.1

CodeReviewBot added a comment.Tue, Sep 8, 7:05 AM

denisse merged https://gitlab.wikimedia.org/repos/sre/grafana-plugins/-/merge_requests/4

Add grafana-metricsdrilldown-app 2.5.1

andrea.denisse closed this task as Resolved.EditedTue, Sep 8, 8:03 AM
In T436045#12296151, @andrea.denisse wrote:

Added loki, had to build it manually as it's not yet part of the trusted runners repo [1].
Will add grafana-metricsdrilldown-app through the GitLab CI pipeline to test building the package from it as part of T436056.

[1] https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/185

Couldn't wait for the CI pipeline, added it and packaged the regular way.
I'll try the pipeline with tamirsuliman-weathermap-panel 1.6.12 [1] as part of T436075.

Task is complete now.

[1] https://gitlab.wikimedia.org/repos/sre/grafana-plugins/-/merge_requests/11

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits