Page MenuHomePhabricator

'provision' Grafana's datasources as YAML in puppet
Closed, ResolvedPublic

Description

Grafana can now read the definition of its datasources from files: http://docs.grafana.org/administration/provisioning/#datasources

This makes them read-only in the UI (probably a good idea anyway), and will allow us to know the connections between our systems better via the usual git grep and other inspection of the puppet repo (would have caught T211712 before it happened).

Event Timeline

CDanis triaged this task as Normal priority.Dec 14 2018, 2:51 PM
CDanis created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2018, 2:51 PM

We have a lot of datasources. Creating JSON for all of them by hand seems tedious.

I filed a FR upstream asking for easy export of existing data sources as JSON, which was duped into https://github.com/grafana/grafana/issues/11495, which seems like it probably won't get love any time soon (last update, in June, was it being removed from the 5.2 milestone).

So I guess I'll look at writing a script myself.

Volans added a subscriber: Volans.EditedDec 19 2018, 7:40 PM

@CDanis I have a quick and dirty solution that seems seems to do 95% of the work, given that is a one-shot I think it might be near-usable, just my 2c at it, feel free to ignore ;)

From sqlite:

select printf(' - name: %s\n   type: %s\n   access: %s\n   orgId: %s\n   url: %s\n   database: %s\n   isDefault: %s\n   editable: false\n   withCredentials: false\n   version: %s\n', name, type, access, org_id, url, database, (CASE WHEN  is_default == 1 THEN 'true' ELSE 'false' END), version) from data_source;

Then pass the output to echo -e. The only missing thing should be the json_data field, but only 2 datasources have it, so I guess doing it manually is quicker.

N.B. I've skipped on purpose all the fields that are empty for all entries, but given that it seems to me that they are set to empty string and not NULL the above query could easily be expanded to add them too if some of them are mandatory in the YAML.

CDanis added a comment.EditedDec 19 2018, 8:23 PM

According to the docs, the only required fields are name, type, and access. So we should be okay there.

I wound up embracing and extending your solution...

cdanis@grafana1001.eqiad.wmnet ~ % cat query.txt                                                                                       
select printf('/bin/echo -e '' - name: %s\n   type: %s\n   access: %s\n   orgId: %s\n   url: %s\n   database: %s\n   isDefault: %s\n   editable: false\n   withCredentials: false\n   version: %s\n%s''', name, type, access, org_id, url, database,
(CASE WHEN is_default == 1 THEN 'true' ELSE 'false' END),
version,
(CASE WHEN json_data == X'7b7d' THEN '' ELSE '# FIXME json_data\n' END)) from data_source;
cdanis@grafana1001.eqiad.wmnet ~ % (echo 'datasources:' && sqlite3 /var/lib/grafana/grafana.db < query.txt| /bin/sh) > datasources.yaml

Change 480833 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] profile::grafana::production: datasources as YAML

https://gerrit.wikimedia.org/r/480833

Change 480833 merged by CDanis:
[operations/puppet@production] profile::grafana::production: datasources as YAML

https://gerrit.wikimedia.org/r/480833

Seems like it's working -- Grafana now displays "This datasource was added by config and cannot be modified using the UI. Please contact your server admin to update this datasource."

Have inspected a variety of consoles and they seem to be happy.

CDanis closed this task as Resolved.Dec 19 2018, 10:43 PM