@MoritzMuehlenhoff thanks for this. Could you give an example of one metric in one dashboard that is using a diamond and what would be the equivalent in prometheus? Just a little example to get us started if it won't take too much time.

As for spotting remaining Diamond metrics, https://phabricator.wikimedia.org/P7680 contains a Paste with remaining Diamond metric references (based on a script by Timo), the patch which lists "Matched" refers to the dashboard in question. This is what I used to create the task.

As for replacing it with Prometheus, there's no good one line summary, but I suggest looking at existing dashboards for starters: On https://grafana.wikimedia.org you can login by clicking on the Wikimedia logo and once logged in, you can search for an existing dashboard, click on the title of a metric and use "Edit" (or alternatively select the wheels icon and select "View JSON"). https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats should be a useful starter, as it already uses several of the metrics in use). If you want to look at the raw metrics provided by an exporter to see what you can use, you can log into the server, figure out the port of the exporter (via Puppet) and have a look at the raw metrics via "curl localhost:$PORT/metrics". When you have more specific questions, don't hesitate to ask!

Thanks arturo! I worked on this a bit last week but didn't make a whole lot of progress.

• https://grafana.wikimedia.org/d/000000339/labs-nova-fullstack

• https://grafana.wikimedia.org/d/000000405/wmcs-api-uptimes

I would like to maintain some version of these two dashboards; I use them from time to time. If there are already updated versions someplace let me know and I'll update my links.

Sorry folks, there are a couple of things that I don't understand. The nova_fullstack_test.py script is sending collected metrics to statsd. Are we deprecating that?
I have some shadows in the different mechanisms we use for collecting/storing/visualizing metrics, I think the one that I understand the most is prometheus.

I'm trying to figure out if the dashboard https://grafana.wikimedia.org/d/000000339/labs-nova-fullstack should be just refreshed from the server names point of view, or we need to rewrite the fullstack script, or something else.

In T210850#5177775, @aborrero wrote:

Sorry folks, there are a couple of things that I don't understand. The nova_fullstack_test.py script is sending collected metrics to statsd. Are we deprecating that?
I have some shadows in the different mechanisms we use for collecting/storing/visualizing metrics, I think the one that I understand the most is prometheus.

I'm trying to figure out if the dashboard https://grafana.wikimedia.org/d/000000339/labs-nova-fullstack should be just refreshed from the server names point of view, or we need to rewrite the fullstack script, or something else.

statsd itself isn't being actively deprecated (although new "things" should use Prometheus) only Diamond is getting deprecated in this case. Diamond writes its metrics to the servers hierarchy and I see nova_fullstack_test.py also writes to servers. That is likely one of the reasons why labs-nova-fullstack dashboard came up in the audit. My suggestion is to keep the script for now but move its metrics to a different top level hierarchy so that the only producer of statsd metrics to servers is Diamond itself, does that sound good ?

In T210850#5179448, @fgiunchedi wrote:

In T210850#5177775, @aborrero wrote:

Sorry folks, there are a couple of things that I don't understand. The nova_fullstack_test.py script is sending collected metrics to statsd. Are we deprecating that?
I have some shadows in the different mechanisms we use for collecting/storing/visualizing metrics, I think the one that I understand the most is prometheus.

I'm trying to figure out if the dashboard https://grafana.wikimedia.org/d/000000339/labs-nova-fullstack should be just refreshed from the server names point of view, or we need to rewrite the fullstack script, or something else.

statsd itself isn't being actively deprecated (although new "things" should use Prometheus) only Diamond is getting deprecated in this case. Diamond writes its metrics to the servers hierarchy and I see nova_fullstack_test.py also writes to servers. That is likely one of the reasons why labs-nova-fullstack dashboard came up in the audit. My suggestion is to keep the script for now but move its metrics to a different top level hierarchy so that the only producer of statsd metrics to servers is Diamond itself, does that sound good ?

Or simply ignore the hierarchy, with Diamond being gone for good it won't matter soon ? The reason it was listed in this task is because it referenced the labnet* servers which are not gone. If the dashboard is still useful from my pov it also seems fine to simply update it to use the relevant metrics from cloudnet* servers.

In T210850#5179495, @MoritzMuehlenhoff wrote:

In T210850#5179448, @fgiunchedi wrote:

In T210850#5177775, @aborrero wrote:

Sorry folks, there are a couple of things that I don't understand. The nova_fullstack_test.py script is sending collected metrics to statsd. Are we deprecating that?
I have some shadows in the different mechanisms we use for collecting/storing/visualizing metrics, I think the one that I understand the most is prometheus.

I'm trying to figure out if the dashboard https://grafana.wikimedia.org/d/000000339/labs-nova-fullstack should be just refreshed from the server names point of view, or we need to rewrite the fullstack script, or something else.

statsd itself isn't being actively deprecated (although new "things" should use Prometheus) only Diamond is getting deprecated in this case. Diamond writes its metrics to the servers hierarchy and I see nova_fullstack_test.py also writes to servers. That is likely one of the reasons why labs-nova-fullstack dashboard came up in the audit. My suggestion is to keep the script for now but move its metrics to a different top level hierarchy so that the only producer of statsd metrics to servers is Diamond itself, does that sound good ?

Or simply ignore the hierarchy, with Diamond being gone for good it won't matter soon ? The reason it was listed in this task is because it referenced the labnet* servers which are not gone. If the dashboard is still useful from my pov it also seems fine to simply update it to use the relevant metrics from cloudnet* servers.

I see! I'd still opt to move out of servers all metrics that are not generated by Diamond so we can safely say that nothing is writing stats there anymore once Diamond is gone.

Ok! I'm fine renaming the metrics. I would need a suggestion though :-)

cloudvps.novafullstack.*?

In T210850#5179570, @aborrero wrote:

Ok! I'm fine renaming the metrics. I would need a suggestion though :-)

cloudvps.novafullstack.*?

Looks good to me

In T210850#5179976, @MoritzMuehlenhoff wrote:

In T210850#5179570, @aborrero wrote:

Ok! I'm fine renaming the metrics. I would need a suggestion though :-)

cloudvps.novafullstack.*?

Looks good to me

same, +1

Change 520636 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-fullstack: rename statsd metrics to cloudvps.novafullstack

https://gerrit.wikimedia.org/r/520636

Change 520636 merged by Andrew Bogott:
[operations/puppet@production] nova-fullstack: rename statsd metrics to cloudvps.novafullstack

https://gerrit.wikimedia.org/r/520636

I've made a new dashboard, https://grafana.wikimedia.org/d/ebJoA6VWz/nova-fullstack -- once I'm convinced that it's doing what I expect I'll delete the older labs-nova-fullstack board.

Cole fixed the remaining dashboards. Andrew, can you have a final look whether everything works as expected, then we can close the task?