Alex said we should look a bit at grafana usage first.

Change 224554 had a related patch set uploaded (by Dzahn):
move grafana from zirconium to netmon1001

https://gerrit.wikimedia.org/r/224554

@ori can it move to netmon1001 maybe?

@Dzahn -- Grafana v2 (which we aren't running yet) has a backend component that is written in Go and which has not been packaged for Debian. Apart from its own backend, Grafana does not depend on any other services. The fetching of data from Graphite is done entirely on the client by making calls to the Graphite API directly. So it's a very good use-case for a virtual machine. Would that be much harder than netmon1001? I don't know what sort of work is involved.

@ori thank you for the detailed reply! So it sounds a VM makes sense indeed in this case. It would involve requesting a new VM (similar to T105507) and then it's pretty much like physical hardware, so add to DHCP, DNS, install an OS via PXE boot, sign puppet certs, apply roles etc. I would be happy to do all the setup part to when the service role gets applied but don't know too much about grafana itself

Change 224554 abandoned by Dzahn:
move grafana from zirconium to netmon1001

https://gerrit.wikimedia.org/r/224554

move it to "krypton" instead (existing VM)

Change 230650 had a related patch set uploaded (by Dzahn):
grafana: prepare for Apache 2.4 syntax

https://gerrit.wikimedia.org/r/230650

Change 230660 had a related patch set uploaded (by Dzahn):
misc-web: switch grafana to backend krypton

https://gerrit.wikimedia.org/r/230660

Change 230650 merged by Dzahn:
grafana: prepare for Apache 2.4 syntax

https://gerrit.wikimedia.org/r/230650

simply adding the role to krypton (jessie, Apache 2.4 as opposed to zirconium) currently fails in relation to grafana trying to use Apache mod proxy_balancer

revert for now:

https://gerrit.wikimedia.org/r/#/c/230664/

currently breaks Apache 2.4 setup

actually this is a good example for not mixing multiple services on the same VM,

if grafana had gotten its own VM it would not have affected other misc. services who don't use mod proxy_balancer but share the host krypton (iegreview tool, wikimania scholarship tool)

that was fixed by https://gerrit.wikimedia.org/r/#/c/230682/

so the role is applied on krypton

but "Could not contact Elasticsearch. Please ensure that Elasticsearch is reachable from your browser."

when proxying to node krypton.eqiad.wmnet via bastion at least

10:08 <mutante> "Could not contact Elasticsearch. Please ensure that Elasticsearch is reachable from your browser."

10:09 <mutante> and the way i tested is:
10:09 <mutante> ssh -D 8080 iron.wikimedia.org
10:09 <mutante> put localhost:8080 as SOCKS proxy in browser
10:09 <mutante> and open http://krypton.eqiad.wmnet/

10:15 <godog> ah yeah, the elastic warning could be because it can't load Request URL:http://grafana.wikimedia.org/grafana-dashboards/dashboard/_search
10:15 <godog> through the socks proxy

10:17 <mutante> i just did that but still get the same message

10:24 <mutante> Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at

http://grafana.wikimedia.org/grafana-dashboards/dashboard/home. (Reason: CORS header 'Access-Control-Allow-Origin' missing).

10:25 <mutante> i need to hack my /etc/hosts locally
10:25 <mutante> and then access it as grafana.wm
10:30 <mutante> and then i get a different error, "dashboard not found" but not the previous one

10:33 <mutante> it's related to mod_proxy, figuring it out
10:34 <mutante> 3945] AH01144: No protocol handler was valid for the URL /grafana-dashboards/dashboard/home

all the modules are loaded: mod_proxy (proxy_balancer depends on it), proxy_balancer and lbmethod_byrequests

In T105008#1528115, @Dzahn wrote:

10:33 <mutante> it's related to mod_proxy, figuring it out
10:34 <mutante> 3945] AH01144: No protocol handler was valid for the URL /grafana-dashboards/dashboard/home

all the modules are loaded: mod_proxy (proxy_balancer depends on it), proxy_balancer and lbmethod_byrequests

Turned out after some debugging that we were missing to load "proxy_http" in addition to the above. It was somehow loaded on zirconium with Apache 2.2 but not on krypton with Apache 2.4.

Change 230824 had a related patch set uploaded (by Dzahn):
grafana: needs to load Apache mod_proxy_http too

https://gerrit.wikimedia.org/r/230824

Change 230824 merged by Dzahn:
grafana: needs to load Apache mod_proxy_http too

https://gerrit.wikimedia.org/r/230824

Change 230660 merged by Dzahn:
misc-web: switch grafana to backend krypton

https://gerrit.wikimedia.org/r/230660