Today during graphite failover (T247963) I noticed superset uses statsd.eqiad.wmnet via gunicorn and superset had to be restarted to pick up dns changes. AFAICS superset itself doesn't emit metrics via statsd but only gunicorn does (?).
Please consider migrating to prometheus metrics via prometheus-statsd-exporter (or possibly turning off statsd if gunicorn metrics are not used/useful)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T228380 Tech debt: sunsetting of Graphite (part 1) | |||
| Open | colewhite | T205870 Fully migrate producers off statsd | |||
| Open | None | T293761 statsd and gunicorn metrics for superset |
Thanks for the heads-up @fgiunchedi
I've just had a quick look and, as far as I can see, the stats from gunicorn haven't been used to date.
The only graph on the Superset dashboard in Grafana was from the apache exporter, operating as the reverse proxy for Superset.
I have just added a quick graph of the Response times from gunicorn and I think that these stats could be handy in tracking down performance problems and timeouts in particular.
So my preference would probably be to migrate them to prometheus-statsd-exporter and possibly rename them, so that they are named something more relevant to Superset.
I've had a quick search for other ways to instrument Superset and gunicorn, but haven't turned up anything yet. We are hoping to spend some energy on improving the the Superset user experience though, so these metrics could have some value.
(cc @razzi and @odimitrijevic )
Ah, it looks like there is another way of instrumenting superset, but it's only with statsd: https://superset.apache.org/docs/installation/event-logging#statsd-logging
Note that it’s also possible to implement you own logger by deriving superset.stats_logger.BaseStatsLogger.
Probably better to implement our own prometheus support. https://github.com/prometheus/client_python
Thank you for the quick followup everyone! Please note that this work isn't super urgent on our (o11y) end, although graphite/statsd are in "support mode only" and deprecated
I can't see the panel, perhaps the dashboard hasn't been saved?
So my preference would probably be to migrate them to prometheus-statsd-exporter and possibly rename them, so that they are named something more relevant to Superset.
Agreed, the gunicorn specific metrics I think should have gunicorn generic names though. We'll be able to tell the metrics apart anyways via prometheus labels, and the generic metrics will be useful when/if there are more gunicorn + statsd users. Also note that our statsd-exporter acts as "passthrough" so gunicorn sends to localhost, and statsd-exporter sends to statsd.eqiad.wmnet. This change alone (i.e. even without mappings) would fix the dns issue right away because statsd-exporter does the right thing. Happy to assist with the migration/reviews too.
I've had a quick search for other ways to instrument Superset and gunicorn, but haven't turned up anything yet. We are hoping to spend some energy on improving the the Superset user experience though, so these metrics could have some value.
(cc @razzi and @odimitrijevic )
Sounds good! As @Ottomata pointed out, if using prometheus' python client is an option we should definitely opt for that
The statsd-exporter part of this work is happening in T345790: Move superset to statsd-exporter because we need to make graphite failovers simpler. The statsd traffic to statsd.eqiad.wmnet will remain unchanged (via a passthrough to statsd-exporter) and metrics will also be exported automatically to Prometheus. The metric names, labels, etc might require some tuning via statsd-exporter mappings profile::prometheus::statsd_exporter::mappings, if you are interested in that please reach out!
Hope that helps!