Proposal: Export GrowthBook metrics to Prometheus

How things work today

These are all things I tried to learn about by reading the current code and config and full disclosure, by conferring with Claude (in the sandbox!):

• GrowthBook runs on the dse-k8s-eqiad Kubernetes cluster. There are two copies: growthbook (production) and growthbook-next (staging). Both are deployed from the charts/growthbook chart in the operations/deployment-charts repo.
• The GrowthBook container image is built by us from the GitLab repo repos/data-engineering/growthbook. It currently packages GrowthBook 4.4.0. The container starts the GrowthBook backend with this command: pm2-runtime start ecosystem.config.js --only back-end. (pm2-runtime is a process manager; ecosystem.config.js is a file from the GrowthBook source that tells it what to run.)
• GrowthBook's telemetry is off by default. Per the GrowthBook docs and the GrowthBook source, setting the environment variable TRACING_PROVIDER=opentelemetry turns it on: ecosystem.config.js checks that variable itself and preloads the instrumentation. No change to our image or its start command is needed. Everything else is configured with standard OTEL_... environment variables.
• With telemetry on, GrowthBook records both generic measurements (HTTP request durations, database calls) and its own application-level counters, gauges, and histograms and pushes them using the OpenTelemetry protocol (OTLP)
• There's a mismatch between OpenTelemetry and Prometheus so something has to sit between GrowthBook and Prometheus and translate:
  • OpenTelemetry pushes data - the application sends its metrics to a receiving endpoint that you point it at.
  • Prometheus does the opposite - it pulls by periodically fetching a /metrics web page from the application (GrowthBook does not bundle the optional package that would let it serve such a page itself).
• Wikimedia does run a shared OpenTelemetry receiver ("collector") but only on the wikikube clusters, not on dse-k8s where GrowthBook lives. Even there it accepts traces only and forwards them to Jaeger. Its metrics handling is explicitly switched off. So currently there is nothing GrowthBook could send metrics to, and nothing on dse-k8s that speaks OTLP at all.
• The Prometheus server that watches dse-k8s already has a standing rule (defined in puppet, modules/profile/manifests/prometheus/k8s.pp): for any pod labelled with the annotation prometheus.io/scrape_by_name: "true", it automatically pulls metrics from every container port whose name ends in -metrics. The chart scaffolding GrowthBook uses can already emit that annotation (via the monitoring.enabled and monitoring.named_ports settings). So the "configure Prometheus" half of the task should need no changes to Prometheus itself.

Proposed approach

Add a second, small container to the GrowthBook backend pod that acts as the translator. This pattern (a helper container running alongside the main one in the same pod) is commonly called a sidecar. The helper would be the standard OpenTelemetry Collector, using the same image Wikimedia already builds and publishes for the wikikube collector (otelcol on docker-registry.wikimedia.org).

The process would be:

1. GrowthBook is started with telemetry on (TRACING_PROVIDER=opentelemetry) and told via OTEL_... environment variables to send metrics to port 4318 on 127.0.0.1 which is the helper container inside the same pod. (4318 is the standard OTLP port; the traffic never leaves the pod.)
2. The helper container receives those metrics and republishes them as a plain /metrics page on a port we choose (e.g. 9464, the conventional port for this which is free in this pod), on a container port named otel-metrics.
3. Because the port name ends in -metrics and the pod carries the scrape annotation, the existing dse-k8s Prometheus picks it up automatically.
4. From Prometheus, the metrics are visible in Grafana through the existing k8s-dse data source so dashboards and alerts can be built on them. Metrics from the two copies are distinguishable by the Kubernetes namespace label Prometheus attaches to everything it scrapes.

Only metrics are collected. GrowthBook's log collection is off by default and stays off. Trace export is suppressed with the standard OTEL_TRACES_EXPORTER=none variable since there is no trace storage reachable from this cluster. GrowthBook's telemetry setup leaves trace configuration to the standard variables so this is expected to work. And if it turns out not to, the helper container simply accepts the traces and discards them.

Why this shape and not something else?

Use the shared Wikimedia collector instead?

It is not deployed on GrowthBook's cluster and it currently handles traces only. Getting it deployed to dse-k8s with metrics support would be a change to shared observability infrastructure owned by SRE, a much larger conversation. The sidecar keeps the whole change inside the chart the team already owns. Does this seem agreeable?

Have Prometheus accept pushed metrics directly?

Prometheus has an optional feature for this but Wikimedia's Prometheus servers are not set up for it, and it goes against how monitoring is done at WMF (pull, not push).

Skip the helper and have GrowthBook serve /metrics itself?

Ruled out: GrowthBook's dependency list does not include the OpenTelemetry package that provides this, so it can only push.

Changes needed

All changes are in charts/growthbook/charts/growthbook-backend in deployment-charts. No image changes, and no Prometheus/puppet changes.

• Add the helper container and a small config file for it (receive OTLP on localhost port 4318, republish on the named port otel-metrics), switchable on and off via a chart setting.
• Open the metrics port in the pod's network policy (the firewall rules attached to the pod), since the current rules only open the ports of the main container.
• Turn on the two monitoring settings so the pod gets the scrape annotation.
• Add three environment variables for GrowthBook: TRACING_PROVIDER=opentelemetry, OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318, and OTEL_TRACES_EXPORTER=none.

Rollout

Apply to growthbook-next first. Check that the helper's metrics port actually serves GrowthBook metrics, confirm the backend logs show no failed trace deliveries, then check the metrics show up in Grafana under the k8s-dse data source. Only then apply to production growthbook. Dashboards and alerts in Grafana would be a follow-up piece of work, informed by the actual list of metrics we see once growthbook-next is running with this enabled.

Open questions

1. Is a per-service sidecar approach permissible by SRE?
2. Are there near-term plans for shared OpenTelemetry metrics handling that we should wait for?
3. Can someone confirm the -next image matches? growthbook-next is built from a pinned git commit rather than the 4.4.0 release - does the same TRACING_PROVIDER mechanism exists at that commit? Presumably it does since -next is newer?

Is a per-service sidecar approach permissible by SRE?

@brouberol, @BTullis: Is the suggested sidecar OTel collector approach acceptable? Are there other deployments in the dse-k8s cluster that use this pattern?

Are there near-term plans for shared OpenTelemetry metrics handling that we should wait for?

Counter question: Is the suggested approach easy enough to reverse once those shared plans are implemented? My guess would be yes.

Can someone confirm the -next image matches? growthbook-next is built from a pinned git commit rather than the 4.4.0 release - does the same TRACING_PROVIDER mechanism exists at that commit? Presumably it does since -next is newer?

This is something that we're going to have to track. I would hope that Growthbook would announce such a major change.

Thanks @cjming and @phuedx for getting this conversation going.

I'll also invite @CDanis into the conversation, since they know all about our Jaeger and distributed tracing setup.
As I understand it, we are currently sending traces into Jaeger from envoy, from MediaWiki, and from certain Abstract Wikipedia services.
However, an invitation is open for other service owners who might like to onboard other services. From the linked Wikitech page...

Any interested service owners who want to emit OTel themselves should get in touch...

Here are a few thoughts and responses in general.

In T430118#12131472, @phuedx wrote:

Is a per-service sidecar approach permissible by SRE?

@brouberol, @BTullis: Is the suggested sidecar OTel collector approach acceptable?

Yes, we can happily run sidecars and would have no problem with running an OTel sidecar

Are there other deployments in the dse-k8s cluster that use this pattern?

Not yet, no. Our DataHub deployment could make use of OTel (https://docs.datahub.com/docs/advanced/monitoring#distributed-tracing) but we have not enabled it.
Nor is it our highest priority feature that we would like to enable in DataHub, but that could change.
Airflow could also use opentelemetry, but we haven't gone there yet, either.

Are there near-term plans for shared OpenTelemetry metrics handling that we should wait for?

Not on the dse-k8s side, although we would have no objection to enabling it for you on the dse-k8s cluster(s).

Counter question: Is the suggested approach easy enough to reverse once those shared plans are implemented? My guess would be yes.

Yes, I don'think that we're painting ourselves into a corner with any of this.

It's also worth knowing that we do have some GrowthBook related metrics in prometheus right now.
We have:

• metrics from the ingressgateway component, which gives latency and availability of requests into the frontend and API: https://grafana-rw.wikimedia.org/goto/ffse416a5sglca?orgId=default
• metrics from the ldap-sync job, which manages users and permissions.
• metrics from kubernetes about pod memory and CPU use etc. Sometimes this is difficult to browse on Grafana, but it's there in prometheus.

This screenshot shows how you can search for any metrics matching growthbook, plus I am sure that there will be plenty more, if you look hard enough:

The ldap-sync job actually uses a feature that might well be relevant. This is the Promethsue Pushgateway: https://wikitech.wikimedia.org/wiki/Prometheus#Pushgateway

Now I will respond to some of the other statements from the proposal, with hopefully due courtesy to both Clare and Claude...

Use the shared Wikimedia collector instead?

It is not deployed on GrowthBook's cluster and it currently handles traces only. Getting it deployed to dse-k8s with metrics support would be a change to shared observability infrastructure owned by SRE, a much larger conversation.

As above, I don't consider it really as big a deal as made out to get OTel enabled for dse-k8s enabled.
The handles traces only statement is probably the biggest thing that I would like to consider. Are you certain that OTel will measure what you are most interested in measuring?

1. Have Prometheus accept pushed metrics directly? Prometheus has an optional feature for this but Wikimedia's Prometheus servers are not set up for it, and it goes against how monitoring is done at WMF (pull, not push).

This is not really correct, as we have the push gateway. However, again it's about what you really want to measure. The push gateway is great for ephemeral jobs, so anything with a start and finish.

Skip the helper and have GrowthBook serve /metrics itself?

Ruled out: GrowthBook's dependency list does not include the OpenTelemetry package that provides this, so it can only push.

Again, not really a big deal. We build GrowthBook ourselves and therefore we could add whatever we want into the image as a dependency.

One thing that occurs to me is that we used to use statsd metrics, which are a push based mechanism.
In order to support these metrics when Graphite was deprecated, we started to use the statsd_exporter as an intermediary.
This is installed as a sidecar and MediaWiki or Superset or whatever sends its statsd metrics to this exporter, which then makes them available for scraping by Prometheus.

We might conceivably be able to do something similar with OpenTelemetry. Here is an exporter that might fit the bill.
https://opentelemetry-python.readthedocs.io/en/latest/exporter/prometheus/prometheus.html

So I suspect that the best thing to do at the moment would be for you to ascertain what information most interests you, and whether opentelemetry supports it.
If it does support it, then let's work out whether we could make this available for scraping by Prometheus, using an exporter like this.

Thanks @BTullis for your corrections! Super helpful - definitely got some things wrong in the proposal - my bad. Let me sort out what assumptions you debunked first, then propose next steps.

Corrections

2 of my/Claude's justifications are wrong:

• I said WMF Prometheus isn't set up to accept pushed metrics. As you noted, that's not right: we have the Pushgateway, and ldap-sync already uses it. (Understood that it's aimed at ephemeral start-and-finish jobs rather than a long-running service like the GrowthBook backend - so maybe not the right fit anyway.)
• I said we couldn't have GrowthBook serve its own /metrics page because the exporter package isn't bundled. Also not a blocker since we build the image ourselves and could add the dependency.

And duly noted that enabling OTel on dse-k8s isn't as big a lift as I assumed it was.

Next steps

I'd like to take up your suggestion to work out what we actually want to measure before settling on how to move it. Since a fair amount is already in Prometheus (ingressgateway latency and availability, ldap-sync, pod CPU/memory), my next step is to define what "GrowthBook health" should cover for us and identify which of those signals, if any, aren't already available. I'll come back with that list rather than guess at it here.

I'll also check what GrowthBook's OpenTelemetry instrumentation actually emits. The backend's tracing setup enables the Node auto-instrumentation bundle and wires up a hook for GrowthBook to register its own custom metrics (i.e GB's custom counters, gauges, and histograms registered via setMetrics()). I haven't confirmed yet whether any are actually registered or what they would measure. I'll read the upstream source at our pinned version and report the concrete list, matched against whatever the health list needs.

Once that's clear we can pick a transport. From your reply there seem to be a few workable shapes (the OTel-collector sidecar, a statsd_exporter-style exporter sidecar, or a Prometheus exporter built into the image). I'd rather let the choice follow from what we're measuring as you suggested.

On traces: I'd originally set them aside because I thought there was nowhere to send them. But with the Jaeger onboarding invitation and @CDanis in the thread, that might not be the case. I'll treat whether to emit traces as a separate question from the metrics work.

I'll follow up with the measurement list and the OTel coverage check before proposing any chart changes.

What GrowthBook's OTel instrumentation actually emits

I tried to read the backend source at the pinned version (v4.4.0) by cloning the growthbook repo and running the following:

grep -rn -E "\.getCounter\(|\.getHistogram\(|\.getGauge\(" packages/back-end packages/shared

The telemetry appears to have two parts:

1. One is the standard Node auto-instrumentation bundle (HTTP, Express, MongoDB and similar) generated by a library which I haven't enumerated by hand.
2. The other is GrowthBook's own custom metrics written in its code - these are the interesting part: 13 of them in two families. The names and types below come straight from the call sites. The "appears to measure" column is my inference from each metric's name and the file it lives in (take with usual grains of salt):

python.*, the statistics engine (a pool of Python worker processes that compute experiment results):

jobs.*, the background job runner (the agenda queue):

Things to note:

• With telemetry off (our current state), these custom metrics are no-ops and emit nothing. They only activate when TRACING_PROVIDER=opentelemetry is set, so this is opt-in and reversible.
• The "counter" types are implemented as OTel UpDownCounters, and OTel-to-Prometheus conversion renames things (dots become underscores, histograms gain _bucket/_sum/_count). So the exact metric names and types in Prometheus should be confirmed from a live scrape and not from these source names.

How this lines up with what we already have in Prometheus

The growthbook metrics we collect today come from Test Kitchen's own instrumentation (its experiment-configuration pipeline: polling the GrowthBook API, validation, drafting, Phabricator comments, and the API-request durations it sees) and I believe the ldap-sync job (the growthbook_sync_* series). Together with the ingressgateway request latency and availability and the pod CPU/memory @BTullis pointed to, those describe the integration around GrowthBook, the user-sync, and the pods, rather than the GrowthBook backend service's own internals.

The custom OTel metrics above don't overlap with those with one exception: growthbook_api_request_duration_seconds already captures GrowthBook API latency from Test Kitchen's client. The generic HTTP timings OTel would add are largely redundant with what we have (and with ingressgateway). The part that's genuinely new is the internal health of the stats engine (python.*) and the background job runner (jobs.*) which nothing currently watches. Whether those are the signals we most want is worth discussing (TBD).

More questions

• This is the custom-metric subset found in packages/back-end and packages/shared - are there other places to look? Should a definitive list come from scraping a running instance?
• The auto-instrumentation set (HTTP/DB/etc.) is additional and I think will show up alongside these?
• If the stats-engine and jobs metrics look valuable, should we enable telemetry on growthbook-next, scrape it, and post the real, complete list (custom plus auto-instrumentation with their actual Prometheus names) so that we can make decisions about dashboards, alerts, and the transport from the scrape?

Per discussion with the team, there was a consensus that the python and jobs metrics are not valuable for what we're looking in terms for GB health metrics. It'd be great if there were some metrics geared toward database queries, etc (we can inquire with GrowthBook). So for now we are treating this ticket as an investigation spike and closing for now