Page MenuHomePhabricator

Configure graphite to be read only
Closed, ResolvedPublic

Description

This is a task that aims to track the planning and configuration implementation of the current Graphite stack to a read-only mode, allowing querying of existing metrics while blocking the ingestion of new metrics.

Target date: Apr 30th 2025 (5pm UTC)

High-level Steps:

Planning

  • Review all applications and services configuration, pushing metrics to Graphite and document ingress points.

Implementation

  • Modify firewall rules or ACLs to block traffic on the ports used for metric ingestion.
  • Ensure that queries to the existing Graphite metrics are still functional and accessible.

Testing & Post implementation clean-up

  • Test using Grafana to confirm that no new metrics are being ingested while old metrics remain queryable.
  • Document changes made to enforce read-only access.

Event Timeline

lmata updated the task description. (Show Details)
lmata updated the task description. (Show Details)
lmata set Due Date to Apr 16 2025, 2:00 AM.Apr 11 2025, 2:02 AM
lmata removed Due Date which was set to Apr 16 2025, 2:00 AM.Apr 11 2025, 2:07 AM

@fgiunchedi noted

  • there is a dedicated statsite instance on CI hosts that should be removed as part of this
  • there are a handful of scripts that publish directly to graphite on port 2003

Change #1135076 merged by Cwhite:

[operations/puppet@production] statsd: remove ferm rule for statsd port 8125

https://gerrit.wikimedia.org/r/1135076

To roll back, revert this patch

Removed:

  • Alerts for new data being ingested into Graphite

We had an internal team discussion, and this alert is not needed because this case would require an SRE to undo the firewall change, and we find that unlikely to happen.

lmata claimed this task.
lmata moved this task from Inbox to Prioritized on the Observability-Metrics board.

All items are confirmed; I am boldly marking this task as completed.

lmata moved this task from Inbox to Done on the SRE Observability (FY2024/2025-Q4) board.