Page MenuHomePhabricator

Prometheus 2 breaking change
Open, Needs TriagePublic

Description

@fgiunchedi - in starting to upgrade hosts to stretch I ran across this message of sadness:

BREAKING CHANGES
The data format in Prometheus 2.0 has completely changed and is not
backwards compatible with 1.x. Prometheus 2 will create a new data directory
in /var/lib/prometheus/metrics2. The old data in /var/lib/prometheus/metrics
can not be read by the new version.
There is no conversion tool; if you want to retain access to the old data,
it is necessary to run a separate Prometheus 1.x instance on that data
directory. (This package makes no provision to allow this.)

How are you handling this in prod? Thanks!

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
ResolvedJgreen
ResolvedJgreen
OpenNone
ResolvedJgreen
ResolvedJgreen
ResolvedJgreen
ResolvedJgreen
DeclinedNone
ResolvedJgreen
ResolvedJgreen
ResolvedJgreen
ResolvedJgreen
ResolvedJgreen
OpenCmjohnson
ResolvedJgreen
ResolvedJgreen
ResolvedCmjohnson
ResolvedEjegg
ResolvedEjegg
ResolvedJgreen
ResolvedJgreen
ResolvedJgreen
Resolvedayounsi
ResolvedPapaul
ResolvedJgreen
ResolvedJgreen
DuplicateNone
Resolved cwdent
ResolvedJgreen
ResolvedJgreen
ResolvedNone
ResolvedPcoombe
Duplicate cwdent
ResolvedJgreen
OpenJgreen
OpenNone
OpenNone
ResolvedJgreen

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2018, 8:21 PM
DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.Jan 22 2018, 8:47 PM

Good question @cwdent. We haven't tackled the problem in production yet, though IIRC Prometheus suggests setting up a v2 instance with remote reading from the existing v1 instance. This way data that's not present in v2 will be read from v1, when enough time has passed (e.g. the Prometheus retention period) we can decom the v1 instance.

Good question @cwdent. We haven't tackled the problem in production yet, though IIRC Prometheus suggests setting up a v2 instance with remote reading from the existing v1 instance. This way data that's not present in v2 will be read from v1, when enough time has passed (e.g. the Prometheus retention period) we can decom the v1 instance.

Forgot to add, we're tracking the migration in T187987: 100% of Prometheus traffic served by Prometheus v2