Page MenuHomePhabricator
Create Task
Maniphest T185134

Prometheus 2 breaking change
Closed, DeclinedPublic
Actions

Description

@fgiunchedi - in starting to upgrade hosts to stretch I ran across this message of sadness:

BREAKING CHANGES
The data format in Prometheus 2.0 has completely changed and is not
backwards compatible with 1.x. Prometheus 2 will create a new data directory
in /var/lib/prometheus/metrics2. The old data in /var/lib/prometheus/metrics
can not be read by the new version.
There is no conversion tool; if you want to retain access to the old data,
it is necessary to run a separate Prometheus 1.x instance on that data
directory. (This package makes no provision to allow this.)

How are you handling this in prod? Thanks!

Related Objects
Search...

Event Timeline

• cwdent created this task.Jan 17 2018, 8:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2018, 8:21 PM
• cwdent added a subtask: T185013: EPIC: migrate fundraising off of Debian Jessie.Jan 17 2018, 8:43 PM
• DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.Jan 22 2018, 8:47 PM
fgiunchedi added a comment.Jan 25 2018, 12:57 AM

Good question @cwdent. We haven't tackled the problem in production yet, though IIRC Prometheus suggests setting up a v2 instance with remote reading from the existing v1 instance. This way data that's not present in v2 will be read from v1, when enough time has passed (e.g. the Prometheus retention period) we can decom the v1 instance.

• DStrine moved this task from FR-Ops to Epics in progress on the Fundraising-Backlog board.Feb 21 2018, 5:16 PM
Jgreen added a parent task: T91508: [Epic] overhaul fundraising cluster monitoring.Jun 21 2018, 6:52 PM
fgiunchedi added a comment.Oct 3 2018, 3:21 PM
In T185134#3922651, @fgiunchedi wrote:

Good question @cwdent. We haven't tackled the problem in production yet, though IIRC Prometheus suggests setting up a v2 instance with remote reading from the existing v1 instance. This way data that's not present in v2 will be read from v1, when enough time has passed (e.g. the Prometheus retention period) we can decom the v1 instance.

Forgot to add, we're tracking the migration in T187987: 100% of Prometheus traffic served by Prometheus v2

Ejegg added subscribers: Jgreen, Ejegg.Aug 29 2019, 10:43 PM

@Jgreen we're using prometheus2 for the private stats server, right? Guessing this can be closed.

Jgreen edited subtasks, added: T198876: Prometheus/Grafana snapshots for long term storage; removed: T185013: EPIC: migrate fundraising off of Debian Jessie.Sep 4 2019, 6:38 PM
Jgreen closed this task as Declined.Oct 8 2019, 3:33 PM
Jgreen moved this task from Triage to Done on the fundraising-tech-ops board.

There's not really anything to be done here. We've archived the v1 historical data, and could, in theory, spin up a v1 instance to feed it to grafana, but it's just not worth the effort. It's a good reminder not to think of prometheus as a store for historical data, and to consider alternatives.

Jgreen closed subtask T198876: Prometheus/Grafana snapshots for long term storage as Resolved.Sep 7 2021, 5:35 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits