Page MenuHomePhabricator
Create Task
Maniphest T233956

Deploy Thanos (long-term storage) stateless components: sidecar and query
Closed, ResolvedPublic
Actions

Description

This task tracks the deployment of Thanos stateless components, the big win being a query endpoint that can reach out to all prometheus instances and merge/deduplicate results as needed.

Outline of what's needed:

  • Thanos Debian package
  • Prometheus instances need to advertise unique external_labels
    • Need to add labels: instance (or name or sth like that) plus replica (A or B)
    • The labels above need to be filtered out before ingestion by our global instance for backwards compatibility
  • Deploy Thanos sidecar alongside each Prometheus instance (save for global)
    • Needs two ports for each of http + grpc interfaces, likely as an offset of the instance's port itself
  • Deploy Thanos query component
    • Deploy on thanos-fe2* hosts
    • Deploy on thanos-fe1* hosts
    • Needs two ports for http+grpc, and labels that are considered for deduplication (replica in our case)
    • Needs to locate and reach all other Thanos sidecars
    • Deploy behind LVS and can be active/active (i.e. discovery DNS records) following https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service.
  • Configure and test datasource in Grafana
  • Audit and document how to port dashboards to Thanos (T256954)

Details

SubjectRepoBranchLines +/-
thanos: set consistency-delay on storeoperations/puppetproduction+4 -1
monitoring: switch to new names for global availability metricsoperations/puppetproduction+2 -2
prometheus: move global availability rules to new namesoperations/puppetproduction+6 -6
prometheus: import availability aggregation rules from Prometheus globaloperations/puppetproduction+36 -6
hieradata: page on thanos swift/query failureoperations/puppetproduction+2 -2
templates: add PTR for thanos-swift / thanos-queryoperations/dnsmaster+4 -0
hieradata: add eqiad for thanos-query / thanos-swiftoperations/puppetproduction+10 -0
conftool-data: add thanos-fe eqiadoperations/puppetproduction+4 -0
site: add thanos-fe1* to frontendsoperations/puppetproduction+5 -2
thanos: add SyslogIdentifier=%N to systemd servicesoperations/puppetproduction+4 -0
role: use_remote_address: true for Envoy on Thanosoperations/puppetproduction+1 -0
grafana: provision Thanos datasourceoperations/puppetproduction+11 -0
prometheus: rename Thanos jobsoperations/puppetproduction+4 -4
profile: fix thanos swift healthcheckoperations/puppetproduction+3 -8
hieradata: move thanos-query service to port 80operations/puppetproduction+2 -2
profile: fix thanos-query httpd proxypassoperations/puppetproduction+1 -1
profile: open port 80 for thanos-queryoperations/puppetproduction+6 -0
profile: add thanos::httpd to proxy thanos-queryoperations/puppetproduction+41 -0
prometheus: add thanos::sidecar to services and analyticsoperations/puppetproduction+10 -0
thanos: add lvs addresses to frontendoperations/puppetproduction+6 -0
hieradata: add thanos-query to service::catalogoperations/puppetproduction+34 -0
wmnet: allocate thanos-query.svc addressesoperations/dnsmaster+2 -0
conftool-data: add thanos-queryoperations/puppetproduction+5 -0
site: assign thanos::frontend to thanos-fe2*operations/puppetproduction+1 -1
thanos: fix query class arguments and pathoperations/puppetproduction+4 -4
prometheus: add thanos sidecar to k8s instancesoperations/puppetproduction+10 -0
role: rename thanos::query in thanos::frontendoperations/puppetproduction+10 -10
prometheus: scrape thanos sidecar/query metricsoperations/puppetproduction+32 -1
Add Thanos queryoperations/puppetproduction+100 -0
prometheus: add thanos-sidecar to prometheus@opsoperations/puppetproduction+10 -2
modules: add thanos-sidecar define and profileoperations/puppetproduction+81 -0
debian: first commitoperations/debs/thanosdebian/buster-wikimedia+684 -0
prometheus: additional external_labels for Thanosoperations/puppetproduction+29 -6
role: fix Prometheus global metric relabel configoperations/puppetproduction+1 -2
role: drop Thanos labels from global Prometheusoperations/puppetproduction+10 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.May 7 2020, 10:10 AM
gerritbot added a comment.May 7 2020, 10:18 AM

Change 594919 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add thanos sidecar to k8s instance

https://gerrit.wikimedia.org/r/594919

gerritbot added a project: Patch-For-Review.May 7 2020, 10:18 AM
gerritbot added a comment.May 8 2020, 7:37 AM

Change 594919 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add thanos sidecar to k8s instances

https://gerrit.wikimedia.org/r/594919

Maintenance_bot removed a project: Patch-For-Review.May 8 2020, 8:10 AM
gerritbot added a comment.May 8 2020, 8:40 AM

Change 595138 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thanos: fix query class arguments and path

https://gerrit.wikimedia.org/r/595138

gerritbot added a project: Patch-For-Review.May 8 2020, 8:40 AM
fgiunchedi updated the task description. (Show Details)May 8 2020, 9:06 AM
gerritbot added a comment.May 8 2020, 9:07 AM

Change 595138 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: fix query class arguments and path

https://gerrit.wikimedia.org/r/595138

Maintenance_bot removed a project: Patch-For-Review.May 8 2020, 9:10 AM
fgiunchedi mentioned this in T252186: Deploy Thanos (Prometheus long-term storage) stateful components.May 8 2020, 9:24 AM
gerritbot added a comment.May 11 2020, 8:08 AM

Change 595473 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] site: assign thanos::frontend to thanos-fe2*

https://gerrit.wikimedia.org/r/595473

gerritbot added a project: Patch-For-Review.May 11 2020, 8:08 AM

Change 595473 merged by Filippo Giunchedi:
[operations/puppet@production] site: assign thanos::frontend to thanos-fe2*

https://gerrit.wikimedia.org/r/595473

Maintenance_bot removed a project: Patch-For-Review.May 11 2020, 8:10 AM
fgiunchedi updated the task description. (Show Details)May 11 2020, 8:48 AM
gerritbot added a comment.May 11 2020, 9:22 AM

Change 595489 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] wmnet: allocate thanos-query.svc addresses

https://gerrit.wikimedia.org/r/595489

gerritbot added a project: Patch-For-Review.May 11 2020, 9:22 AM
gerritbot added a comment.May 11 2020, 9:25 AM

Change 595491 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] conftool-data: add thanos-query

https://gerrit.wikimedia.org/r/595491

fgiunchedi updated the task description. (Show Details)May 11 2020, 9:41 AM
gerritbot added a comment.May 11 2020, 9:41 AM

Change 595493 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add thanos-query to service::catalog

https://gerrit.wikimedia.org/r/595493

gerritbot added a comment.May 11 2020, 9:41 AM

Change 595494 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thanos: add lvs addresses to frontend

https://gerrit.wikimedia.org/r/595494

gerritbot added a comment.May 12 2020, 9:23 AM

Change 595491 merged by Filippo Giunchedi:
[operations/puppet@production] conftool-data: add thanos-query

https://gerrit.wikimedia.org/r/595491

gerritbot added a comment.May 12 2020, 9:26 AM

Change 595489 merged by Filippo Giunchedi:
[operations/dns@master] wmnet: allocate thanos-query.svc addresses

https://gerrit.wikimedia.org/r/595489

gerritbot added a comment.May 12 2020, 2:20 PM

Change 595493 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add thanos-query to service::catalog

https://gerrit.wikimedia.org/r/595493

gerritbot added a comment.May 12 2020, 2:20 PM

Change 595494 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: add lvs addresses to frontend

https://gerrit.wikimedia.org/r/595494

Maintenance_bot removed a project: Patch-For-Review.May 12 2020, 3:10 PM
fgiunchedi updated the task description. (Show Details)May 12 2020, 3:45 PM
gerritbot added a comment.May 13 2020, 8:06 AM

Change 596147 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add thanos::sidecar to services and analytics

https://gerrit.wikimedia.org/r/596147

gerritbot added a project: Patch-For-Review.May 13 2020, 8:06 AM
gerritbot added a comment.May 13 2020, 8:14 AM

Change 596147 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add thanos::sidecar to services and analytics

https://gerrit.wikimedia.org/r/596147

fgiunchedi updated the task description. (Show Details)May 13 2020, 8:16 AM
Maintenance_bot removed a project: Patch-For-Review.May 13 2020, 9:10 AM
gerritbot added a comment.May 19 2020, 9:52 AM

Change 597245 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: add thanos::httpd to proxy thanos-query

https://gerrit.wikimedia.org/r/597245

gerritbot added a project: Patch-For-Review.May 19 2020, 9:52 AM
gerritbot added a comment.May 19 2020, 1:45 PM

Change 597245 merged by Filippo Giunchedi:
[operations/puppet@production] profile: add thanos::httpd to proxy thanos-query

https://gerrit.wikimedia.org/r/597245

Maintenance_bot removed a project: Patch-For-Review.May 19 2020, 2:10 PM
gerritbot added a comment.May 20 2020, 9:29 AM

Change 597481 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: open port 80 for thanos-query

https://gerrit.wikimedia.org/r/597481

gerritbot added a project: Patch-For-Review.May 20 2020, 9:29 AM
gerritbot added a comment.May 20 2020, 9:32 AM

Change 597481 merged by Filippo Giunchedi:
[operations/puppet@production] profile: open port 80 for thanos-query

https://gerrit.wikimedia.org/r/597481

gerritbot added a comment.May 20 2020, 9:38 AM

Change 597482 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: fix thanos-query httpd proxypass

https://gerrit.wikimedia.org/r/597482

gerritbot added a comment.May 20 2020, 9:44 AM

Change 597482 merged by Filippo Giunchedi:
[operations/puppet@production] profile: fix thanos-query httpd proxypass

https://gerrit.wikimedia.org/r/597482

gerritbot added a comment.May 20 2020, 9:49 AM

Change 597485 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: move thanos-query service to port 80

https://gerrit.wikimedia.org/r/597485

gerritbot added a comment.May 20 2020, 12:19 PM

Change 597485 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: move thanos-query service to port 80

https://gerrit.wikimedia.org/r/597485

Stashbot added a comment.May 20 2020, 12:28 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-20T12:28:04Z] <godog> roll-restart pybal on codfw low-traffic - T233956

Maintenance_bot removed a project: Patch-For-Review.May 20 2020, 1:11 PM
gerritbot added a comment.May 20 2020, 2:29 PM

Change 597557 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: fix thanos swift healthcheck

https://gerrit.wikimedia.org/r/597557

gerritbot added a project: Patch-For-Review.May 20 2020, 2:29 PM
gerritbot added a comment.May 20 2020, 2:32 PM

Change 597557 merged by Filippo Giunchedi:
[operations/puppet@production] profile: fix thanos swift healthcheck

https://gerrit.wikimedia.org/r/597557

Maintenance_bot removed a project: Patch-For-Review.May 20 2020, 3:10 PM
fgiunchedi added a comment.EditedMay 20 2020, 3:19 PM

The https/envoy part is up, however check_http is failing while curl works:

icinga1001:~$ /usr/lib/nagios/plugins/check_http -H thanos-swift.discovery.wmnet -S -I 10.192.0.192 -u /healthcheck
CRITICAL - Cannot make SSL connection.
icinga1001:~$ curl --resolve thanos-swift.discovery.wmnet:443:10.192.0.192 https://thanos-swift.discovery.wmnet/healthcheck
OK

Turns out this is due to SNI requirement for Thanos (profile::tlsproxy::envoy::sni_support: 'strict')

icinga1001:~$ /usr/lib/nagios/plugins/check_http -H thanos-swift.discovery.wmnet -S --sni -I 10.192.0.192 -u /healthcheck
HTTP OK: HTTP/1.1 200 OK - 279 bytes in 1.155 second response time |time=1.154771s;;;0.000000;10.000000 size=279B;;;0

I think we (in decreasing order of preference) could:

  1. default check_http with --sni for the https cases
  2. relax the sni requirement on the envoy/thanos side
  3. add yet another specialized icinga command definition for https+sni
fgiunchedi mentioned this in T253292: check_http and SNI support.May 21 2020, 8:30 AM
gerritbot added a comment.May 27 2020, 2:20 PM

Change 599040 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: rename Thanos jobs

https://gerrit.wikimedia.org/r/599040

gerritbot added a project: Patch-For-Review.May 27 2020, 2:20 PM

Change 599040 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: rename Thanos jobs

https://gerrit.wikimedia.org/r/599040

Maintenance_bot removed a project: Patch-For-Review.May 27 2020, 3:10 PM
fgiunchedi added a comment.May 27 2020, 3:36 PM

I've added the Thanos datasource to Grafana (manually for now, will add to the provisioned datasources) and the query part seems to work, also provisional dashboards taken from upstream:

gerritbot added a comment.May 27 2020, 3:39 PM

Change 599059 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] grafana: provision Thanos datasource

https://gerrit.wikimedia.org/r/599059

gerritbot added a project: Patch-For-Review.May 27 2020, 3:39 PM
gerritbot added a comment.May 28 2020, 4:14 PM

Change 599059 merged by Filippo Giunchedi:
[operations/puppet@production] grafana: provision Thanos datasource

https://gerrit.wikimedia.org/r/599059

Maintenance_bot removed a project: Patch-For-Review.May 28 2020, 5:11 PM
Stashbot added a comment.May 29 2020, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-29T12:15:13Z] <godog> roll-restart to upgrade thanos to 0.13.0rc0 - T252186 T233956

fgiunchedi updated the task description. (Show Details)Jun 1 2020, 9:34 AM
gerritbot added a comment.Jun 1 2020, 9:56 AM

Change 601320 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] role: use_remote_address: true for Envoy on Thanos

https://gerrit.wikimedia.org/r/601320

gerritbot added a project: Patch-For-Review.Jun 1 2020, 9:56 AM
gerritbot added a comment.Jun 1 2020, 10:04 AM

Change 601320 merged by Filippo Giunchedi:
[operations/puppet@production] role: use_remote_address: true for Envoy on Thanos

https://gerrit.wikimedia.org/r/601320

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2020, 10:10 AM
gerritbot added a comment.Jun 9 2020, 1:31 PM

Change 604009 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thanos: add SyslogIdentifier=%N to systemd services

https://gerrit.wikimedia.org/r/604009

gerritbot added a project: Patch-For-Review.Jun 9 2020, 1:31 PM
gerritbot added a comment.Jun 10 2020, 9:49 AM

Change 604009 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: add SyslogIdentifier=%N to systemd services

https://gerrit.wikimedia.org/r/604009

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2020, 10:10 AM
gerritbot added a comment.Jun 10 2020, 4:12 PM

Change 604433 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] site: add thanos-fe1* to frontends

https://gerrit.wikimedia.org/r/604433

gerritbot added a project: Patch-For-Review.Jun 10 2020, 4:12 PM
gerritbot added a comment.Jun 10 2020, 4:16 PM

Change 604433 merged by Filippo Giunchedi:
[operations/puppet@production] site: add thanos-fe1* to frontends

https://gerrit.wikimedia.org/r/604433

gerritbot added a comment.Jun 11 2020, 8:23 AM

Change 604608 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] conftool-data: add thanos-fe eqiad

https://gerrit.wikimedia.org/r/604608

gerritbot added a comment.Jun 11 2020, 8:28 AM

Change 604608 merged by Filippo Giunchedi:
[operations/puppet@production] conftool-data: add thanos-fe eqiad

https://gerrit.wikimedia.org/r/604608

gerritbot added a comment.Jun 11 2020, 8:33 AM

Change 604613 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add eqiad for thanos-query / thanos-swift

https://gerrit.wikimedia.org/r/604613

gerritbot added a comment.Jun 11 2020, 10:29 AM

Change 604613 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add eqiad for thanos-query / thanos-swift

https://gerrit.wikimedia.org/r/604613

gerritbot added a comment.Jun 11 2020, 11:00 AM

Change 604664 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] templates: add PTR for thanos-swift / thanos-query

https://gerrit.wikimedia.org/r/604664

gerritbot added a comment.Jun 11 2020, 1:39 PM

Change 604664 merged by Filippo Giunchedi:
[operations/dns@master] templates: add PTR for thanos-swift / thanos-query

https://gerrit.wikimedia.org/r/604664

fgiunchedi updated the task description. (Show Details)Jun 11 2020, 2:06 PM

This is complete! Thanos frontend is available in codfw and eqiad

gerritbot added a comment.Jun 16 2020, 10:21 AM

Change 605859 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: page on thanos swift/query failure

https://gerrit.wikimedia.org/r/605859

gerritbot added a comment.Jun 16 2020, 10:22 AM

Change 605859 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: page on thanos swift/query failure

https://gerrit.wikimedia.org/r/605859

fgiunchedi updated the task description. (Show Details)Jun 22 2020, 12:37 PM
gerritbot added a comment.Jun 22 2020, 1:58 PM

Change 607031 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: import availability aggregation rules from Prometheus global

https://gerrit.wikimedia.org/r/607031

gerritbot added a comment.Jun 23 2020, 9:24 AM

Change 607031 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: import availability aggregation rules from Prometheus global

https://gerrit.wikimedia.org/r/607031

gerritbot added a comment.Jun 23 2020, 9:57 AM

Change 607256 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: move global availability rules to new names

https://gerrit.wikimedia.org/r/607256

gerritbot added a comment.Jun 23 2020, 3:42 PM

Change 607256 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: move global availability rules to new names

https://gerrit.wikimedia.org/r/607256

gerritbot added a comment.Jun 25 2020, 1:11 PM

Change 607783 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thanos: set consistency-delay on store

https://gerrit.wikimedia.org/r/607783

gerritbot added a comment.Jun 29 2020, 1:24 PM

Change 608319 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] monitoring: switch to new names for global availability metrics

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608319

gerritbot added a comment.Jul 2 2020, 8:07 AM

Change 608319 merged by Filippo Giunchedi:
[operations/puppet@production] monitoring: switch to new names for global availability metrics

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608319

gerritbot added a comment.Jul 2 2020, 8:32 AM

Change 607783 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: set consistency-delay on store

https://gerrit.wikimedia.org/r/c/operations/puppet/ /607783

fgiunchedi closed this task as Resolved.Jul 8 2020, 8:44 AM
fgiunchedi claimed this task.
fgiunchedi updated the task description. (Show Details)

Resolving, porting dashboards to Thanos is tracked in T256954: Port Prometheus dashboards to Thanos

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL