Page MenuHomePhabricator
Create Task
Maniphest T327753

jmx_presto prometheus job down for some an-presto hosts
Open, Needs TriagePublic
Actions

Description

The following hosts are not pollable by prometheus eqiad on their jmx_exporter ports:

https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets?orgId=1&var-datasource=thanos&var-Filters=job%7C%3D%7Cjmx_presto

2023-01-24 09:48:33	an-presto1006:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1007:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1008:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1009:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1010:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1011:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1012:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1013:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1014:10281	jmx_presto	analytics	eqiad	0
2023-01-24 09:48:33	an-presto1015:10281	jmx_presto	analytics	eqiad	0

cc @BTullis @Stevemunene

Related Objects

Event Timeline

fgiunchedi created this task.Jan 24 2023, 9:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2023, 9:50 AM
Stevemunene added a comment.Jan 24 2023, 4:08 PM

Hi @fgiunchedi the 10 servers were taken out of the cluster due to challenges in joining the presto cluster documented here T325809 T325331 T323783

Last message from @BTullis being " Icinga downtime and Alertmanager silence (ID=4f65baff-bd05-4e2f-8578-eae4b972dc3b) set by btullis@cumin1001 for 30 days, 0:00:00 on 10 host(s) and their services with reason: Still not ready to add these new presto servers to the cluster - btullis "

fgiunchedi added a comment.Jan 25 2023, 10:13 AM

Thank you @Stevemunene ! I have acked the alert

Aklapper added a project: Data-Engineering-Icebox.Feb 10 2023, 5:44 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL