Page MenuHomePhabricator
Create Task
Maniphest T335943

prometheus-openstack-exporter: collected data shows regular null intervals
Closed, ResolvedPublic
Actions

Description

See screenshot:

image.png (887×1 px, 96 KB)

This happens with other metrics generated by the same exporter.

My only hint is that currently the exporter takes a lot of time to return the data, see:

aborrero@cloudcontrol1007:~ $ curl localhost:12345/metrics -o metrics.prom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2980k    0 2980k    0     0  45469      0 --:--:--  0:01:07 --:--:--  774k

It took curl 1:07 to complete the query. I wonder if the Prometheus scraper is somehow timing out from time to time.

Details

SubjectRepoBranchLines +/-
team-wmcs: add openstack_apis_response.yamloperations/alertsmaster+40 -0
wmcs: prometheus: increase scrape frequency for openstack APIsoperations/puppetproduction+3 -1
Revert "Revert "Revert "P:wmcs::prometheus: set openstack scrape_interval to 4m"""operations/puppetproduction+1 -1
Revert "Revert "P:wmcs::prometheus: set openstack scrape_interval to 4m""operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedAndrewT305828 upgrade cloud-vps openstack to Openstack version 'Yoga'
 ResolvedAndrewT296561 upgrade cloud-vps openstack to Openstack version 'Xena'
 ResolvedrookT281275 upgrade cloud-vps openstack to Openstack version 'Wallaby'
 ResolvedAndrewT281276 Upgrade cloud-vps openstack hosts to Debian 'Bullseye'
 ResolvedaborreroT302050 prometheus-openstack-exporter in Bullseye
 ResolveddcaroT302178 prometheus-openstack-exporter No module named 'urlparse'
 In ProgressNoneT333975 Cloud VPS: refresh openstack resources grafana dashboard
ResolvedAndrewT335943 prometheus-openstack-exporter: collected data shows regular null intervals
 ResolvedAndrewT335978 openstack: consider removing references to old hardware from the database
 OpenAndrewT362443 Learn how to do what Taavi does
 OpenNoneT362444 Taavi knowledge transfer: maintain-kubeusers
OpenNoneT362445 Taavi knowledge transfer: Toolforge k8s upgrades
 OpenNoneT362446 Taavi knowledge transfer: toolforge job investigation
 OpenNoneT362447 Taavi knowledge transfer: Toolforge misc services (e.g. mail server)
 OpenNoneT362448 Taavi knowledge transfer: rebuild toolforge docker images
 OpenAndrewT362449 Taavi knowledge transfer: python-flask-keystone, novaproxy, enc api
 OpenNoneT362450 Taavi knowledge transfer: Cloud VPS OpenTofu provider
 OpenAndrewT362452 Taavi knowledge transfer: cloud-vps monitoring
 ResolvedaborreroT339152 Cloud VPS: figure out metrics for openstack API response times and create a grafana dashboard for it
 ResolvedaborreroT340819 Cloud VPS: create alert for when the openstack API response times are too high

Event Timeline

aborrero created this task.May 4 2023, 10:53 AM
aborrero updated the task description. (Show Details)May 4 2023, 11:10 AM
aborrero changed the task status from Open to In Progress.May 4 2023, 11:38 AM
aborrero triaged this task as Low priority.

Now discovering this https://gerrit.wikimedia.org/r/c/operations/puppet/+/802434 and this https://gerrit.wikimedia.org/r/c/operations/puppet/+/802956

As of this writing the scrape interval is set to 15m. Maybe the gaps in the data comes directly from the long scrape interval.

gerritbot added a comment.May 4 2023, 11:40 AM

Change 915385 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] Revert "Revert "P:wmcs::prometheus: set openstack scrape_interval to 4m""

https://gerrit.wikimedia.org/r/915385

gerritbot added a project: Patch-For-Review.May 4 2023, 11:40 AM
gerritbot added a comment.May 4 2023, 12:27 PM

Change 915385 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] Revert "Revert "P:wmcs::prometheus: set openstack scrape_interval to 4m""

https://gerrit.wikimedia.org/r/915385

Maintenance_bot removed a project: Patch-For-Review.May 4 2023, 12:30 PM
aborrero added a comment.May 4 2023, 2:15 PM

The data shows no gap after merging the patch:

image.png (909×2 px, 94 KB)

aborrero added a comment.May 4 2023, 2:24 PM

please @Andrew be aware of this change as it might impact openstack stability. I don't think you need to do anything at the moment, just keep an eye open for any signals of instability.

Andrew added a comment.May 4 2023, 5:06 PM

I am pretty sure that the openstack APIs have gotten much slower... horizon times out for me now and then. Is it possible to reduce the number of metrics gathered rather than the frequency of checks?

aborrero added a comment.May 5 2023, 11:24 AM
In T335943#8827384, @Andrew wrote:

I am pretty sure that the openstack APIs have gotten much slower... horizon times out for me now and then. Is it possible to reduce the number of metrics gathered rather than the frequency of checks?

Ok, I'll revert while we find a solution.

gerritbot added a comment.May 5 2023, 11:29 AM

Change 915713 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] Revert "Revert "Revert "P:wmcs::prometheus: set openstack scrape_interval to 4m"""

https://gerrit.wikimedia.org/r/915713

gerritbot added a project: Patch-For-Review.May 5 2023, 11:29 AM
gerritbot added a comment.May 5 2023, 11:32 AM

Change 915713 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] Revert "Revert "Revert "P:wmcs::prometheus: set openstack scrape_interval to 4m"""

https://gerrit.wikimedia.org/r/915713

Maintenance_bot removed a project: Patch-For-Review.May 5 2023, 12:11 PM
Andrew mentioned this in T336060: cloudbackup2001 lockup on 2023-05-05.May 5 2023, 2:11 PM
aborrero added a comment.May 5 2023, 3:52 PM

What about increasing API HTTP workers for the most busy endpoints? @Andrew what do you think is the bottleneck here?

aborrero added a comment.May 10 2023, 2:39 PM

related: T336379: Openstack API slowdowns

aborrero reassigned this task from aborrero to Andrew.Jun 7 2023, 10:09 AM
aborrero added a project: User-aborrero.

hey @Andrew do you think we are ready to experiment with increasing the frequency again given T336379: Openstack API slowdowns is now completed?

aborrero added a comment.Jun 14 2023, 2:56 PM

WMCS meeting: try to have a clear view of API performance before turning this on again. This can be, for example, HAproxy backend response time.

aborrero mentioned this in T339152: Cloud VPS: figure out metrics for openstack API response times and create a grafana dashboard for it.Jun 14 2023, 4:02 PM
aborrero closed subtask T339152: Cloud VPS: figure out metrics for openstack API response times and create a grafana dashboard for it as Resolved.Jun 27 2023, 12:46 PM
gerritbot added a comment.Jun 27 2023, 12:48 PM

Change 933451 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] wmcs: prometheus: increase scrape frequency for openstack APIs

https://gerrit.wikimedia.org/r/933451

gerritbot added a project: Patch-For-Review.Jun 27 2023, 12:48 PM
gerritbot added a comment.Jun 27 2023, 2:25 PM

Change 933451 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] wmcs: prometheus: increase scrape frequency for openstack APIs

https://gerrit.wikimedia.org/r/933451

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2023, 2:29 PM
gerritbot added a comment.Jun 27 2023, 2:31 PM

Change 933477 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/alerts@master] team-wmcs: add openstack_apis_response.yaml

https://gerrit.wikimedia.org/r/933477

gerritbot added a project: Patch-For-Review.Jun 27 2023, 2:31 PM
gerritbot added a comment.Jun 28 2023, 2:34 PM

Change 933477 merged by Arturo Borrero Gonzalez:

[operations/alerts@master] team-wmcs: add openstack_apis_response.yaml

https://gerrit.wikimedia.org/r/933477

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2023, 3:11 PM
aborrero closed this task as Resolved.Jul 18 2023, 9:56 AM
fnegri moved this task from Backlog to Done on the cloud-services-team (FY2022/2023-Q4) board.Jul 27 2023, 3:18 PM
Andrew closed subtask T335978: openstack: consider removing references to old hardware from the database as Resolved.Wed, May 1, 4:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL