Page MenuHomePhabricator
Create Task
Maniphest T388804

Update api-gateway ratelimit version
Closed, ResolvedPublic
Actions

Description

The API gateway's version of the ratelimit service is very old (using the 1.5.1 branch). We have started to see issues where the service is throwing 5xx errors and there is very little way to debug why this is happening. We're also seeing this manifesting when other services are impaired, which is confusing and a bad signal, especially when it pages.

We should update the service to a recent version, and as part of this migration we should switch to using the new prometheus metrics it offers. In addition we can remove the statsd gateway as part of this work.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
api-gateway: use ratelimit's inbuilt promethus-statsd agentoperations/deployment-chartsmaster+116 -75
ratelimit: bump version numberoperations/docker-images/production-imagesmaster+7 -1
api-gateway: use latest build of ratelimit service in prodoperations/deployment-chartsmaster+1 -2
api-gateway: use more recent ratelimit imageoperations/deployment-chartsmaster+7 -2
Customize query in gerrit

Related Objects

Event Timeline

hnowlan created this task.Mar 13 2025, 3:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2025, 3:20 PM
Kappakayala assigned this task to • Jasmine.Mar 13 2025, 5:53 PM
jasmine_ claimed this task.Mar 13 2025, 10:13 PM
jasmine_ added a subscriber: • Jasmine.
jasmine_ removed a subscriber: • Jasmine.
hnowlan added a comment.Mar 27 2025, 4:53 PM

Related - it'd be nice if this work could get us some logging enhancements. We saw in T390215 that the current version's debug logging (which is the only log level that we can currently use to get useful output) was crushing logstash, and the default log levels are far too quiet when it comes to failures to connect etc, so if there have been improvements in the last 4 (!) years that would be nice.

gerritbot added a comment.Jul 1 2025, 9:26 AM

Change #1165475 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] api-gateway: use more recent ratelimit image

https://gerrit.wikimedia.org/r/1165475

gerritbot added a project: Patch-For-Review.Jul 1 2025, 9:26 AM
gerritbot added a comment.Jul 2 2025, 10:28 AM

Change #1165475 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: use more recent ratelimit image

https://gerrit.wikimedia.org/r/1165475

Maintenance_bot removed a project: Patch-For-Review.Jul 2 2025, 10:31 AM
gerritbot added a comment.Jul 3 2025, 3:28 PM

Change #1166221 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] api-gateway: use latest build of ratelimit service in prod

https://gerrit.wikimedia.org/r/1166221

gerritbot added a project: Patch-For-Review.Jul 3 2025, 3:28 PM
gerritbot added a comment.Jul 3 2025, 3:43 PM

Change #1166221 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: use latest build of ratelimit service in prod

https://gerrit.wikimedia.org/r/1166221

Maintenance_bot removed a project: Patch-For-Review.Jul 3 2025, 4:30 PM
gerritbot added a comment.Jul 4 2025, 3:39 PM

Change #1166412 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/docker-images/production-images@master] ratelimit: bump version number

https://gerrit.wikimedia.org/r/1166412

gerritbot added a project: Patch-For-Review.Jul 4 2025, 3:39 PM
hnowlan mentioned this in rOSER1bef32adea18: Vendor dependencies.Jul 4 2025, 8:47 PM
gerritbot added a comment.Jul 7 2025, 10:37 AM

Change #1166790 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] api-gateway: use ratelimit's inbuilt promethus-statsd agent

https://gerrit.wikimedia.org/r/1166790

gerritbot added a comment.Jul 7 2025, 1:57 PM

Change #1166412 merged by Hnowlan:

[operations/docker-images/production-images@master] ratelimit: bump version number

https://gerrit.wikimedia.org/r/1166412

gerritbot added a comment.Jul 8 2025, 11:45 AM

Change #1166790 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: use ratelimit's inbuilt promethus-statsd agent

https://gerrit.wikimedia.org/r/1166790

hnowlan added a comment.Jul 8 2025, 12:22 PM

We're now using the latest HEAD of the ratelimit service on bullseye and have removed the prometheus-statsd-exporter from the api-gateway deployment.

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2025, 12:30 PM
hnowlan closed this task as Resolved.Dec 10 2025, 11:38 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits