Page MenuHomePhabricator
Create Task
Maniphest T348633

[api-gateway] add alert for uptime
Closed, ResolvedPublic
Actions

Description

Created as prompted by @taavi, the Toolforge API gateway (api.svc.tools.eqiad1.wikimedia.cloud) does not appear to be monitored for uptime.

Details

SubjectRepoBranchLines +/-
toolforge:haproxy: use the external name and ip and force tlsoperations/puppetproduction+4 -1
toolforge:haproxy: monitor the https port, not the internal oneoperations/puppetproduction+1 -9
toolforge:haproxy: add api gateway health checkoperations/puppetproduction+15 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 In ProgressdcaroT334240 [cloudceph] Slow operations - tracking task
 ResolvedtaaviT348634 ceph slow ops 2023-10-11
 ResolveddcaroT348633 [api-gateway] add alert for uptime

Event Timeline

TheresNoTime created this task.Oct 11 2023, 11:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2023, 11:49 AM
taavi added a project: cloud-services-team.Oct 11 2023, 11:51 AM
taavi added a parent task: T348634: ceph slow ops 2023-10-11.Oct 11 2023, 1:40 PM
dcaro renamed this task from Monitor the Toolforge API gateway to [api-gateway] add alert for uptime.Apr 18 2024, 9:16 AM
dcaro triaged this task as High priority.
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
dcaro moved this task from Ready to be worked on to Toolforge iteration 09 on the Toolforge board.May 22 2024, 1:48 PM
dcaro edited projects, added Toolforge (Toolforge iteration 09); removed Toolforge.
dcaro edited projects, added Toolforge (Toolforge iteration 10); removed Toolforge (Toolforge iteration 09).May 22 2024, 4:41 PM
dcaro edited projects, added Toolforge (Toolforge iteration 11); removed Toolforge (Toolforge iteration 10).Jun 4 2024, 2:57 PM
Slst2020 subscribed.Jun 13 2024, 7:52 AM

I'd like to do this if someone could guide me or point me to some similar past task

dcaro subscribed.Jun 13 2024, 10:20 AM

On one side you have to register the url to be added to the pingthing/blackbox monitoring config, same as these but for the new url:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/toolforge/k8s/haproxy.pp#52

That will add it to the monitored list, note that it will add also an alert by default, so that should be enough.

This task is related (not the same, not overridden by, just related) T367389: [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction

dcaro assigned this task to Slst2020.Jul 2 2024, 1:35 PM
dcaro edited projects, added Toolforge (Toolforge iteration 12); removed Toolforge (Toolforge iteration 11).Jul 2 2024, 1:59 PM
dcaro edited projects, added Toolforge (Toolforge iteration 13); removed Toolforge (Toolforge iteration 12).Jul 17 2024, 11:23 AM
dcaro removed Slst2020 as the assignee of this task.Jul 30 2024, 1:57 PM
dcaro edited projects, added Toolforge (Toolforge iteration 14); removed Toolforge (Toolforge iteration 13).Aug 5 2024, 7:40 AM
dcaro edited projects, added Toolforge (Toolforge iteration 15); removed Toolforge (Toolforge iteration 14).Sep 25 2024, 8:01 AM
dcaro edited projects, added Toolforge (Toolforge iteration 16); removed Toolforge (Toolforge iteration 15).Oct 15 2024, 7:33 AM
dcaro claimed this task.Tue, Oct 29, 2:39 PM
CodeReviewBot added a project: Patch-For-Review.Wed, Nov 20, 1:53 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/53

api: expose the healthz endpoint too

gerritbot added a comment.Wed, Nov 20, 1:58 PM

Change #1093339 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] toolforge:haproxy: add api gateway health check

https://gerrit.wikimedia.org/r/1093339

dcaro changed the task status from Open to In Progress.Wed, Nov 20, 2:02 PM
dcaro moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 16) board.
dcaro moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 16) board.Wed, Nov 20, 2:19 PM
CodeReviewBot added a comment.Wed, Nov 20, 2:44 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/53

api: expose the healthz endpoint too

CodeReviewBot added a comment.Wed, Nov 20, 2:47 PM

group_203_bot_4866fc124f4b41659f667468a6115cf3 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/616

api-gateway: bump to 0.0.56-20241120144516-f10abf2a

CodeReviewBot added a comment.Wed, Nov 20, 3:17 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/616

api-gateway: bump to 0.0.56-20241120144516-f10abf2a

gerritbot added a comment.Wed, Nov 20, 3:17 PM

Change #1093339 merged by David Caro:

[operations/puppet@production] toolforge:haproxy: add api gateway health check

https://gerrit.wikimedia.org/r/1093339

dcaro closed this task as Resolved.Wed, Nov 20, 3:18 PM
dcaro moved this task from In Progress to Done on the Toolforge (Toolforge iteration 16) board.
Maintenance_bot removed a project: Patch-For-Review.Wed, Nov 20, 3:30 PM
dcaro reopened this task as In Progress.Wed, Nov 20, 4:54 PM
dcaro moved this task from Done to In Review on the Toolforge (Toolforge iteration 16) board.
dcaro moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 16) board.
gerritbot added a comment.Wed, Nov 20, 4:55 PM

Change #1093384 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] toolforge:haproxy: monitor the https port, not the internal one

https://gerrit.wikimedia.org/r/1093384

gerritbot added a project: Patch-For-Review.Wed, Nov 20, 4:55 PM
gerritbot added a comment.Wed, Nov 20, 5:29 PM

Change #1093384 merged by David Caro:

[operations/puppet@production] toolforge:haproxy: monitor the https port, not the internal one

https://gerrit.wikimedia.org/r/1093384

Maintenance_bot removed a project: Patch-For-Review.Wed, Nov 20, 5:30 PM
gerritbot added a comment.Wed, Nov 20, 5:50 PM

Change #1093395 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] toolforge:haproxy: use the external name and force tls

https://gerrit.wikimedia.org/r/1093395

gerritbot added a project: Patch-For-Review.Wed, Nov 20, 5:50 PM
gerritbot added a comment.Wed, Nov 20, 6:10 PM

Change #1093395 merged by David Caro:

[operations/puppet@production] toolforge:haproxy: use the external name and ip and force tls

https://gerrit.wikimedia.org/r/1093395

dcaro closed this task as Resolved.Wed, Nov 20, 6:20 PM
dcaro moved this task from In Progress to Done on the Toolforge (Toolforge iteration 16) board.
Maintenance_bot removed a project: Patch-For-Review.Wed, Nov 20, 6:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits