Page MenuHomePhabricator
Create Task
Maniphest T380832

[jobs-api] crashing
Open, LowPublic
Actions

Description

This is also maybe (or maybe not) related to the NFS outage; it started crashing in earnest after I started rebooting nfs k8s worker nodes.

@Slst2020 is bailing me out now, but I was very confused by the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Jobs_framework that don't really explain where/how the api is run. I'm not clear on if those docs are just really out of date or if I'm badly misreading. For instance 'jobs-framework-api (code) --- uses flask-restful and runs inside the k8s cluster as a webservice' sent me to the 'jobs' tool but that seems to be unrelated?

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT380882 openstack network problems (November 2024)
 ResolvedaborreroT380827 tools-nfs outage 2024-11-25
 OpenNoneT380832 [jobs-api] crashing
 OpenNoneT380959 [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components
 ResolveddcaroT325172 [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service
 ResolvedNoneT325174 [builds-builder,harbor,bulid-service,docs] user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing component and how to get the logs for each of them.
 ResolveddcaroT325166 tbs: user-story 10: I want to know how to manage the service
 ResolveddcaroT325167 tbs: user-story 10: Create admin wiki page for the toolforge build service
 ResolvedRaymond_NdibeT325175 tbs: user-story 11: Add a runbook for each of the service alerts.
 ResolvedRaymond_NdibeT325160 tbs: user-story 9: I want to know when the service is down
 ResolvedRaymond_NdibeT325162 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on toolsbeta
 ResolvedRaymond_NdibeT325163 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on tools
 ResolvedRaymond_NdibeT325164 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on tools
 ResolvedRaymond_NdibeT325165 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on toolsbeta
 ResolveddcaroT330096 tbs: user-story 9: Gather statistics for harbor uptime in toolsbeta
 ResolveddcaroT330757 [tbs.harbor] Harbor core being down is not noticed by simple http request
 In ProgressdcaroT320284 [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components
 In ProgressdcaroT379924 [jobs-emailer] http requests are blocked by the loops

Event Timeline

Andrew created this task.Tue, Nov 26, 7:23 AM
Andrew added a comment.Tue, Nov 26, 7:35 AM

This seems to be resolved now, pending questions are:

  • why no alerts?
  • are the docs as wrong as the look to me at 2AM?
dcaro subscribed.Tue, Nov 26, 7:57 AM
Count_Count subscribed.Tue, Nov 26, 9:37 AM
aborrero triaged this task as Low priority.Tue, Nov 26, 12:01 PM
In T380832#10356349, @Andrew wrote:

This seems to be resolved now, pending questions are:

  • why no alerts?

Were jobs-api pods crashing? I think the monitoring for it was recently reviewed in T320284: [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components.

I think the actual problem was T380844: 2024-11-26 Toolforge DNS incident.

  • are the docs as wrong as the look to me at 2AM?

I think the docs you selected are 'how this works' kind of document, and not a 'runbook' kind of document.

The page has been refreshed just now, but the content remains in the same tone 'how this works'.

Maybe we need to:

  • review and create 'runbook' kind of documents for Toolforge components
  • have a round of training about Toolforge components
aborrero added a project: User-aborrero.Tue, Nov 26, 12:01 PM
aborrero moved this task from Backlog to Radar/observer on the User-aborrero board.
Slst2020 added a comment.Tue, Nov 26, 12:23 PM
In T380832#10357169, @aborrero wrote:

I think the actual problem was T380844: 2024-11-26 Toolforge DNS incident.

Yes, I can confirm this:

sed by NameResolutionError(
    "<urllib3.connection.HTTPSConnection object at 0x7f60e6211a50>: 
    Failed to resolve 'tools-harbor.wmcloud.org' 
    ([Errno -3] Temporary failure in name resolution)"
)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/app/tjf/api/jobs.py", line 110, in create_job
    new_job = NewJob.model_validate(request.json)
dcaro added a comment.EditedWed, Nov 27, 8:43 AM

why no alerts?

Patch for the alerts https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/20

For some reason I was expecting to get an alert for any scrape target being down, but that's only on prod alertmanager I think, this adds it explicitly.

Added a runbook for each alert too (that helps a bit with the docs side, giving an idea on how to check the pods, get the logs, ...)

aborrero mentioned this in T380959: [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components.Wed, Nov 27, 10:09 AM
dcaro renamed this task from jobs-api crashing to [jobs-api] crashing.Wed, Nov 27, 2:20 PM
dcaro edited projects, added Toolforge (Toolforge iteration 16); removed Toolforge.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits