Page MenuHomePhabricator
Create Task
Maniphest T251917

Design the Jobs service in k8s
Closed, ResolvedPublic
Actions

Description

We've found that specific settings and possible even alpha features (ttlSecondsAfterFinished, specifically) are needed to make the jobs/cronjobs work well.

We likely also need some tooling (new backend for jsub or a new command)!

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
jobs-framework-api: include deployment configuration for toolsbeta kubernetescloud/toolforge/jobs-framework-apimain+425 -0
jobs-framework-api: introduce initial docker imagecloud/toolforge/jobs-framework-apimain+10 -1
toolforge: nginx-ingress-jobs: specify ingress-classoperations/puppetproduction+2 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• JHeddenT251027 "signatures" tool has failed job pods on Kubernetes cluster
 Resolved• aborreroT251917 Design the Jobs service in k8s
 Resolved• aborreroT274123 introduce TLS support in toolsbeta front proxy
 Resolved• aborreroT274139 Decide TLS auth proxy method for the new toolforge jobs framework
 Resolved• BstormT275478 toolsbeta: ingress admission controller doesn't accept valid FQDN patterns
 Resolved• aborreroT283238 Toolforge: develop jobs-framework-api
 Resolved• aborreroT285944 Toolforge: beta phase for the new jobs framework
 ResolvedJJMC89T285961 jobs-framework-cli: display something when involved with no arguments
 Resolved• aborreroT285963 jobs framework should not error out when executed by a non-tool account
 ResolvedBUG REPORT• aborreroT285979 toolforge-jobs with --wait hangs indefinitely if the job fails
 ResolvedBUG REPORT• aborreroT286107 toolforge-jobs: Allow specifying arguments to commands
 ResolvedBUG REPORT• aborreroT286108 toolforge-jobs: Clean up old individual job objects
 ResolvedFeature• aborreroT286126 toolforge-jobs: allow setting limits and requests
 ResolvedBUG REPORT• aborreroT286132 toolforge-jobs: "Status: Unknown" when job is running
 ResolvedFeature• aborreroT286135 Toolforge jobs framework: email maintainers on job failure
 Resolved• aborreroT286485 toolforge-jobs: figure out logging
Resolved• aborreroT286492 toolforge-jobs: load jobs from a file
 Resolved• BstormT286784 toolforge-jobs: figure out default quotas and limits
 InvalidBUG REPORTNoneT286824 toolforge-jobs cli: commands are missing or truncated in list/show output
 ResolvedFeature• aborreroT287077 toolforge-jobs: Indicate which containers are deprecated
 ResolvedtaaviT287275 toolforge-jobs: reject jobs with more resource requests than single pods can use
 ResolvedBUG REPORTSo9qT299039 All started jobs failed on Kubernetes during 24h with no visible error or output
 Resolved• aborreroT301665 Toolforge jobs framework: create documentation on wikitech
 ResolvedFeatureRaymond_NdibeT301901 Allow specifying the path for log files for jobs executed on the new toolforge Jobs framework
 ResolvedFeatureRaymond_NdibeT302211 toolforge-jobs: merge stdout/stderr output
 DuplicateNoneT302988 Toolforge kubernetes and pywikibot
 DuplicateFeatureRaymond_NdibeT304421 Allow customizing the out/err files with toolforge-jobs
 ResolvedBUG REPORTJJMC89T304861 Typo in toolforge-jobs output
 ResolvedBUG REPORTRaymond_NdibeT304893 Rethink job retries in case of failures
 ResolvedBUG REPORT• aborreroT304900 toolforge-jobs should properly process 'out of quota' errors
 ResolvedBUG REPORTtaaviT305592 Underscore in job name gives non-helpful error in toolforge-jobs
 ResolvedFeaturetaaviT306311 toolforge-jobs should have a method to easily restart a specific job
 ResolvedFeaturetaaviT306312 toolforge-jobs should have a method to load a single task (or a subset of tasks) from a yaml file
 DuplicateBUG REPORTNoneT306454 toolforge-jobs rejects valid schedule time
 DuplicateFeatureNoneT307486 add mysql to toolforge-bullseye-standalone
 ResolvedtaaviT309146 Perl Docker image not available for Toolforge Kubernetes jobs
 ResolvedRaymond_NdibeT315114 Make it possible to configure retry policy for jobs executed on the toolforge jobs framework
 Resolved• aborreroT327278 Toolforge jobs framework: define end of beta phase

Event Timeline

• Bstorm created this task.May 5 2020, 4:25 PM
• Bstorm removed • JHedden as the assignee of this task.
• Bstorm triaged this task as High priority.
• Bstorm removed a project: Tools.
• Bstorm edited subscribers, added: • JHedden; removed: AntiCompositeNumber.
• Bstorm updated the task description. (Show Details)May 5 2020, 6:26 PM

The conversation in T251027: "signatures" tool has failed job pods on Kubernetes cluster is a good touchpoint for this ticket.

Probably the most simple note here is that batch objects don't appear to be quota-able directly (just the pods they schedule), which may or may not merit controllers to do just that.

AntiCompositeNumber subscribed.May 5 2020, 11:19 PM
• nskaggs moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.Sep 8 2020, 4:34 PM
• Bstorm assigned this task to • aborrero.Feb 1 2021, 5:53 PM
• Bstorm added a subscriber: • aborrero.

I think you are working on this @aborrero, so assigning to you to prevent duplication. Feel free to delete if you have another one.

• Bstorm mentioned this in T273487: Confusion about the os & package versions used on Toolforge bastions & grid vs Kubernetes containers.Feb 1 2021, 6:03 PM
• aborrero added a comment.Feb 3 2021, 9:59 AM

design document here: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_jobs

• aborrero moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.Feb 3 2021, 9:59 AM
taavi subscribed.Feb 8 2021, 11:54 AM
• aborrero closed subtask T274139: Decide TLS auth proxy method for the new toolforge jobs framework as Resolved.Feb 23 2021, 10:45 AM
• aborrero closed subtask T274123: introduce TLS support in toolsbeta front proxy as Resolved.Apr 20 2021, 11:10 AM
• aborrero closed subtask T275478: toolsbeta: ingress admission controller doesn't accept valid FQDN patterns as Resolved.Apr 20 2021, 3:36 PM
gerritbot added a comment.Apr 20 2021, 3:51 PM

Change 681424 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-api@main] tjf: include deployment configuration for toolsbeta kubernetes

https://gerrit.wikimedia.org/r/681424

gerritbot added a project: Patch-For-Review.Apr 20 2021, 3:51 PM
• aborrero added a comment.Apr 20 2021, 4:06 PM
NOTE: https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/
taavi added a comment.Apr 20 2021, 4:08 PM
In T251917#7021004, @aborrero wrote:
NOTE: https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

Just a note, that will have significant changes with Kubernetes 1.18/1.19.

gerritbot added a comment.Apr 21 2021, 5:12 PM

Change 681725 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] toolforge: nginx-ingress-jobs: specify ingress-class

https://gerrit.wikimedia.org/r/681725

gerritbot added a comment.May 18 2021, 3:56 PM

Change 692633 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-api@main] jobs-framework-api: introduce initial docker image

https://gerrit.wikimedia.org/r/692633

gerritbot added a comment.May 18 2021, 3:58 PM

Change 692633 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-api@main] jobs-framework-api: introduce initial docker image

https://gerrit.wikimedia.org/r/692633

gerritbot added a comment.May 18 2021, 3:58 PM

Change 681424 merged by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-api@main] jobs-framework-api: include deployment configuration for toolsbeta kubernetes

https://gerrit.wikimedia.org/r/681424

• aborrero mentioned this in rCTJF002bbc26d8de: jobs-framework-api: introduce initial docker image.May 18 2021, 4:51 PM
• aborrero mentioned this in rCTJF8e5c8e01b1c0: jobs-framework-api: include deployment configuration for toolsbeta kubernetes.
• aborrero closed this task as Resolved.May 20 2021, 11:37 AM

Closing this task as the design phase is mostly completed. Now moving on to the devel phase.

• aborrero closed subtask T283238: Toolforge: develop jobs-framework-api as Resolved.Jul 1 2021, 12:09 PM
• aborrero merged a task: T254617: Create a command in Toolforge to run scripts in kubernetes as easy as use jsub to run in grid engine.Sep 6 2021, 12:06 PM
• aborrero added subscribers: Danilo, zhuyifei1999.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits