Page MenuHomePhabricator
Create Task
Maniphest T364889

Figure Out Why Wasmedge Subprocesses Aren't Always Killed; Kill Them
Open, In Progress, HighPublicBUG REPORT
Actions

Description

Description

Somehow, some wasmedge subprocesses keep running past their timeout. This should not be remotely possible. Why aren't they being killed/timed out?

Desired behavior/Acceptance criteria (returned value, expected error, performance expectations, etc.)

  • try to replicate this problem in a test environment
  • plug the leak

Completion checklist

Details

SubjectRepoBranchLines +/-
wikifunctions: Upgrade evaluators from 2024-05-13-145650 to 2024-05-28-185827operations/deployment-chartsmaster+2 -2
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Kill the wasmedge process directly, hopefully leaving no zombie processes.repos/abstract-wiki/wikifunctions/function-evaluator!216apineapine-kill-grandbabymain
Customize query in GitLab

Event Timeline

cmassaro renamed this task from Figure Out Why Wasmedge Subprocesses Aren't Always Killed to Figure Out Why Wasmedge Subprocesses Aren't Always Killed; Kill Them.Tue, May 14, 5:16 PM
cmassaro created this task.
Jdforrester-WMF changed the subtype of this task from "Task" to "Bug Report".Tue, May 14, 5:18 PM
Jdforrester-WMF triaged this task as High priority.Wed, May 15, 4:38 PM
Jdforrester-WMF moved this task from To triage to 24Q4 (Apr–Jun) on the Abstract Wikipedia team board.
Jdforrester-WMF edited projects, added Abstract Wikipedia team (24Q4 (Apr–Jun)); removed Abstract Wikipedia team.
Jdforrester-WMF moved this task from Incoming to Ready, Features on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.
cmassaro added a comment.Mon, May 20, 4:12 PM

I have replicated this locally. There are a few unfortunate things happening here. One is that our last-ditch effort (process.kill) still doesn't work in some cases, and every solution I've tried so far leaves us with zombie processes. It seems like the wasmedge environment does something that messes up process parent relationships, so I'll keep digging into that.

That said, it's also baffling that wasmedge's own resource limitations (which we recently added) aren't helping here. I'm also looking into that.

CodeReviewBot added a project: Patch-For-Review.Tue, May 21, 2:50 AM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/216

Kill the wasmedge process directly, hopefully leaving no zombie processes.

Jdforrester-WMF changed the task status from Open to In Progress.Tue, May 21, 7:13 PM
Jdforrester-WMF assigned this task to cmassaro.
Jdforrester-WMF removed cmassaro as the assignee of this task.
Jdforrester-WMF assigned this task to cmassaro.
Jdforrester-WMF moved this task from Ready, Features to In Progress on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.
CodeReviewBot added a comment.Tue, May 21, 10:04 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/216

Kill the wasmedge process directly, hopefully leaving no zombie processes.

Maintenance_bot removed a project: Patch-For-Review.Tue, May 21, 10:30 PM
cmassaro moved this task from In Progress to Ready to deploy on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.Wed, May 22, 3:10 PM
cmassaro added a comment.Wed, May 22, 4:22 PM

For what it's worth, I am not seeing this issue on Beta cluster

cmassaro added a comment.Wed, May 22, 4:28 PM
In T364889#9822313, @cmassaro wrote:

For what it's worth, I am not seeing this issue on Beta cluster

(I don't know if it was a problem before, to be clear, but it isn't a problem now)

gerritbot added a comment.Wed, May 29, 2:10 PM

Change #1037085 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-05-13-145650 to 2024-05-28-185827

https://gerrit.wikimedia.org/r/1037085

gerritbot added a project: Patch-For-Review.Wed, May 29, 2:10 PM
gerritbot added a comment.Wed, May 29, 2:21 PM

Change #1037085 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-05-13-145650 to 2024-05-28-185827

https://gerrit.wikimedia.org/r/1037085

Maintenance_bot removed a project: Patch-For-Review.Wed, May 29, 2:31 PM
Jdforrester-WMF moved this task from Ready to deploy to Needs Sign-off on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.Wed, May 29, 2:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL