From #wikimedia-releng:

<thcipriani> yes...there are a massive number of ffmpeg processes running on integration-agent-docker-1048
<thcipriani> load average: 232.20, 234.51, 231.36

So, I killed both running jobs for https://gerrit.wikimedia.org/r/c/mediawiki/core/+/721790, which runs selenium tests 100 times to find flaky tests. Normally, that kind of patch would not be a problem for the CI infrastructure. However, this particular patch is dependent on r1002578, which bumps wdio to version 8 (T324766). Apparently, the new version of wdio breaks our screen recording code and leaves the ffmpeg processes hanging?

Mentioned in SAL (#wikimedia-releng) [2025-04-29T20:28:19Z] <mutante> integration-agent-docker-1048.integration - killall -9 ffpmeg - T392963

Mentioned in SAL (#wikimedia-releng) [2025-04-29T21:04:51Z] <mutante> integration-agent-docker-1051.integration - killall -9 ffmpeg - T392963

FTR, this is being discussed in IRC: https://wm-bot.wmcloud.org/browser/index.php?start=04%2F29%2F2025&end=04%2F29%2F2025&display=%23wikimedia-releng

Things to look into:

• Is the theory in T392963#10778330 correct?
  • How to get a list of builds for a given executor? https://integration.wikimedia.org/ci/computer/integration%2Dagent%2Ddocker%2D1048/builds is a 403 for me
• If it is correct: why does a single agent bring everything down?
• (maybe) Is there a way to improve failure modes? Maybe print a message that explicitly says that the job was killed due to insufficient resources?
• Some unresponsiveness when aborting jobs via the UI: for example, I clicked "abort" for https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php74/25384/console before 20:39:33Z, but that did nothing, and the job failed with java exceptions two minutes later.

Change #1139943 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] selenium: set explicit timeout when spawning ffmpeg

https://gerrit.wikimedia.org/r/1139943

In T392963#10778428, @Daimona wrote:

• Is the theory in T392963#10778330 correct?

Based on what we know now: the shell loops triggering 100x selenium runs used in r721790 outlived the container for some reason. They accumulated and kept spawning ffmpeg processes, which are also executed outside the container. This already gives us a couple action items.

• How to get a list of builds for a given executor? https://integration.wikimedia.org/ci/computer/integration%2Dagent%2Ddocker%2D1048/builds is a 403 for me

I didn't figure this out, but it's not so important now.

• If it is correct: why does a single agent bring everything down?

I think it's not a single agent, but rather a bunch of agents that were affected. All new builds on those agents failed.

• (maybe) Is there a way to improve failure modes? Maybe print a message that explicitly says that the job was killed due to insufficient resources?
• Some unresponsiveness when aborting jobs via the UI: for example, I clicked "abort" for https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php74/25384/console before 20:39:33Z, but that did nothing, and the job failed with java exceptions two minutes later.

Still valid.

We saw ffmpeg processes come back after being killed.. then figure out there were shell loops (sh -c for i in ...) as shown below on instances. In most cases one but at least one of them had 2 running at the same time.

Then killed these loops and node processes on a couple more instances.

It's unclear if we can see just from the command lines if it's using "wdio 8" or not but it looked like this: (and on at least one instance this but 2 loops and double the node processes).

nobody   1964455  0.0  0.0   2484   100 ?        S    16:39   0:00 sh -c for i in $(seq 1 100); do wdio ./tests/selenium/wdio.conf.js; done
nobody   2197070  2.7  0.5 1288868 130420 ?      Sl   22:27   0:01 node /workspace/src/node_modules/.bin/wdio ./tests/selenium/wdio.conf.js
nobody   2197111  3.4  0.5 1293564 140108 ?      Sl   22:27   0:02 /usr/bin/node /workspace/src/node_modules/@wdio/local-runner/build/run.js ./tests/selenium/wdio.conf.js
nobody   2197114  3.0  0.5 1289972 130044 ?      Sl   22:27   0:01 /usr/bin/node /workspace/src/node_modules/@wdio/local-runner/build/run.js ./tests/selenium/wdio.conf.js

Where ffmpeg gets spawned:

https://gerrit.wikimedia.org/g/mediawiki/core/+/d3090254b0e8b2284b100d77e32c18155df75f0a/tests/selenium/wdio-mediawiki/index.js#65

code change that adds the "seq 1 100" loop but is NOT MERGED?!

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/721790/25/package.json

In T392963#10778640, @Dzahn wrote:

We saw ffmpeg processes come back after being killed.. then figure out there were shell loops (sh -c for i in ...) as shown below on instances. In most cases one but at least one of them had 2 running at the same time.

We run selenium tests for both php81 and php74 so that would explain why there would be two shell loops on one instance.

Then killed these loops and node processes on a couple more instances.

It's unclear if we can see just from the command lines if it's using "wdio 8" or not but it looked like this: (and on at least one instance this but 2 loops and double the node processes).

nobody   1964455  0.0  0.0   2484   100 ?        S    16:39   0:00 sh -c for i in $(seq 1 100); do wdio ./tests/selenium/wdio.conf.js; done
nobody   2197070  2.7  0.5 1288868 130420 ?      Sl   22:27   0:01 node /workspace/src/node_modules/.bin/wdio ./tests/selenium/wdio.conf.js
nobody   2197111  3.4  0.5 1293564 140108 ?      Sl   22:27   0:02 /usr/bin/node /workspace/src/node_modules/@wdio/local-runner/build/run.js ./tests/selenium/wdio.conf.js
nobody   2197114  3.0  0.5 1289972 130044 ?      Sl   22:27   0:01 /usr/bin/node /workspace/src/node_modules/@wdio/local-runner/build/run.js ./tests/selenium/wdio.conf.js

wdio is still 7 everywhere that it's merged: see codesearch

So bumping to wdio8 would have to be a patch.

The only way I know of to trace these back to a patch is by finding out the workspace mounted in the container and looking at the change at the top in the src directory of that workspace.

So, in this case, using ps faux (or something else) to find the docker container that initiated pid 1964455 (from the output above), then ls -lA /proc/<parent-pid>/cwd to find that workspace directory. Then git -C <workspace-directory>/src show HEAD should get you the patch.

There is probably an easier way to do that, but that's the way I know how to do it.

In T392963#10778630, @Daimona wrote:

In T392963#10778428, @Daimona wrote:

• How to get a list of builds for a given executor? https://integration.wikimedia.org/ci/computer/integration%2Dagent%2Ddocker%2D1048/builds is a 403 for me

I didn't figure this out, but it's not so important now.

There's not a good way to do this from the web UI, IIRC that's a deliberate 403 due to some kind of abuse thing that I've lost all memory of.

Here's a oneliner that should do it: curl -s "https://integration.wikimedia.org/ci/api/json?tree=jobs\[name,url,builds\[id,url,timestamp,duration,builtOn\]\]" | jq -r '.jobs[] | .builds[] | select(.builtOn != null) | "\(.builtOn)\t\(.url)"' | sort

In T392963#10778630, @Daimona wrote:

Based on what we know now: the shell loops triggering 100x selenium runs used in r721790 outlived the container for some reason. They accumulated and kept spawning ffmpeg processes, which are also executed outside the container. This already gives us a couple action items.

There are multiple issues happening and the short answer is Docker is a client/server application.

When a signal is sent to the client (docker run), some old versions of Docker did not relay the signals to the daemon (I did report it with an extensive report to upstream and based on my analysis that eventually got solved at some point). If a KILL signal is sent to the client, the process is terminated immediately by the kernel and the signal is not relayed.

The container might have bash running which happily ignores the TERM signal and does not relay it back. Then any running code can choose to catch the TERM signal and ignore it, so that is not a reliable way to terminate the processes.

When Jenkins aborts a job, it sends a TERM signal to the process group (which in our case is solely docker run) and after some seconds (15 IIRC) a KILL signal to the whole process group. But that certainly leaves containers running, we would need Jenkins to be aware of the containers and terminate them using the client (docker stop / docker kill).

Anyway, all of that is moot if we switch to one off virtual machines that are entirely reclaimed at the end of each build. That is the model we had for some time (with Nodepool) and I hope to adopt again as we upgrade Zuul/CI :)

@Daimona have you filed the two action items you mention? Or maybe it is on another task I might have missed. Maybe docker run can be given limits to the number of process/memory/number of CPUs.

In T392963#10778692, @thcipriani wrote:

In T392963#10778630, @Daimona wrote:

In T392963#10778428, @Daimona wrote:

• How to get a list of builds for a given executor? https://integration.wikimedia.org/ci/computer/integration%2Dagent%2Ddocker%2D1048/builds is a 403 for me

I didn't figure this out, but it's not so important now.

There's not a good way to do this from the web UI, IIRC that's a deliberate 403 due to some kind of abuse thing that I've lost all memory of.

Here's a oneliner that should do it: curl -s "https://integration.wikimedia.org/ci/api/json?tree=jobs\[name,url,builds\[id,url,timestamp,duration,builtOn\]\]" | jq -r '.jobs[] | .builds[] | select(.builtOn != null) | "\(.builtOn)\t\(.url)"' | sort

We deny requests to the history of builds. The reason is that the build of history of jobs is kept as XML files, stored per job, one per build . When Jenkins want to retrieve the history of a specific job, it crawls the directory for that job. If one asks for builds that happened on jobs, Jenkins must process the whole build history which might be hundred of thousands of XML files to read. So the Jenkins controller crawls to its death. More or less, I haven't tried in a while and maybe that is less of an issue if the builds are lazy loaded on start and new builds kept in some memory cache.

In T392963#10779109, @hashar wrote:

In T392963#10778692, @thcipriani wrote:

There's not a good way to do this from the web UI, IIRC that's a deliberate 403 due to some kind of abuse thing that I've lost all memory of.

Here's a oneliner that should do it: curl -s "https://integration.wikimedia.org/ci/api/json?tree=jobs\[name,url,builds\[id,url,timestamp,duration,builtOn\]\]" | jq -r '.jobs[] | .builds[] | select(.builtOn != null) | "\(.builtOn)\t\(.url)"' | sort

We deny requests to the history of builds. The reason is that the build of history of jobs is kept as XML files, stored per job, one per build . When Jenkins want to retrieve the history of a specific job, it crawls the directory for that job. If one asks for builds that happened on jobs, Jenkins must process the whole build history which might be hundred of thousands of XML files to read. So the Jenkins controller crawls to its death. More or less, I haven't tried in a while and maybe that is less of an issue if the builds are lazy loaded on start and new builds kept in some memory cache.

Thanks both! This makes sense, and I'm under the impression that this was recently discussed elsewhere, but sadly I can't find that conversation.

In T392963#10779109, @hashar wrote:

@Daimona have you filed the two action items you mention? Or maybe it is on another task I might have missed.

I didn't :) One would be for the MW selenium setup to make sure that ffmpeg is terminated. This is now tracked in T392974 that Peter filed. The other action item I had in mind would be to figure out why the shell loop (and thus the node processes, and thus ffmpeg) consistently outlived the container. I think you already gave a good explanation of that! So, my question would be:

we would need Jenkins to be aware of the containers and terminate them using the client (docker stop / docker kill).

Anyway, all of that is moot if we switch to one off virtual machines that are entirely reclaimed at the end of each build. That is the model we had for some time (with Nodepool) and I hope to adopt again as we upgrade Zuul/CI :)

Is there a task to track this (either making jenkins aware of containers or switching to one-off VMs)?

Maybe docker run can be given limits to the number of process/memory/number of CPUs.

That sounds sensible, in principle.

In T392963#10778428, @Daimona wrote:

• (maybe) Is there a way to improve failure modes? Maybe print a message that explicitly says that the job was killed due to insufficient resources?
• Some unresponsiveness when aborting jobs via the UI: for example, I clicked "abort" for https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php74/25384/console before 20:39:33Z, but that did nothing, and the job failed with java exceptions two minutes later.

Do you feel like there's anything that can be done about these? Maybe some low-hanging fruit?

Is there a task to track this (either making jenkins aware of containers or switching to one-off VMs)?

When the container is terminated, all the contained processes should have terminated. But if the container is never stopped, I imagine they they stay behind. Then I am not sure what happened with ffmpeg :-/ The future is to upgrade Zuul and change the execution model (Continuous-Integration-Infrastructure (Zuul upgrade)), potentially using one of VM, starting builds with a clean environment and removing it entirely is definitely nicer.

Overall, I would not spend much time on the current Zuul/Jenkins system given the aim is to replace it :)