Hello @Mainframe98 you are right, the link pointing to error was pointing to a previous issue, i corrected the link. Sorry for confusion.

The check seems to be stuck or abort here:

12:26:45 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 54 / 115 ( 69%) 3015MB

Looking at the error log the following line seems suspect:

15:21:11 ░░░░░░░░░░░░░░░░░░░░░░░░░░/run.sh: line 5:     8 Killed                  /run-generic.sh "/mediawiki/$THING_SUBNAME" "$@"

It suggests that, according to https://stackoverflow.com/questions/726690/what-killed-my-process-and-why, there is a memory issue. Comparing other runs of mwext-php74-phan-docker, they all finish before hitting the 3GB.

Previously, Parsoid had this issue, (T221872). The solution was to narrow what phan scans (https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/513359/) and increase the job's memory.

I can reproduce this locally, phan just gets stuck at some point with high memory usage. Can only reproduce with taint-check enabled. Maybe it's trying to merge super-complex array shapes, that has caused similar issues in the past. Annotating some parameters/return values as safe may help, but investigating taint-check issues is very time-consuming...

We had the issue with Phan T219114#5084302 . It triggers a fork() when writing the process bar, then Linux checks whether the fork can use the same amount of memory and if there is not enough bails out:

/srv/phan/vendor/symfony/console/Terminal.php:127 [2] proc_open(): fork failed - Cannot allocate memory

Maybe that is the same issue?, at the time I went with three potential solutions:

1. Set sysctl vm.overcommit_memory = 1, which is global to the instance and can have side effects.
2. Set COLUMNS=80 on CI to bypass Symfony\console\Terminal::getWidth() heavy logic
3. Disable the progress bar when not running interactively. I have not looked at how that is done in Phan.

From a quick glance at T219114, we first disabled the progress bar and then went to set COLUMNS=80 (which at the time caused Phan to not have to fork to stty to retrieve the number of columns).

Does it happen everywhere or "just" on the Math extension?

Adding

/**
 * @param-taint $mhchemParser none
 */

to MhchemStateMachines::__construct fixes this for me locally. Taint-check simply can't cope with the huge $stateMachines array, and it just goes OOM when trying to infer its shape.

13:21:11 ░░░░░░░░░░░░░░░░░░░░░░░░░░/run.sh: line 5:     8 Killed                  /run-generic.sh "/mediawiki/$THING_SUBNAME" "$@"

From the instance dmesg -T:

[Wed Jul 19 13:19:25 2023] php invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
...
[Wed Jul 19 13:19:25 2023] Tasks state (memory values in pages):
[Wed Jul 19 13:19:25 2023] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Wed Jul 19 13:19:25 2023] [4058861] 65534 4058861  6552646  5877371 47513600        0             0 php
...
[Wed Jul 19 13:19:25 2023] Out of memory: Killed process 4058861 (php) total-vm:26210584kB, anon-rss:23509480kB, file-rss:4kB, shmem-rss:0kB, UID:65534 pgtables:46400kB oom_score_adj:0

For reference, the instances have 24G of RAM while the php process had a 22.4gB anon-rss. One can try sending a dummy change to Math to see whether that is related to this change, else I guess the entrypoint in composer.json can be changed to remove the progress bar and maybe that will solves it.

Change 939713 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: limit Phan memory usage to 7G

https://gerrit.wikimedia.org/r/939713

Mentioned in SAL (#wikimedia-releng) [2023-07-19T14:45:15Z] <hashar> Updating all Jenkins phan jobs to use docker run --memory 7G # T342100

Change 939713 merged by jenkins-bot:

[integration/config@master] jjb: limit Phan memory usage to 7G

https://gerrit.wikimedia.org/r/939713

With the docker memory limit of 7G the process still get killed:

14:47:28 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  54 / 115 ( 69%) 3016MB
14:51:04 ░░░░░░░░░░░░░░░░░░░░░░░░░░/run.sh: line 5:     8 Killed                  /run-generic.sh "/mediawiki/$THING_SUBNAME" "$@"
14:51:05 Build step 'Execute shell' marked build as failure

The limit is implemented via a cgroup and the Linux Kernel out of memory manager does kill it at 7G:

[Wed Jul 19 14:49:15 2023] php invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0

[Wed Jul 19 14:49:15 2023] memory: usage 7340032kB, limit 7340032kB, failcnt 116

[Wed Jul 19 14:49:15 2023] Memory cgroup out of memory: Killed process 162507 (php) total-vm:7959572kB, anon-rss:7324080kB, file-rss:27744kB, shmem-rss:0kB, UID:65534 pgtables:14600kB oom_score_adj:0

Thanks for the quick help !