Page MenuHomePhabricator

ECONNREFUSED error when running Selenium tests on M1 Mac
Open, Needs TriagePublicBUG REPORT

Description

What happens?:

I used what I believe to be the latest fresh:

# fresh: 22.05.1
# image: docker-registry.wikimedia.org/releng/node14-test-browser:0.0.2-s4
# software: Debian GNU/Linux 11 (bullseye)
#           Node.js v14.17.5 (npm 7.21.0)
#           Chromium 97.0.4692.99
#           Mozilla Firefox 91.5.0esr
#           JSDuck 5.3.4 (Ruby 2.7.4) ruby 2.7.4p191
# mount: /mediawiki      ➟ /Users/montehurd/mediawiki-test/mediawiki      (read-write)
#        /mediawiki/.git ➟ /Users/montehurd/mediawiki-test/mediawiki/.git (read-only)

I followed mediawiki setup instructions here:

https://gerrit.wikimedia.org/g/mediawiki/core/+/HEAD/DEVELOPERS.md

Running npm run selenium-test, I see the following:

nobody@docker-desktop:/mediawiki$ npm run selenium-test

> selenium-test
> wdio ./tests/selenium/wdio.conf.js


Execution of 5 workers started at 2022-05-20T22:15:19.540Z

[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
[0-0] 2022-05-20T22:15:59.605Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:59207
[0-0]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-3] 2022-05-20T22:15:59.625Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:56115
[0-3]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-2] 2022-05-20T22:15:59.624Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:56655
[0-2]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-1] 2022-05-20T22:15:59.639Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:60141
[0-1]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-0] RETRYING in chrome - /tests/selenium/specs/page.js
[0-1] RETRYING in chrome - /tests/selenium/specs/recentchanges.js
[0-2] RETRYING in chrome - /tests/selenium/specs/user.js
[0-3] RETRYING in chrome - /tests/selenium/specs/watchlist.js
[0-0] RUNNING in chrome - /tests/selenium/specs/page.js
[0-2] RUNNING in chrome - /tests/selenium/specs/user.js
[0-3] RUNNING in chrome - /tests/selenium/specs/watchlist.js
[0-1] RUNNING in chrome - /tests/selenium/specs/recentchanges.js
[0-0] 2022-05-20T22:16:38.766Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:58619
[0-0]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-2] 2022-05-20T22:16:38.914Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:56295
[0-2]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-3] 2022-05-20T22:16:38.918Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:55895
[0-3]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-0] FAILED in chrome - /tests/selenium/specs/page.js (1 retries)
[0-1] 2022-05-20T22:16:39.065Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:58169
[0-1]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-2] FAILED in chrome - /tests/selenium/specs/user.js (1 retries)
[0-3] FAILED in chrome - /tests/selenium/specs/watchlist.js (1 retries)
[0-1] FAILED in chrome - /tests/selenium/specs/recentchanges.js (1 retries)
[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
[0-4] 2022-05-20T22:17:10.637Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:58063
[0-4]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-4] RETRYING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
[0-4] RUNNING in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js
[0-4] 2022-05-20T22:17:42.220Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:57439
[0-4]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)
[0-4] FAILED in chrome - /tests/selenium/wdio-mediawiki/specs/BlankPage.js (1 retries)

Spec Files:	 0 passed, 5 retries, 5 failed, 5 total (100% completed) in 00:02:22

What should have happened instead?:

@zeljkofilipin Did the same steps on his Intel Mac and everything works ok.

I suspect there's some chromium configuration which isn't playing nice with arm64 hosts. If I understand correctly, since it doesn't look like we're building a fresh image for arm64, the amd64 image is used. My inclination was to see about building a fresh image for arm64, but I haven't tracked down specifics for how to go about this with fresh.

I know headless chrome can play nice on M1 from using browserless-chrome, which has an arm64 build:

https://hub.docker.com/r/browserless/chrome/tags

I'm using it elsewhere (though I do have an unrelated bug using it). Here's some info on what they use spinning up headless chrome:

https://docs.browserless.io/blog/2018/06/04/puppeteer-best-practices.html#7-use-docker-to-contain-it-all

Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc.:

  • MacOS Monterey 12.3.1
  • Apple M1
  • Docker Desktop 4.8.2, Engine 20.10.14, Compose 2.5.1

Any help is appreciated. I may be missing something glaringly obvious...


Upstream issue https://github.com/docker/for-mac/issues/5766 got declined

Closing as documented. https://docs.docker.com/docker-for-mac/apple-silicon/#known-issues

We don't have control over qemu and it's "best effort" only. We have documented both the occasional crashes and the lack of inotify support.

https://docs.docker.com/desktop/troubleshoot/known-issues/ and hitting For Mac with Apple Silicon has a list of issues including:

Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.
However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.
In summary, running Intel-based containers on Arm-based machines should be regarded as "best effort" only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers.

There is also https://gitlab.com/qemu-project/qemu/-/issues/324 chrome based apps can not be run under qemu user mode which refers to a patch which theoretically can be applied to the QEmu Debian package to see whether it improves things and then find a way to commit time to review/test it in order to get the patch merged by upstream. That needs some commitment and knowledge about system calls / OS development.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Mhurd updated the task description. (Show Details)

@Mhurd Are you using the propietary "Docker Desktop for Mac" app to host your Linux VM in which the containers run, or are you using a different docker-machine server? If the former, can you confirm that you've installed anew (not carried over from OS upgrade) the Apple Silicon version from https://docs.docker.com/desktop/mac/install/, including the rosetta command which installs the transparent layer for running Intel applications?

From their manual (link): Another thing that might work is to run export DOCKER_DEFAULT_PLATFORM=linux/amd64 on your outer shell before the fresh-node command. If that works, we can add that to Fresh. For reasons I don't understand, upstream Docker is refusing to let macOS engage Rosetta unless you specifically set this.

Hi, I've been using M1 arm64 for a while and the workaround --platform linux/amd64 almost never works (at least for me) I think we need to have arm64 builds. I've rebuilt my projects that used Chrome and Firefox to build both amd/arm containers.

In a nut shell, Docker Desktop for Mac uses Qemu to run the Linux VM, which either inherrently or due to the specific way that Docker uses it, makes inotify not work when emulated. Chromium requires this during startup and thus refuses to start. This isn't configurable as far as I know, and affects all uses of Chromium inside Docker on Apple M1, so long as the container is run under emulation. There are dozens of upstream bug reports and other projects affected by this as well. In addition to that, Chromium also makes use of zygote which causes Qemu to crash.

See also:

In general, Chromium works on Apple M1 when run directly by macOS (transparently via Rosetta), e.g. when using Google Chrome as desktop mac app. It fails only when run with the indirection of Qemu/Linux/Docker.

I know headless chrome can play nice on M1 from using browserless-chrome, which has an arm64 build:

Chromium could be compiled directly for ARM-type chips (such as Apple M1) and run fine within Qemu/Docker then. However, as far as I know there are not yet any such official distributions by Google. It is my understanding that the way browserless/chrome works on M1, is by actually using the Microsoft Edge distribution of Chromium (from playwright instead of puppeteer). Unlike Google, Microsoft does support ARM-type processors. Noting that Apple M1 is not the first chipset to use ARM-type processors, there are plenty of PC laptops with ARM-type chips, instead of AMD-type chips like Intel.

My inclination was to see about building a fresh image for arm64 […]

Fresh is mostly just an idea that, concretely, is merely a 10-line shell alias to run docker run --rm --interactive --tty --entrypoint /bin/bash docker-registry.wikimedia.org/releng/node12-test-browser. Its main purpose is to resemble and reflect CI and does so by literally using the same container, and CI in turn intends to generally share the base images and packages with production.

To create a native arm64 Linux base image, we'd need a lot more than Chromium.

See also:

@Krinkle Thanks for the great feedback!

Are you using the propietary "Docker Desktop for Mac"

Yes

can you confirm that you've installed anew (not carried over from OS upgrade) the Apple Silicon version from https://docs.docker.com/desktop/mac/install/, including the rosetta command

Yes, but just to be extra sure I re-installed Docker Desktop and re-ran the Rosetta command

To create a native arm64 Linux base image, we'd need a lot more than Chromium.

I see, and that makes sense.


It may be helpful to explain what I'm trying to do...

I have a repo here with a makefile which lets you spin up mediawiki from scratch with basically a single command. Keep in mind I'm still relatively novice with Docker, but so far this has been a really fun learning experience.

My next goal was to add make commands for running tests, which was easy for parser and unit tests ( make runparsertests, make runphpunittests... forgive the all lower case, will tweak this in the future... ) They worked as expected.

So next I wanted to tackle selenium tests. Because I couldn't seem to get the tests to work with fresh, I tried using a browserless chrome container as seen is my WIP selenium branch, but even though the tests begin, and can actually be seen running in a chrome window served up by the browserless chrome container, the tests get stuck for some reason after a little bit.

My suspicion is the problem is related to how I'm configuring browserless, and it's maybe spawning too many sessions, so when a test causes a new page to be loaded, browserless isn't just re-using the session in the same way it would if you were running the tests outside of a dockerized environment.

While debugging this we thought it would be instructive to circle back and see if running the tests hangs in the *same way* on my machine when using fresh, or if they actually work but I had just been using fresh incorrectly to run the selenium tests, but they seem to be hanging more immediately when run via fresh...

That was a lot of text :)

I'm still digesting your second comment...

One thing I did notice in fresh which I wasn't sure how to interpret was...

nobody@docker-desktop:/mediawiki$ chromium --version
Error: Can't open display: 
nobody@docker-desktop:/mediawiki$ /usr/lib/chromium/chromium --version
Chromium 97.0.4692.99 
nobody@docker-desktop:/mediawiki$

Edit: perhaps it's related to this this:

And to avoid running into zombie processes (which commonly happen with Chrome), you'll want to use something like dumb-init to properly start-up:

ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init

I'm pretty out of my depth here haha.

I still think the problem is a mismatch between arm/amd containers. I remember I had the same problem when I tried to run Chrome in AMD containers on ARM. I think that the nodeX-test-browser needs to be built for ARM and then Fresh too.

I was able to hack Fresh so it worked for me. Changing the base image to a version that was built for linux/arm64 and then removed the --platform linux/amd64 in Fresh makes Selenium/Chromium work.

What's strange though is that I have other images where setting the platform to linux/amd64 make Chrome work but that has been on Ubuntu based images.

With the help from @hashar I finally understand what's going on. Somehow chromium --headless do not work anymore without a display in our container (I also tried the new --headless=new flag). When we try to run the tests without an exported display, our webdriver.io config adds --headless but that gives us that error. If I manually starts xvfb and export the display it works. What's strange is that this then only happens on a Mac M1 when you force to run the image as amd.

I did try a couple of other things: I rollbacked the docker container to the version that uses Chrome 103 and then headless works. Trying to build a minimal test case with Dockerfile:

FROM debian:11
ARG TARGETPLATFORM=linux/amd64
RUN apt-get update && apt-get install chromium -y

And then build it: docker buildx build --load --platform linux/amd64 -t debian/debianwithchromium .

And then /running in my ARM machine with : docker run -it --platform linux/amd64 --entrypoint bash

I get:

root@801e3447542f:/# chromium --version
The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/112335

Also I missed to upgrade to latest Fresh on my machine when tried out xvfb. If you use latest, start xvfb my output is:

nobody@17aef58260e9:/$ chromium --version
Warning: Missing charsets in String to FontSet conversion

And the chromium just hangs.

root@801e3447542f:/# chromium --version
The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/112335

The proper bug link is http://crbug.com/1123353 which is Chromium requiring the SSE3 instructions set since Chromium 89. To catch that up ahead of starting Chromium, the Debian shell script starting Chromium got adjusted ( https://salsa.debian.org/chromium-team/chromium/-/commit/836b9da55c776a27d884d0405f385dcb7ef6f12e ):

case `uname -m` in
    i386|i586|i686|x86_64)
        # Check whether this system supports SSE3 (aka PNI)
        if ! grep -q 'sse3\|pni' /proc/cpuinfo; then
            xmessage "$nosse3"
            exit 1
        fi
        ;;
esac

uname -m ends up being the hardware machine type as compiled in by the Linux kernel which gives x86_64 and is I guess "normal". Then either Apple Rosetta or Qemu is not exposing the ss3 instructions set since it is not found in /proc/cpuinfo. Then that code path is working properly in your second output (using latest Fresh) which puzzles me :)

I see that same error:

nobody@bae9efe5b9a5:/test$ /usr/bin/chromium --version
The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/1123353

But interestingly, if I just side-step that script and go to straight to the binary, it seems to work:

root@31469916ee0e:/test# /usr/lib/chromium/chromium --version
Chromium 120.0.6099.224
root@31469916ee0e:/test#

Do we need that script?
Where does it even come from? 😆
Could I just point /usr/bin/chromium to /usr/lib/chromium/chromium?

@hashar @Peter

I threw this example together:

https://gitlab.wikimedia.org/mhurd/fresh-node-novnc-example

You can see Chrome runs when the script is bypassed and we just use /usr/lib/chromium/chromium

Tested on ARM mac, but probably works elsewhere too

I see that same error:

nobody@bae9efe5b9a5:/test$ /usr/bin/chromium --version
The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/1123353

The shell script is provided by the Debian package. Its source is available in a repository that has a branch for each of the Debian distribution (BusterBullseyeBookworm.... So for Bullseye the script source is https://salsa.debian.org/chromium-team/chromium/-/blob/bullseye/debian/scripts/chromium?ref_type=heads

From there you can git blame to find the original commit which leads me to https://salsa.debian.org/chromium-team/chromium/-/commit/98718255f00a55fdf319796d2bb55ff36390b711 which was driven by removing the package dependency upon sse3-support which is meant to fail. I think they never supported sse3, or at least stopped for a long time. The instruction set was introduced back in 2004 and if you check the list of CPUs having it, that brings some good memory (if you are 40+ years old) https://en.wikipedia.org/wiki/SSE3#CPUs_with_SSE3

Anyway, the script check is rather basic:

case `uname -m` in
    i386|i586|i686|x86_64)

Which I guess is fulfilled. Next check is:

# Check whether this system supports SSE3 (aka PNI)
if ! grep -q 'sse3\|pni' /proc/cpuinfo; then
    display_msg "$nosse3"
    exit 1
fi

On my system (8th gen Intel core) I get both pni and ssse3. You can check the available flags with grep ^flags /proc/cpuinfo |head -n1. Example for me:

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust sgx bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities

If I remember correctly, Docker on Mac uses Qemu and I guess it should be given whatever option to add those instructions sets? I am willing to bet it is made to emulate an old cpu that does not have that instruction set ;) A more or less related bug about pshufb instructions missing https://github.com/docker/for-mac/issues/5561

But interestingly, if I just side-step that script and go to straight to the binary, it seems to work:

root@31469916ee0e:/test# /usr/lib/chromium/chromium --version
Chromium 120.0.6099.224
root@31469916ee0e:/test#

Do we need that script?
Where does it even come from? 😆
Could I just point /usr/bin/chromium to /usr/lib/chromium/chromium?

@hashar @Peter

Well because I guess the code needed to --version and exit does not end up invoking any sse3 instruction? But tentatively loading a web page, or rendering a gif or whatever more complicated would end up hitting the unknown sse3 instruction and explode. Well I can't remember how that behaves, maybe just a segmentation fault.

So I'll note try the adventure of shortcircuiting the script which is a guard to prevent a later failure if Chromium got compiled with sse3 enabled. Well based on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1010407 that would report:

Received signal Illegal instruction

From signal(7) man page:

       Signal      Standard   Action   Comment
       ────────────────────────────────────────────────────────────────────────
...
       SIGILL       P1990      Core    Illegal Instruction

Looks like people in the 1970's thought about everything for us!

And that bug report has the lead explaining why the script fails:

I will add a test to the chromium script to refuse to run without sse3, though and points at https://salsa.debian.org/chromium-team/chromium/-/commit/836b9da55c776a27d884d0405f385dcb7ef6f12e . Full circle.

I believe the actual bug is in Docker for Mac, QEmu should emulate a more recent processor.

@hashar Thanks for checking and the info!! Will dig deeper

Also, I just browsed around various web pages using https://gitlab.wikimedia.org/mhurd/fresh-node-novnc-example for about ten minutes with no issues... 🤔

Oh, I have "Rosetta for x86_64/amd64 emulation on Apple Silicon" enabled

Screenshot 2024-07-10 at 3.07.51 PM.png (121×994 px, 25 KB)

Side note for @Krinkle

Any thoughts on maybe adding an env var to fresh-node which when used would spin up a supplemental container similar to this proof-of-concept for viewing Chromium/Firefox or other processes as they run in a fresh container?

For example, would be nice when kicking off Selenium tests through fresh-node to have an easy way to watch them run

Oh, I have "Rosetta for x86_64/amd64 emulation on Apple Silicon" enabled

Screenshot 2024-07-10 at 3.07.51 PM.png (121×994 px, 25 KB)

When inside the container, can you run the commands used by the Debian script /usr/bin/chromium?

uname -m
grep -q 'sse3\|pni' /proc/cpuinfo

Well, can you also paste the full script trace using: /bin/sh -x /usr/bin/chromium?

But interestingly, if I just side-step that script and go to straight to the binary, it seems to work:
(...)
Could I just point /usr/bin/chromium to /usr/lib/chromium/chromium?

I recently documented how to run Selenium tests using any chromium-based browser: Selenium/How-to/Run tests using a specific browser. Selenium tests on M1 machine fail when using /usr/bin/chromium but work just fine when using /usr/lib/chromium/chromium.

Environment:

~$ sysctl -n machdep.cpu.brand_string
Apple M1 Max

~$ sw_vers
ProductName:		macOS
ProductVersion:		14.5
BuildVersion:		23F79

~$ docker --version
Docker version 27.1.1, build 6312585

~$ docker-compose --version
Docker Compose version v2.29.1-desktop.1

~/Documents/gerrit/mediawiki/core$ fresh-node18 -env -net            
# 🌱 Fresh 24.05.1 ░ Node.js 18 ░ npm 10 ░ Firefox 115 ░ Chromium 120 ░ Debian 11 Bullseye
# image: docker-registry.wikimedia.org/releng/node18-test-browser:18.20.2-s1
# mount: /core      ➟ /Users/z/Documents/gerrit/mediawiki/core      (read-write)
#        /core/.git ➟ /Users/z/Documents/gerrit/mediawiki/core/.git (read-only)
# env: MW_*, MEDIAWIKI_*
# net: expose host

Using /usr/bin/chromium (full output at P67204):

nobody@docker-desktop:/core$ npm run selenium-test
(...)
Spec Files:      0 passed, 6 retries, 6 failed, 6 total (100% completed) in 00:01:52

Using /usr/lib/chromium/chromium (full output at P67205):

nobody@docker-desktop:/core$ npm run selenium-test
(...)
Spec Files:      6 passed, 6 total (100% completed) in 00:00:21

The only difference between the two test runs is using /usr/lib/chromium/chromium.

~/Documents/gerrit/mediawiki/core$ git diff
diff --git a/tests/selenium/wdio-mediawiki/wdio-defaults.conf.js b/tests/selenium/wdio-mediawiki/wdio-defaults.conf.js
index ba7b2e344c6..0aaf069ad21 100644
--- a/tests/selenium/wdio-mediawiki/wdio-defaults.conf.js
+++ b/tests/selenium/wdio-mediawiki/wdio-defaults.conf.js
@@ -59,6 +59,7 @@ exports.config = {
                // For Chrome/Chromium https://www.w3.org/TR/webdriver
                browserName: 'chrome',
                'goog:chromeOptions': {
+                       binary: '/usr/lib/chromium/chromium',
                        // If DISPLAY is set, assume developer asked non-headless or CI with Xvfb.
                        // Otherwise, use --headless.
                        args: [

@zeljkofilipin see my previous comment above T308889#9980430 Monte is using Rosetta, I guess you the use same.

Can you run the following commands and paste the result:

uname -m
egrep ^flags /proc/cpuinfo|tail n-1
grep -q 'sse3\|pni' /proc/cpuinfo
/bin/sh -x /usr/bin/chromium

@zeljkofilipin see my previous comment above T308889#9980430 Monte is using Rosetta, I guess you the use same.

This is the Rosetta Docker setting you are referring to?

docker.png (1×1 px, 549 KB)

Can you run the following commands and paste the result:

uname -m
egrep ^flags /proc/cpuinfo|tail n-1
grep -q 'sse3\|pni' /proc/cpuinfo
/bin/sh -x /usr/bin/chromium

I'm assuming I should run this from Fresh:

~/Documents/gerrit/mediawiki/core$ fresh-node18 -env -net
# 🌱 Fresh 24.05.1 ░ Node.js 18 ░ npm 10 ░ Firefox 115 ░ Chromium 120 ░ Debian 11 Bullseye
# image: docker-registry.wikimedia.org/releng/node18-test-browser:18.20.2-s1
# mount: /core      ➟ /Users/z/Documents/gerrit/mediawiki/core      (read-write)
#        /core/.git ➟ /Users/z/Documents/gerrit/mediawiki/core/.git (read-only)
# env: MW_*, MEDIAWIKI_*
# net: expose host

nobody@docker-desktop:/core$ uname -m
x86_64

nobody@docker-desktop:/core$ egrep ^flags /proc/cpuinfo|tail n-1
tail: cannot open 'n-1' for reading: No such file or directory

nobody@docker-desktop:/core$ grep -q 'sse3\|pni' /proc/cpuinfo

nobody@docker-desktop:/core$ /bin/sh -x /usr/bin/chromium
+ APPNAME=chromium
+ GDB=/usr/bin/gdb
+ LIBDIR=/usr/lib/chromium
+ BUILD_DIST=11.8
+ nosse3=The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/1123353
+ noneon=The hardware on this system lacks support for NEON SIMD extensions.
We now require NEON or equivalent architecture extensions on ARM-based
machines. See https://lists.debian.org/debian-devel/2023/09/msg00175.html
for more information.
+ uname -m
+ grep -q sse3\|pni /proc/cpuinfo
+ display_msg The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/1123353
+ [ -z  ]
+ echo The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/1123353
The hardware on this system lacks support for the sse3 instruction set.
The upstream chromium project no longer supports this configuration.
For more information, please read and possibly provide input to their
bug tracking system at http://crbug.com/1123353
+ exit 1

Monte is using Rosetta, I guess you the use same.

I guess using Rosetta is the default. I didn't make any changes to Docker settings.

I disabled Rosetta in Docker and re-ran the tests. Now tests with both /usr/bin/chromium (P67206) and /usr/lib/chromium/chromium (P67207) fail, but with different error messages.

/usr/bin/chromium with:

[0-0] 2024-08-02T13:51:04.441Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:56169
[0-0]     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16)

/usr/lib/chromium/chromium with:

[0-1] 2024-08-02T13:54:47.226Z ERROR @wdio/runner: FetchError: Failed to fetch browser webSocket URL from http://localhost:59387/json/version: request to http://localhost:59387/json/version failed, reason: connect ECONNREFUSED 127.0.0.1:59387
[0-1]     at ClientRequest.<anonymous> (/core/node_modules/node-fetch/lib/index.js:1491:11)
[0-1]     at ClientRequest.emit (node:events:517:28)
[0-1]     at ClientRequest.emit (node:domain:489:12)
[0-1]     at Socket.socketErrorListener (node:_http_client:501:9)
[0-1]     at Socket.emit (node:events:517:28)
[0-1]     at Socket.emit (node:domain:489:12)
[0-1]     at emitErrorNT (node:internal/streams/destroy:151:8)
[0-1]     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
[0-1]     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
nobody@docker-desktop:/core$ egrep ^flags /proc/cpuinfo|tail n-1
tail: cannot open 'n-1' for reading: No such file or directory

Sorry I have screwed up a command tail n-1 should obviously have been tail -n 1 :)

With Rosetta enabled and within a Fresh container can you paste the output of:

egrep ^flags /proc/cpuinfo|tail -n 1

That list the CPU capabilities detected by the Linux Kernel. They would be exposed by Rosetta. It is not exposing the SSSE3 instructions set which causes Debian wrapper script to bails out early since Chrome apparently requires it? My guess is the instruction set can't be translated to an ARM CPU.

While at it, can you paste the full /proc/cpuinfo (from within a Fresh container)?

With Rosetta enabled and within a Fresh container can you paste the output of:

egrep ^flags /proc/cpuinfo|tail -n 1

It outputs nothing:

nobody@docker-desktop:/core$ egrep ^flags /proc/cpuinfo|tail -n 1
nobody@docker-desktop:/core$

While at it, can you paste the full /proc/cpuinfo (from within a Fresh container)?

nobody@docker-desktop:/core$ cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 1
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 2
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 3
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 4
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 5
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 6
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 7
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 8
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

processor       : 9
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0
processor       : 0
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer : 0x61
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0x000
CPU revision    : 0

So Qemu/Rosetta exposes the cpu instruction sets as Features. The second one is asimd which is a short for Advanced SIMD and is also known as Neon ( https://en.wikipedia.org/wiki/ARM_architecture_family#Advanced_SIMD_(Neon) ).

That explains that bit in the chromium script:

armv7l)
    # Check whether this system supports NEON or ASIMD. Even though there
    # could be ARMv8 systems without ASIMD, we're far more likely to hit
    # v8 hardware emulating v7 systems with buggy VMs; so don't even
    # bother checking armv8l for now. If we do check in the future, they
    # advertise 'asimd' instead of 'neon'.
    if ! grep -q 'neon\|asimd' /proc/cpuinfo; then
        display_msg "$noneon"
        exit 1
    fi
    ;;

But you have reported uname -m yields x86_64:

nobody@docker-desktop:/core$ uname -m
x86_64

That comes from systemd call uname(2) and the machine is hardcoded with the target architecture chosen at compile time (x86_64). Thus the script assumes it is running under an x86 arch and looks for ssse3 or pni instructions but it is actually emulated. My guess is QEmu should report in CPUInfo the machine it is emulating instead of the host?

My guess is QEmu should report in CPUInfo the machine it is emulating instead of the host?

Sorry that is not QEmu but Rosetta. The issue I believe is https://github.com/docker/for-mac/issues/7080 which shows /proc/cpuinfo is populated with the host CPU sets instead of the emulated one.

So yeah you could use /usr/lib/chromium/chromium instead, if it does use ssse3 instructions, my guess is Rosetta manages to translate them and Chromium does not crash with SIGILL (Illegal Instruction) hence why it works :)