As far as I can tell, QuickStart tests run on Linux, hosted in a Docker image, in both CI and local development.
Test behaviour should be identical in both environments. For the Popups extension, tests seem to pass in CI:
For local development, tests throw the following error:
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] 1) Dwelling on a valid page link I should see a page preview
[chrome 138.0.7204.157 linux #0-0] element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0] Error: element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] 2) Dwelling on a valid page link Abandoning link hides page preview
[chrome 138.0.7204.157 linux #0-0] element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0] Error: element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] 3) Dwelling on a valid page link Quickly hovering, abandoning and re-hovering a link shows page preview
[chrome 138.0.7204.157 linux #0-0] element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0] Error: element (".mwe-popups") still not existing after 10000ms
------------------------------------------------------------------
[chrome 138.0.7204.157 linux #0-0] Running: chrome (v138.0.7204.157) on linux
[chrome 138.0.7204.157 linux #0-0] Session ID: 13133e1ae705ef87bfc61bb105f9dfa9
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] » /tests/selenium/specs/page_previews.js
[chrome 138.0.7204.157 linux #0-0] Dwelling on a valid page link
[chrome 138.0.7204.157 linux #0-0] ✖ I should see a page preview
[chrome 138.0.7204.157 linux #0-0] ✓ I should not see a page preview on hash fragment
[chrome 138.0.7204.157 linux #0-0] ✖ Abandoning link hides page preview
[chrome 138.0.7204.157 linux #0-0] ✖ Quickly hovering, abandoning and re-hovering a link shows page preview
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] 1 passing (37s)
[chrome 138.0.7204.157 linux #0-0] 3 failing
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] 1) Dwelling on a valid page link I should see a page preview
[chrome 138.0.7204.157 linux #0-0] element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0] Error: element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] 2) Dwelling on a valid page link Abandoning link hides page preview
[chrome 138.0.7204.157 linux #0-0] element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0] Error: element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0]
[chrome 138.0.7204.157 linux #0-0] 3) Dwelling on a valid page link Quickly hovering, abandoning and re-hovering a link shows page preview
[chrome 138.0.7204.157 linux #0-0] element (".mwe-popups") still not existing after 10000ms
[chrome 138.0.7204.157 linux #0-0] Error: element (".mwe-popups") still not existing after 10000ms
Spec Files: 0 passed, 1 retries, 1 failed, 1 total (100% completed) in 00:01:43To confirm the issue is not isolated to my machine (Apple M1 max), we ran the same tests on a colleague's laptop (Apple Silicon)
and the results were similar. The steps taken before running the tests were:
- git pull to make sure we were running the latest code
- Confirmed we had the latest docker version
- "./remove" and "docker system prune -af" to clean the system #./fresh_install
This seems like an environment issue that might be isolated to Apple silicon and needs to be investigated.
AC:
Test results in CI and the local environment should be identical.