@jaredblumer I think I know what the problem is. selenium NPM script is removed, but the documentation wasn't updated. 🤦‍♂️

From your terminal output:

...
I have no name!@jdot:/mediawiki$ npm run selenium
	npm ERR! missing script: selenium
...

The script was removed last month in 650226.

I'll update the documentation (at Selenium/How-to/Run tests targeting MediaWiki-Docker using Fresh).

Step #1, reproduced the error:

nobody@e7208c76e29e:/core$ npm run selenium
npm ERR! missing script: selenium

npm ERR! A complete log of this run can be found in:
npm ERR!     /cache/_logs/2021-01-12T18_58_34_344Z-debug.log

Thanks for looking into this, Zeljko! As you can see in https://phabricator.wikimedia.org/P13721 beneath the command above, I saw that 'npm run selenium' was an invalid script, so I found the proper 'selenium-test' script in the package.json and tried running that, resulting in the error that I described in this issue. If you take another look you'll see that the error in question is resulting from the command 'npm run selenium-test'.

@jaredblumer I stopped at the first error 🤦‍♂️ I've tried updating the docs and got to the same place as you, selenium-test script failing. It was too late in my day so I gave up. I'll continue today.

@jaredblumer I've updated Selenium/Getting Started/Run tests targeting MediaWiki-Docker. Can you run the tests that way?

It will need more time to debug Selenium/How-to/Run tests targeting MediaWiki-Docker using Fresh).

I'm not sure what is wrong. 🤔 @Krinkle, do you know what is the problem?

I'm following updated instructions at Selenium/How-to/Run tests targeting MediaWiki-Docker using Fresh

The same images and infrastructure are used in CI, so overall it seems to still work. I'll take a look, to see if I can reproduce the issue and/or decude what might be causing this issue for you.

@Krinkle thanks! I'm not sure what the problem is. The tests fail with connect ECONNREFUSED 127.0.0.1:8080.

I'm not able to reproduce the issue. I tried both with MediaWiki-Docker and with my usual mediawiki-docker-dev setup:

@zeljkofilipin Based on the Wdio error, I suspect it is likely a networking issue. In other words, the browser is just seeing an empty browser tab.

The tests fail with connect ECONNREFUSED 127.0.0.1:8080.

Right, so this means your MediaWiki server isn't visible to the container.

Some things to check:

• What do the screenshots and videos/frames look like? (Presumably just an empty tab, but with confirming)
• Did you pass -env and -net?
• What does running env | grep -E 'MW|MEDIAW' | sort inside Fresh return?
• What do curl -I $MW_SERVER and curl -I $MW_SERVER/w/load.php return inside Fresh? Does it give more detail than ECONNREFUSED? Does it work if you run it outside of docker, from your primary shell?

It looks like this might be an issue with Docker for Linux where -net (which we map to docker run --network=host) might not correctly by default.

Try this patch and let me know if that works. According to some internet forums, one of these is meant to work on Ubuntu since Docker v20.04 (or Docker v20.10?):

 	docker_args+=("--network" "host")
+	docker_args+=("--add-host=host.docker.internal:host-gateway")

Or

-	docker_args+=("--network" "host")
+	docker_args+=("--add-host=host.docker.internal:host-gateway")

I removed my previous comments as I believe I may have made a mistake when following @Krinkle's recommendations. I'll take a look again tomorrow and will share my results.

I ran the tests again using "chromedriver --url-base=wd/hub --port=4444 &" and passed only 3 of 6 tests. Do you have any insights as to why these 3 failed?

I receive this error when attempting the "host-gateway" patch.

jaredblumer@jdot:~/Desktop/mediawiki$ fresh-node -env -net
docker: Error response from daemon: invalid IP address in add-host: "host-gateway".
See 'docker run --help'.

In T271772#6754200, @jaredblumer wrote:

I ran the tests again using "chromedriver --url-base=wd/hub --port=4444 &" and passed only 3 of 6 tests. Do you have any insights as to why these 3 failed?

Error: ENOENT: no such file or directory, open '/mediawiki/tests/selenium/log/rawSeleniumVideoGrabs/User--should-be-able-to-log-in-daily--CHROME--01-18-2021--03-04-23-646/0007.png'
…
Spec Files:	 3 passed, 3 failed, 6 total (100% completed) in 00:02:13

If even a single test passes, then fortunately that means Docker or Fresh networking is working fine, so no patch needed. Were those 3 passing tests without any patch applied to fresh-node? If so, then you and Zeljko are experiecing two distinct issues. In any event, good to know that at least for you it seems Docker and Fresh -net work correctly on Ubuntu out of the box.

The above error means MediaWiki's webdriverio code was unable to create or add to the tests/selenium/log/ directory. This is likely a permission problem.

To confirm whether this is specific to webdriverio or a more general permission issue with the directory, try the following without any MW or selenium code:

you:mediawiki$ fresh-node -net -env

nobody@fresh:mediawiki$ mkdir -p tests/selenium/log/anewdir/
nobody@fresh:mediawiki$ touch tests/selenium/log/foo.txt

In T271772#6754214, @jaredblumer wrote:

I receive this error when attempting the "host-gateway" patch.

jaredblumer@jdot:~/Desktop/mediawiki$ fresh-node -env -net
docker: Error response from daemon: invalid IP address in add-host: "host-gateway".
See 'docker run --help'.

What is the version returned by docker --version? (We're still on Ubuntu 20.04, right?)

I don't seem to have any problem with permissions in fresh-node. I was able to make a directory and create a file as you asked (code below). Any other ideas as to why I'm not passing all 6 selenium tests?

jaredblumer@jdot:~/Desktop/mediawiki$ fresh-node -env -net
# fresh: 20.08.1 (2020-08-27)
# image: docker-registry.wikimedia.org/releng/node10-test-browser:0.6.2
# software: Debian GNU/Linux 9 (stretch)
#           Node.js v10.15.2 (npm 6.14.5)
#           Chromium 71.0.3578.80
#           Mozilla Firefox 68.11.0esr
#           JSDuck 5.3.4 (Ruby 2.3.3)
# mount: /mediawiki      ➟ /home/jaredblumer/Desktop/mediawiki      (read-write)
#        /mediawiki/.git ➟ /home/jaredblumer/Desktop/mediawiki/.git (read-only)
# env: MW_*, MEDIAWIKI_*
# net: expose host

🌱  Fresh!

I have no name!@jdot:/mediawiki$ mkdir -p tests/selenium/log/anewdir/
I have no name!@jdot:/mediawiki$ touch tests/selenium/log/foo.txt
I have no name!@jdot:/mediawiki$ ls tests/selenium/log/anewdir/
I have no name!@jdot:/mediawiki$ ls tests/selenium/log/
anewdir
...
foo.txt

What is the version returned by docker --version? (We're still on Ubuntu 20.04, right?)

My Docker version is 20.10.2, build 2291f61 and I am using Ubuntu 20.04.1 LTS.

In T271772#6749946, @Krinkle wrote:

• Did you pass -env and -net?

No. 🤦‍♂️

I think @jaredblumer and I are having two completely separate problems. I've reported my problem as T272316: `fresh-node -env -net` not working on macOS 10.15.

In T271772#6755924, @jaredblumer wrote:

Any other ideas as to why I'm not passing all 6 selenium tests?

Can you upload screenshots and videos that Selenium created in tests/selenium/log? (Make sure you delete the folder before the test run, so it's not populated with files from a previous run.)

Can you run the tests targeting mediawiki-docker without fresh? (Documentation: Selenium/Getting Started/Run tests targeting MediaWiki-Docker.)

I thought perhaps my issue was due to my Ubuntu Linux setup, so I attempted to run the same code on my iMac (macOS Catalina v 10.15.7) and received the same error

ERROR webdriver: RequestError: connect ECONNREFUSED 127.0.0.1:4444

Any suggestions?

@zeljkofilipin Thank you so much! Doing this alone passed three tests, then I realized I had to add the Vector skin for the other three to pass so we're all set. I'll close the task.