Page MenuHomePhabricator

Usage instructions in tests/selenium/README.md are confusing
Closed, ResolvedPublic

Description

This is the current documentation about the different test run modes in core’s tests/selenium/README.md:

  • Headless. It's the default. You will not see the browser while tests are running because it's running in a headless mode. This mode should run fine on all supported platforms.
  • Headless recording. Set DISPLAY environment variable to a value that starts with colon (:) and video of each test will be recorded. Browser will run headless. Recording videos works only on Linux.
  • Visible. If you want to see the browser, set DISPLAY environment variable to any value that does not start with colon. This mode will not work in a headless environment like MediaWiki-Vagrant.

I found this very confusing, since as far as I can tell it doesn’t match what the tests actually do.

  • It’s not very helpful to describe headless as “the default”. Headless mode is used if the $DISPLAY variable is unset or empty, but on a desktop system, that is not the default – the user would have to unset DISPLAY or export DISPLAY='' manually.
  • As far as I can tell, it’s not true that DISPLAY=:0, for example, would use headless mode. wdio.conf.js only passes --headless to Chrome if process.env.DISPLAY is falsy, it doesn’t check whether the value begins with a colon or not.
  • It’s also not clear to me how “visible” mode is supposed to work – which display will it use then? – but I can’t try it out anyways for unrelated reasons.

The whole paragraph seems to assume a MediaWiki-Vagrant setup, whereas the rest of the README.md is more careful to mention the changes necessary to support other setups (MW_SERVER, MW_SCRIPT_PATH, etc.). The upshot of it seems to be that, in fact, the “default” mode on system (no containerization of any kind, just a plain MediaWiki install on a desktop Linux system) would work perfectly well (DISPLAY=:0, not headless, recording), except for the fact that I use Wayland, whereas ffmpeg -f x11grab requires X11 (which should be documented alongside the Linux requirement, I think).

Details

Related Gerrit Patches:
mediawiki/core : masterselenium: Improve README docs

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2019, 5:12 PM
Krinkle added a comment.EditedJan 25 2019, 7:54 PM

The text is written based on the experience from Zeljko, Antoine, myself, and various others having used it since. And is based on these environments:

  1. Jenkins job in Wikimedia CI (Debian Linux), ... outside Docker.
  2. ... inside Docker.
  3. Terminal on macOS (raw shell, not containerised), .. with MW itself raw on Apache.
  4. ... with MW from mediawik-docker-dev.
  5. ... with MW from mediawiki-vagrant.
  6. MediaWiki-Vagrant (using the shell inside the Vagrant VM), ssh'ed from.. Windows.
  7. ... ssh'ed from Linux.
  8. ... ssh'ed from Mac.

In all of these three environments, and their variations, I believe DISPLAY is not set by default.

There is only one environment I knew of that sets DISPLAY, and that's when explicitly using Xvfb inside CI or inside Vagrant.

Based on your experience, I guess the second environment I now know of that also sets DISPLAY is when using an X11 Window manager on desktop Linux, with a shell prompt that itself is also started as an X11 program, which presumably inherits this DISPLAY environment variable?

Based on your experience, I guess the second environment I now know of that also sets DISPLAY is when using an X11 Window manager on desktop Linux, with a shell prompt that itself is also started as an X11 program, which presumably inherits this DISPLAY environment variable?

Yes, that’s more or less my environment. (Strictly speaking, the GNOME Terminal Server is started as a systemd user service, but the systemd user instance also has the $DISPLAY set.) More generally, as far as I’m aware most graphical terminals on Linux desktop systems would have a $DISPLAY set. I’m surprised the macOS terminal doesn’t have a $DISPLAY, but apparently macOS doesn’t run X by default.

By the way, I’d be happy to submit a patch to the file for discussion, except that I don’t know what to make of the “visible” mode.

zeljkofilipin triaged this task as Normal priority.Sep 11 2019, 11:52 AM

Change 538307 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Improve docs and allow passing args to 'npm run selenium'

https://gerrit.wikimedia.org/r/538307

Change 538307 merged by jenkins-bot:
[mediawiki/core@master] selenium: Improve README docs

https://gerrit.wikimedia.org/r/538307

Krinkle closed this task as Resolved.Oct 3 2019, 7:42 PM
Krinkle claimed this task.
Krinkle added a project: Performance-Team.