Status	Assigned	Task
Resolved	hashar	T119138 [keyresult] Migrate majority of CI jobs to Nodepool (part 2)
Duplicate	None	T136249 Track what tests we have left to convert to nodepool
Resolved	hashar	T119140 [keyresult] Migrate as many misc CI jobs as possible to Nodepool
Resolved	hashar	T114315 [keyresult] Migrate majority of CI jobs to Nodepool (part 1)
Duplicate	None	T126774 Run 'npm' job with Node 4 (instead of Node 0.10)
Resolved	Krinkle	T129617 Jobs sometimes fail with "/usr/local/bin/npm: No such file or directory"
Resolved	hashar	T119143 Migrate javascript npm CI jobs to Nodepool
Resolved	Paladox	T128089 Migrate OOjs npm CI job to Nodepool
Resolved	hashar	T128093 Migrate VisualEditor core npm CI job to Nodepool
Resolved	hashar	T128090 Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing
Resolved	hashar	T129320 Diskimage fail on xvfb/systemd
Resolved	hashar	T129345 Xvfb service does not start on Nodepool instances
Invalid	None	T128135 Jessie should have the ability to run several tests at once like UbuntuTrusty can

Event Timeline

Jdforrester-WMF created this task.Feb 25 2016, 3:54 PM

Jdforrester-WMF renamed this task from Provide the Chrome binary on the nodepool servers so OOjs can use it in testing to Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing.Feb 25 2016, 3:56 PM

Jdforrester-WMF added a parent task: T128093: Migrate VisualEditor core npm CI job to Nodepool.

Change 273269 had a related patch set uploaded (by Paladox):
Add npm-setup.sh script to npm-node-4.3

https://gerrit.wikimedia.org/r/273269

gerritbot added a project: Patch-For-Review.Feb 25 2016, 4:31 PM

The patch above just adds the support to detect chromium, The actual binary needs to be installed on the server.

It also needs to have an Xvfb (X11 display server) running on port 94 (or some other unused number) because we run Chrome in its normal (graphical) mode. Without that, the DISPLAY=:94 will point to a display that does not exist.

Change 275832 had a related patch set uploaded (by Hashar):
dib: browsers Xvfb

https://gerrit.wikimedia.org/r/275832

hashar claimed this task.Mar 8 2016, 3:56 PM

hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.

hashar moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.

Change 275832 merged by jenkins-bot:
dib: browsers Xvfb

https://gerrit.wikimedia.org/r/275832

Stuff is being provisionned

Notice: /Stage[main]/Xvfb/Package[xvfb]/ensure: ensure changed 'purged' to 'present'
Notice: /Stage[main]/Xvfb/Group[xvfb]/ensure: created
Notice: /Stage[main]/Xvfb/User[xvfb]/ensure: created
Notice: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/File[/lib/systemd/system/xvfb.service]/ensure: created
Info: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/File[/lib/systemd/system/xvfb.service]: Scheduling refresh of Service[xvfb]
Info: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/File[/lib/systemd/system/xvfb.service]: Scheduling refresh of Exec[systemd reload for xvfb]
Notice: /Stage[main]/Contint::Browsers/Package[chromedriver]/ensure: ensure changed 'purged' to 'latest'
Notice: /Stage[main]/Contint::Browsers/Package[iceweasel]/ensure: ensure changed 'purged' to 'latest'
Notice: /Stage[main]/Contint::Browsers/Package[xfonts-cyrillic]/ensure: ensure changed 'purged' to 'present'
Notice: /Stage[main]/Main/Git::Clone[jenkins CI slave scripts]/Exec[git_pull_jenkins CI slave scripts]/returns: executed successfully
Info: /Stage[main]/Main/Git::Clone[jenkins CI slave scripts]/Exec[git_pull_jenkins CI slave scripts]: Scheduling refresh of Exec[git_submodule_update_jenkins CI slave scripts]
Notice: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Exec[systemd reload for xvfb]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Service[xvfb]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Service[xvfb]: Unscheduling refresh on Service[xvfb]

With chromium package installed, the VisualEditor/VisualEditor repo fails with Cannot start Chrome but there is not much details:

Running "karma:main" (karma) task
08 03 2016 16:10:33.706:INFO [karma]: Karma v0.13.21 server started at http://localhost:9876/
08 03 2016 16:10:33.723:INFO [launcher]: Starting browser Chrome
08 03 2016 16:10:34.714:ERROR [launcher]: Cannot start Chrome
    
08 03 2016 16:10:34.722:INFO [launcher]: Trying to start Chrome again (1/2).
08 03 2016 16:10:34.823:ERROR [launcher]: Cannot start Chrome
    
08 03 2016 16:10:34.827:INFO [launcher]: Trying to start Chrome again (2/2).
08 03 2016 16:10:34.941:ERROR [launcher]: Cannot start Chrome
    
08 03 2016 16:10:34.942:ERROR [launcher]: Chrome failed 2 times (cannot start). Giving up.
Warning: Task "karma:main" failed.Use --force to continue.

Aborted due to warnings.

Can be retried by commenting check experimental on a VE change.

@hashar But doint we need display set for npm 4.3 since on npm we have it set using a script but doint set it in 4.3.

@Paladox good catch! I rebased your patch https://gerrit.wikimedia.org/r/#/c/273269/8 which set DISPLAY directly from Zuul scheduler. Will merge and try again.

Change 273269 merged by jenkins-bot:
Sets DISPLAY in parameter_functions.py

https://gerrit.wikimedia.org/r/273269

@hashar thanks.

My VisualEditor sandbox change is https://gerrit.wikimedia.org/r/#/c/275264/ and DISPLAY=:94 does not fix it

@hashar still doesn't work maybe we will won't to create a script in Jenkins like we have for it any ways but one that is separate and add that to npm 4.3 and try that since I think @Krinkle said that it would not work that way.

Also visualeditor will need this https://gerrit.wikimedia.org/r/#/c/274441/ since it needs to git submodules update.

Is there anyway to debug that so we know where the problem is comming from.

Can you do what you did in mediawiki/core to debug karma and do it for this repo please.

The instances definitely have /usr/bin/chromium. They do not seem to have xvfb running though albeit https://gerrit.wikimedia.org/r/#/c/275832/ is supposed to ship it ...

The puppet run mention:

Notice: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Service[xvfb]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Service[xvfb]: Unscheduling refresh on Service[xvfb]

But Xvfb is not running for sure and:

$ systemctl list-units
Failed to get D-Bus connection: No such file or directory

So somehow the xvfb service is present when building the snapshot image:

debian    2381  0.0  0.0  12728  2132 pts/0    S+   21:32   0:00                  \_ grep -i xvfb
xvfb      2101  0.0  0.3  93428 13700 ?        Ss   21:32   0:00 /usr/bin/Xvfb :94 -screen 0 1280x1024x24 -ac -noreset

● xvfb.service - Permanent X virtual framebuffer
   Loaded: loaded (/lib/systemd/system/xvfb.service; disabled)
   Active: active (running) since Tue 2016-03-08 21:32:23 UTC; 13s ago
 Main PID: 2101 (Xvfb)
   CGroup: /system.slice/xvfb.service
           └─2101 /usr/bin/Xvfb :94 -screen 0 1280x1024x24 -ac -noreset
           
Mar 08 21:32:23 ci-jessie-wikimedia-1457472606 systemd[1]: Starting Permanent...
Mar 08 21:32:23 ci-jessie-wikimedia-1457472606 systemd[1]: Started Permanent ...

When booting the snapshot image via wikitech, there is a few worrying output in the boot messages:

[    6.457193] systemd-sysv-generator[132]: Ignoring creation of an alias umountiscsi.service for itself
[    6.815027] systemd[1]: Cannot add dependency job for unit dbus.socket, ignoring: Unit dbus.socket failed to load: No such file or directory.
[    6.817435] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.

systemd mentions:

[[32m  OK  [0m] Reached target Network.
[[32m  OK  [0m] Reached target Network is Online.

And the Xvfb service has:

[Unit]
Description=Permanent X virtual framebuffer
After=network.target

And it Xvfb is not started. I am blaming Unit display-manager.service failed to load

Image build could not provide Xvfb properly T129320

Mentioned in SAL [2016-03-09T10:29:43Z] <hashar> Nodepool: created new image and refreshing snapshot in attempt to get Xvfb running T129320 T128090

hashar closed subtask T129320: Diskimage fail on xvfb/systemd as Resolved.Mar 9 2016, 10:37 AM

hashar created subtask T129345: Xvfb service does not start on Nodepool instances.Mar 9 2016, 12:58 PM

status

Xvfb is running on Nodepool instances now.
DISPLAY=:94 is set via Zuul scheduler parameters.

Progress!!

It works now see https://integration.wikimedia.org/ci/job/npm-node-4.3/265/console :). Now we just need to update the visualeditor tests and switch it over.

Chromium is provided as well as DISPLAY set and Xvfb running. All of that done via the puppet class contint::browsers.

Krinkle added a parent task: T119143: Migrate javascript npm CI jobs to Nodepool.Mar 17 2016, 8:59 PM

Krinkle moved this task from Inbox to Meta on the OOjs core board.Aug 14 2020, 1:49 PM

Krinkle removed a project: Patch-For-Review.Dec 1 2020, 3:06 AM

Krinkle removed a project: OKR-Work.

Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testingClosed, ResolvedPublicActions

Details

Related ObjectsSearch...

Event Timeline

Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing
Closed, ResolvedPublic
Actions

Related Objects
Search...