Page MenuHomePhabricator
Create Task
Maniphest T129345

Xvfb service does not start on Nodepool instances
Closed, ResolvedPublic
Actions

Description

Had to do some magic to get the xvfb systemd unit installed via puppet (T129320). It is now shipped properly but the service does not start on instance boot.

$ find / -name '*xvfb*' 2>/dev/null
/usr/share/man/man1/xvfb-run.1.gz
/usr/share/doc/xvfb
/usr/bin/xvfb-run
/puppet/modules/xvfb
/puppet/modules/xvfb/templates/initscripts/xvfb.systemd.erb
/puppet/modules/xvfb/templates/initscripts/xvfb.upstart.erb
/lib/systemd/system/xvfb.service
/var/lib/dpkg/info/xvfb.md5sums
/var/lib/dpkg/info/xvfb.list

We have /lib/systemd/system/xvfb.service but that does not seem sufficient to get it to spawn on boot. The unit file is provisioned by puppet and has:

[Unit]
Description=Permanent X virtual framebuffer
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/Xvfb :94 -screen 0 1280x1024x24 -ac -noreset
Restart=always
User=xvfb
Group=xvfb

[Install]
WantedBy=multi-user.target

And we have:

$ find /lib/systemd -iwholename '*multi-user*'
/lib/systemd/system/multi-user.target
/lib/systemd/system/multi-user.target.wants
/lib/systemd/system/multi-user.target.wants/systemd-logind.service
/lib/systemd/system/multi-user.target.wants/systemd-user-sessions.service
/lib/systemd/system/multi-user.target.wants/systemd-update-utmp-runlevel.service
/lib/systemd/system/multi-user.target.wants/getty.target
/lib/systemd/system/multi-user.target.wants/systemd-ask-password-wall.path
$ find /etc/systemd -iwholename '*multi-user*'
/etc/systemd/system/multi-user.target.wants
/etc/systemd/system/multi-user.target.wants/ssh.service
/etc/systemd/system/multi-user.target.wants/remote-fs.target
/etc/systemd/system/multi-user.target.wants/puppet.service
/etc/systemd/system/multi-user.target.wants/etcd.service

On a non Nodepool instance integration-slave-jessie1001.integration.eqiad.wmflabs the unit is loaded but marked as disabled

$ systemctl status xvfb
● xvfb.service - Permanent X virtual framebuffer
   Loaded: loaded (/lib/systemd/system/xvfb.service; disabled)
   Active: active (running) since Thu 2016-02-18 21:49:48 UTC; 2 weeks 5 days ago
 Main PID: 2942 (Xvfb)
   CGroup: /system.slice/xvfb.service
           └─2942 /usr/bin/Xvfb :94 -screen 0 1280x1024x24 -ac -noreset

$ systemctl show  xvfb|grep -i disab
UnitFileState=disabled

From https://www.freedesktop.org/wiki/Software/systemd/dbus/

UnitFileState
encodes the install state of the unit file of FragmentPath. It currently knows the following states: enabled, enabled-runtime, linked, linked-runtime, masked, masked-runtime, static, disabled, invalid.

enabled indicates that a unit file is permanently enabled.

enable-runtime indicates the unit file is only temporarily enabled, and will no longer be enabled after a reboot (that means, it is enabled via /run symlinks, rather than /etc).
...

static indicates that the unit is statically enabled, i.e. always enabled and doesn't need to be enabled explicitly.

invalid indicates that it could not be determined whether the unit file is enabled.

The documentation does not mention disabled but I guess that means the unit is not started on boot.

Looking at the instances console there is no mention of the unit description Permanent X virtual frame buffer. I suspect it is puppet starting up later on.

Details

SubjectRepoBranchLines +/-
xvfb: start service at boot timeoperations/puppetproduction+11 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Mar 9 2016, 12:58 PM
hashar added a comment.Mar 9 2016, 1:02 PM

Indeed after rebooting integration-slave-jessie1001:

$ systemctl status xvfb
● xvfb.service - Permanent X virtual framebuffer
   Loaded: loaded (/lib/systemd/system/xvfb.service; disabled)
   Active: inactive (dead)

Puppet runs on boot (unlike on Nodepool instances) and spawns xvfb:

Notice: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Service[xvfb]/ensure: ensure changed 'stopped' to 'running'

Puppet also starts redis which is also setup via base::service_unit

Notice: /Stage[main]/Role::Ci::Slave::Browsertests/Redis::Instance[6379]/Base::Service_unit[redis-instance-tcp_6379]/Service[redis-instance-tcp_6379]/ensure: ensure changed 'stopped' to 'running'

It seems the way the service is defined in puppet via base_unit is borked in the sense the service is not enabled on boot.

hashar added a comment.Mar 9 2016, 2:09 PM

And the xvfb puppet stanza in modules/xvfb/manifests/init.pp is:

base::service_unit { 'xvfb':
    ensure  => present,
    refresh => true,
    upstart => true,
    systemd => true,
    require => [
        Package['xvfb'],
        User['xvfb']
    ],
}
hashar added a comment.Mar 9 2016, 2:29 PM

I have dig in the systemctl documentation and the unit needs to be enabled:

$ sudo systemctl enable xvfb.service
Created symlink from /etc/systemd/system/multi-user.target.wants/xvfb.service to /lib/systemd/system/xvfb.service.
$ 

$ systemctl status xvfb.service
● xvfb.service - Permanent X virtual framebuffer
                                                     vvvvvvv
   Loaded: loaded (/lib/systemd/system/xvfb.service; enabled)
                                                     ^^^^^^^
   Active: active (running) since Wed 2016-03-09 13:00:18 UTC; 1h 27min ago
 Main PID: 2923 (Xvfb)
   CGroup: /system.slice/xvfb.service
           └─2923 /usr/bin/Xvfb :94 -screen 0 1280x1024x24 -ac -noreset
$

Can pass --now to have systemd start it immediately.

hashar added a comment.Mar 9 2016, 2:36 PM

Now that I have manually enabled the unit, it is started on boot (was not the case previously):

[[32m  OK  [0m] Reached target Timers.
[[32m  OK  [0m] Reached target Basic System.
         Starting Permanent X virtual framebuffer...
[[32m  OK  ] Started Permanent X virtual frame buffer.
gerritbot subscribed.Mar 9 2016, 2:44 PM

Change 276169 had a related patch set uploaded (by Hashar):
xvfb: start service at boot time

https://gerrit.wikimedia.org/r/276169

gerritbot added a project: Patch-For-Review.Mar 9 2016, 2:44 PM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.Mar 9 2016, 2:48 PM
hashar moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
gerritbot added a comment.Mar 9 2016, 2:50 PM

Change 276169 merged by Alexandros Kosiaris:
xvfb: start service at boot time

https://gerrit.wikimedia.org/r/276169

Stashbot subscribed.Mar 9 2016, 2:57 PM

Mentioned in SAL [2016-03-09T14:57:04Z] <hashar> Rebuilding snapshot image to get Xvfb enabled at boot time T129345

hashar added a comment.Mar 9 2016, 2:59 PM

Now puppet seems to enable xvfb properly:

Debug: Executing '/bin/systemctl is-active xvfb'
Debug: Executing '/bin/systemctl is-enabled xvfb'
Debug: Executing '/bin/systemctl start xvfb'
Debug: Executing '/bin/systemctl is-enabled xvfb'
Debug: Executing '/bin/systemctl enable xvfb'
Notice: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Service[xvfb]/ensure: ensure changed 'stopped' to 'running'

Keeping task open until I confirm the Nodepool instance actually spawn the Xvfb service at boot and that it is properly running.

Stashbot added a comment.Mar 9 2016, 2:59 PM

Mentioned in SAL [2016-03-09T14:59:54Z] <hashar> Image ci-jessie-wikimedia-1457535250 in wmflabs-eqiad is ready T129345

hashar closed this task as Resolved.Mar 9 2016, 3:08 PM
jenkins@ci-jessie-wikimedia-47306:~$ ps -A f|grep -i xvfb
  375 ?        Ss     0:00 /usr/bin/Xvfb :94 -screen 0 1280x1024x24 -ac -noreset

And looking at console:

[[32m  OK  [0m] Reached target Basic System.
         Starting Permanent X virtual framebuffer...
[[32m  OK  [0m] Started Permanent X virtual framebuffer.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL