| hashar | |
| Apr 20 2016, 7:37 PM |
| F3929760: Capture d’écran 2016-04-25 à 21.44.29.png | |
| Apr 25 2016, 7:45 PM |
From T125821: Provide a HHVM package for jessie-wikimedia matching version of trusty-wikimedia we now have a hhvm package for Jessie. There are concerned about ICU lib version being different between Trusty and Jessie and @akosiaris suggested to run the tests on the same OS as what is used on production.
This task is about creating a disk image using Trusty and see what happens.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T119138 [keyresult] Migrate majority of CI jobs to Nodepool (part 2) | |||
| Resolved | hashar | T119139 [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool | |||
| Resolved | hashar | T133203 Attempt to provide a Trusty image for Nodepool | |||
| Restricted Task |
Change 284900 had a related patch set uploaded (by Hashar):
dib: glue for Ubuntu Trusty imaging
I have tried to build an image and puppet apply fails with invalid option: --test
Preparing puppet links... removed directory: '/etc/puppet/files' '/etc/puppet/files' -> '/puppet/files' '/etc/puppet/hieradata' -> '/puppet/hieradata' removed directory: '/etc/puppet/manifests' '/etc/puppet/manifests' -> '/puppet/manifests' removed directory: '/etc/puppet/modules' '/etc/puppet/modules' -> '/puppet/modules' removed directory: '/etc/puppet/templates' '/etc/puppet/templates' -> '/puppet/templates' Applying puppet manifests ... Applying puppet manifest /tmp/in_target.d/wikimedia-puppet/bootstrap.pp Error: Could not parse application options: invalid option: --test Puppet run failed. Exit code 1 Starting debug shell. Exit to resume building. At stage after-error
The command being:
puppet apply --test --verbose --modulepath=$PUPPET_DIR/modules
Jessie has puppet 3.7.2 and the apply command comes with:
* --test: Enable the most common options used for testing. These are 'verbose', 'detailed-exitcodes' and 'show_diff'.
Trusty has puppet 3.4.3 which does not have --test.
Patchset 6 expands --test to --detailed-exitcodes --show_diff
https://gerrit.wikimedia.org/r/#/c/284900/5..6/dib/elements/wikimedia/bin/puppet-apply,cm
Later fails with:
Upstart/ missing packages solved in PS8 https://gerrit.wikimedia.org/r/#/c/284900/7..8/dib/puppet/ciimage.pp,cm
Now fails doing the final dist-upgrade:
Performing apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be DOWNGRADED: libpcre3 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded. Need to get 275 kB of archives. After this operation, 127 kB of additional disk space will be used. E: There are problems and -y was used without --force-yes
# apt-cache policy libpcre3
libpcre3:
Installed: 1:8.31-2ubuntu2.3
Candidate: 1:8.31-2ubuntu2.1+wm1
Version table:
*** 1:8.31-2ubuntu2.3 0
500 http://mirrors.wikimedia.org/ubuntu/ trusty-updates/main amd64 Packages
100 /var/lib/dpkg/status
1:8.31-2ubuntu2.2 0
500 http://mirrors.wikimedia.org/ubuntu/ trusty-security/main amd64 Packages
1:8.31-2ubuntu2.1+wm1 0
1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/main amd64 Packages
1:8.31-2ubuntu2 0
500 http://mirrors.wikimedia.org/ubuntu/ trusty/main amd64 PackagesDebootstrap got us 1:8.31-2ubuntu2.3 from trusty-updates.
trusty-wikimedia is pinned with an higher priority '1001' and comes with 1:8.31-2ubuntu2.1+wm1.
Change 285173 had a related patch set uploaded (by Hashar):
dib: allow dist-upgrade to downgrade package
Filled T133528 about libpcre3. Meanwhile I have hacked dist-upgrade to allow downgrade with APT::Get::force-yes "true" with https://gerrit.wikimedia.org/r/285173
Mentioned in SAL [2016-04-25T13:15:00Z] <hashar> openstack image create --file /home/hashar/image-trusty-20160425T124552Z.qcow2 ci-trusty-wikimedia --disk-format qcow2 --property show=true # T133203
Change 285178 had a related patch set uploaded (by Hashar):
nodepool: pool in 2 TRUSTY instances
Mentioned in SAL [2016-04-25T13:34:19Z] <hashar> Nodepool is attempting to create a Trusty snapshot with name ci-trusty-wikimedia-1461591203 | T133203
The image more or less boot on labs but fails to acquire network connection (ping does not work) :-( There are now specific informations in the Horizon log.
Some boot trouble I had with Jessie back in July 2015 is documented via T105152, specially T105152#1440683 gives some debug steps.
The console log is:
Change 285184 had a related patch set uploaded (by Hashar):
dib: enable network on Trusty image
Needed the simple-init element to be added. Now fails with:
* Zuul Merger: /etc/default/zuul-merger is not set to START_DAEMON=1: exiting [80G [74G[[31mfail[39;49m] + * Starting Bridge socket events into upstart[74G[ OK ] +debconf: unable to initialize frontend: Dialog +debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.) +debconf: falling back to frontend: Readline +debconf: unable to initialize frontend: Readline +debconf: (This frontend requires a controlling tty.) +debconf: falling back to frontend: Teletype +dpkg-query: package 'openssh-server' is not installed and no information is available +Use dpkg --info (= dpkg-deb --info) to examine archive files, +and dpkg --contents (= dpkg-deb --contents) to list their contents. +/usr/sbin/dpkg-reconfigure: openssh-server is not installed + * Stopping Run glean to configure network interfaces[74G[ OK ]
dpkg-query: package 'openssh-server' is not installed and no information is available
`
Change 285186 had a related patch set uploaded (by Hashar):
dib: install openssh-server for Trusty
Booted a Trusty image:
* Starting OpenSSH server[74G[ OK ] ssh start/running, process 687
I managed to ssh to it from labnodepool1001.eqiad.wmnet as jenkins. \O/
Change 285194 had a related patch set uploaded (by Hashar):
dib: always install cloud-init
cloud-init ends up lacking the EC2 datasource and instead has None, None. Seems to be due to the simple-init element.
Will try again tomorrow morning as time allow.
Have to replace elements ubuntu-minimal, simple-init with simply ubuntu and:
Cloud-init v. 0.7.5 finished at Mon, 25 Apr 2016 18:00:46 +0000. Datasource DataSourceEc2. Up 18.65 seconds
Change 285218 had a related patch set uploaded (by Hashar):
dib: fix cloud-init on Trusty to use Ec2
Summary
Provisioning a Trusty image was more or less straightforward. A bunch of puppet classes have been stripped out since they are meant for Jessie and no jobs will make use of them in Trusty anyway.
I have confirmed Nodepool manage to boot and snapshot an instance out of the Trusty image. The one thing left is apply Nodepool setting to let it know about the Trusty image which is https://gerrit.wikimedia.org/r/#/c/285178/
Change 285194 abandoned by Hashar:
dib: always install cloud-init
Reason:
Replaced 'ubuntu-minimal' by 'ubuntu' and it seems to work fine.
Mentioned in SAL [2016-04-25T19:39:50Z] <hashar> Nodepool now booting 2 trusty instances for CI needs. T133203
Nodepool processed:
2016-04-25 19:41 INFO nodepool.image.build.wmflabs-eqiad.ci-trusty-wikimedia: ./setup_node.sh complete (hostname: ci-trusty-wikimedia-1461613142) INFO nodepool.SnapshotImageUpdater: Image ci-trusty-wikimedia-1461613142 in wmflabs-eqiad is ready INFO nodepool.NodePool: Need to launch 2 ci-trusty-wikimedia nodes for gallium.wikimedia.org on wmflabs-eqiad INFO nodepool.NodeLauncher: Creating server with hostname ci-trusty-wikimedia-83124 in wmflabs-eqiad from image ci-trusty-wikimedia for node id: 83124 INFO nodepool.NodeLauncher: Creating server with hostname ci-trusty-wikimedia-83125 in wmflabs-eqiad from image ci-trusty-wikimedia for node id: 83125 INFO nodepool.NodeLauncher: Node id: 83124 added to jenkins INFO nodepool.NodeLauncher: Node id: 83125 added to jenkins
They are available in Jenkins and ready to be consumed:
libpcre3 is in theory still a blocker but is worked around by forcing dist-upgrade to downgrade.
Mentioned in SAL [2016-04-25T19:46:19Z] <hashar> Nodepool now has a couple trusty instances intended to experiment with Zend 5.5 / HHVM migration . https://phabricator.wikimedia.org/T133203#2236625