When provisioning the puppet tree under /puppet a bunch of files are empty:
jenkins@ci-jessie-wikimedia-44086:~$ find /puppet/hieradata/ -type f -empty|wc -l 62
Examples:
/puppet/hieradata/eqiad.yaml /puppet/hieradata/labs.yaml
And thus hiera fails to lookup keys.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| dib: sync filesystem before snapshotting | integration/config | master | +4 -0 | |
| dib: fully clone /puppet | integration/config | master | +3 -1 |
That is definitely a fault in the disk image provisioning. The way I have set it up is the build host as a copy of git repositories. They are then git cloned under the image as bare repositories under /srv/git and we clone from there to populate /puppet.
I am suspecting one of the cloning phase ends up doing hardlinks that ends up being broken.
On an instance:
$ find / -inum 6059 2> /dev/null /puppet/hieradata/eqiad.yaml /sys/module/libata/parameters/acpi_gtf_filter
The later being owned by root.
The reference image created by dib is ci-jessie-wikimedia. Nodepool boot it up, run a setup script to refresh what was in the image, then snapshot that instance and will use the snapshot to build images.
On contintcloud labs project I have boot the image and the snapshot and search for empty files in /puppet using the very lame find /puppet -type f -empty|wc -l results:
| Image | 23 |
| Snapshot | 563 |
The script is https://phabricator.wikimedia.org/diffusion/CICF/browse/master/nodepool/scripts/setup_node.sh
When building the image, there is a git bare mirror created under /srv/git which is owned by root. To populate /puppet a git clone is made out of the mirror and that creates hardlink. Later the git mirrors in the build phase, the mirrors ownership is changed to be owned by jenkins. When the setup script runs sudo git -C /puppet pull the different ownership might be confusing git.
On labnodepool I have hacked the setup script to count empty files before and after the git -C /puppet pull and the count of empty file matches.
But when booting an instance from the image, the files are gone :-\
I have downloaded a snapshot image and looked at some random files, they are empty:
$ virt-ls --extra-stats -l -R -a ci-jessie-wikimedia-1457097785.qcow2 /puppet/hieradata/hosts - 0644 292 8:1 6332 1 0:0 8 /puppet/hieradata/hosts/restbase1009.yaml - 0644 0 8:1 6694 1 0:0 0 /puppet/hieradata/hosts/restbase1010.yaml - 0644 0 8:1 6718 1 0:0 0 /puppet/hieradata/hosts/restbase1011.yaml - 0644 0 8:1 6722 1 0:0 0 /puppet/hieradata/hosts/restbase1012.yaml - 0644 0 8:1 6725 1 0:0 0 /puppet/hieradata/hosts/restbase1013.yaml - 0644 0 8:1 8634 1 0:0 0 /puppet/hieradata/hosts/restbase1014.yaml - 0644 0 8:1 8635 1 0:0 0 /puppet/hieradata/hosts/restbase1015.yaml
927M Mar 1 11:15 image-jessie-20160301T100721Z.qcow2 1020M Mar 4 16:02 image-jessie-20160304T145330Z.qcow2
I have rebuild an image with the patch above image-t128846. Will see what happens over the week-end.
Mentioned in SAL [2016-03-04T19:31:51Z] <hashar> Nodepool Image ci-jessie-wikimedia-1457119603 in wmflabs-eqiad is ready - T128846
The snapshot ci-jessie-wikimedia-1457119603 has been made out of the image image-t128846 which has a /puppet cloned without hardlinks. Spawning an instance show the same behavior.
Fast rewind to 20 years ago when I have learned about the sync command to have buffers flushed/written to disk. Deemed as yet another useless UNIX command by the younger me.
The OpenStack documentation at http://docs.openstack.org/openstack-ops/content/snapshots.html quotes OpenStack: Perform Consistent Snapshots. It mentions sync and also recommend fsfreeze.
Image ci-jessie-wikimedia-1457121113 in wmflabs-eqiad is ready
Created with sync called in the setup script.
Change 275037 had a related patch set uploaded (by Hashar):
dib: sync filesystem before snapshotting
Image ci-jessie-wikimedia-1457122001 in wmflabs-eqiad is ready
Went back to use a clone with hardlinks but with sync at snapshot time. Seems the resulting instance is happy.
Change 274954 abandoned by Hashar:
dib: fully clone /puppet
Reason:
Unneeded. We really want 'sync'
Summary
An instance is booted from the reference image. It has a git workspace under /puppet. Nodepool git pull then snapshot the instance.
inodes have been written/registered but actual content did not make it to the file system. When booting from snapshot the result is files that got written by git pull are 0 bytes.
Invoking sync seems good enough to catch up.
I eventually found OpenStack has a doc for it http://docs.openstack.org/openstack-ops/content/snapshots.html
I believe the issue is now fixed. I am keeping this task open for monitoring.
So my earlier summary in T128846#2089789 was accurate and sync before snapshotting is enough to have a good snapshot. I have checked the two last snapshots created by Nodepool and they do not show the bad behavior.