Error: Could not prefetch package provider 'apt': Execution of '/usr/bin/apt-mark showmanual' returned 100: E: Write error - write (28: No space left on device) E: IO Error saving source cache E: The package lists or status file could not be parsed or opened.
This was weird. The first time I ran puppet I got the error message in T409254#11347692. When I checked to see the disk usage however I saw:
bd808@deployment-docker-cxserver02:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 979M 0 979M 0% /dev tmpfs 199M 544K 198M 1% /run /dev/sda1 20G 6.6G 13G 35% / tmpfs 993M 0 993M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda15 124M 11M 114M 9% /boot/efi tmpfs 199M 0 199M 0% /run/user/0 tmpfs 199M 0 199M 0% /run/user/3518
There seemed to be space everywhere. Just to see if I could get an error message that made more sense to me I ran apt update:
bd808@deployment-docker-cxserver02:~$ sudo apt update Get:1 http://apt.wikimedia.org/wikimedia bullseye-wikimedia InRelease [158 kB] Hit:2 http://mirrors.wikimedia.org/debian bullseye InRelease Hit:3 http://mirrors.wikimedia.org/debian bullseye-updates InRelease Get:4 http://security.debian.org/debian-security bullseye-security InRelease [27.2 kB] Hit:5 http://deb.debian.org/debian-debug bullseye-debug InRelease Get:6 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main Sources [49.7 kB] Get:7 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages [74.7 kB] Get:8 http://security.debian.org/debian-security bullseye-security/main Sources [272 kB] Get:9 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [422 kB] Get:10 http://security.debian.org/debian-security bullseye-security/main Translation-en [281 kB] Fetched 1283 kB in 1s (1728 kB/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done 5 packages can be upgraded. Run 'apt list --upgradable' to see them.
Ok, that didn't fail, but I thought that was what Puppet was complaining about? Oh. no I didn't read carefully. It was /usr/bin/apt-mark showmanual that blew things up. Ok:
bd808@deployment-docker-cxserver02:~$ /usr/bin/apt-mark showmanual acct ack adduser apparmor apt apt-utils augeas-tools base-files base-passwd bash bash-completion bind9-host bsd-mailx bsdutils byobu ca-certificates cloud-guest-utils cloud-init cloud-initramfs-growroot colordiff coreutils curl dash debconf debdeploy-client debian-archive-keyring debian-goodies debianutils diffutils dirmngr dnsutils docker.io dpkg dstat e2fsprogs ebtables ethtool exim4-config exim4-daemon-light fd-find findutils fzf gcc-10-base gcc-9-base gdb git gnupg golang-cfssl gpg gpgv grep grub-cloud-amd64 gzip hostname htop httpry icdiff ifupdown init init-system-helpers iotop iperf iptables iputils-ping isc-dhcp-client jq kitty-terminfo ldap-utils less libacl1 libapt-pkg6.0 libattr1 libaudit-common libaudit1 libblkid1 libbz2-1.0 libc-bin libc6 libcap-ng0 libcom-err2 libsemanage1 libsepol1 libsmartcols1 libss2 libssl1.1 libsss-sudo libstdc++6 libsystemd0 libtasn1-6 libtemplate-perl libtinfo6 libtirpc-common libtirpc3 libudev1 libunistring2 libuuid1 libxxhash0 libzstd1 linux-image-cloud-amd64 linux-perf lldpd login logrotate logsave lsb-base lshw man-db manpages mawk molly-guard moreutils mount mtr-tiny nano ncdu ncurses-base ncurses-bin netbase netcat-openbsd ngrep numactl openssh-server passwd pciutils perl-base pigz prometheus-node-exporter prometheus-rsyslog-exporter psm python3-keystoneclient python3-novaclient python3-openstackclient python3-prometheus-client python3-pyldap python3-tenacity python3-troveclient python3-wmflib quickstack reportbug resolvconf ripgrep rsyslog rsyslog-gnutls rsyslog-kafka ruby-net-ssh ruby-sys-filesystem runsc screen sed socat ssl-cert sssd strace sudo sysstat systemd-timesyncd sysvinit-utils tar tcpdump traceroute tree tshark tzdata unattended-upgrades util-linux uuid-runtime vim vim-addon-manager vim-scripts vim-tiny virt-what whiptail wipe xfsprogs xz-utils zlib1g zsh
That worked too... hmmm. What was Puppet's problem again?
bd808@deployment-docker-cxserver02:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for deployment-docker-cxserver02.deployment-prep.eqiad1.wikimedia.cloud Info: Applying configuration version '(5c2dc9de52) gitpuppet - puppetserver: Generalize git-rebase fix to work for labs/private' Notice: Applied catalog in 9.00 seconds bd808@deployment-docker-cxserver02:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for deployment-docker-cxserver02.deployment-prep.eqiad1.wikimedia.cloud Info: Applying configuration version '(5c2dc9de52) gitpuppet - puppetserver: Generalize git-rebase fix to work for labs/private' Notice: Applied catalog in 8.38 seconds
Nothing to see here I guess?