Change Details

It seems that the puppetdb-populate script fails for hosts that don't already have data in the puppet instance of the compiler server. From testing it seems that the puppet master --compile command no longer reads the facts cache if you are using `storeconfigs_backend=puppetdb`. withthe current policuy we see this with errors like the following ``` Compiling catalog for mw1363.eqiad.wmnet [ 2020-03-26T16:35:01 ] ERROR: Compilation failed for host mw1363.eqiad.wmnet: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdbf7e2GJ/1/production/private/modules:/tmp/fill-puppetdbf7e2GJ/1/production/src/modules', '--confdir=/tmp/fill-puppetdbf7e2GJ/1/production/src', '--compile=mw1363.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30 Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/puppet_compiler/puppet.py", line 83, in compile_storeconfigs subprocess.check_call(cmd, stdout=out, stderr=err, env=env) File "/usr/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdbf7e2GJ/1/production/private/modules:/tmp/fill-puppetdbf7e2GJ/1/production/src/modules', '--confdir=/tmp/fill-puppetdbf7e2GJ/1/production/src', '--compile=mw1363.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30 Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` We can see that this server is not allready in the database with the following ``` $ curl -X GET 'http://localhost:8080/pdb/query/v4/nodes' | jq .[].certname |grep mw1363.eqiad.wmnet: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 198k 0 198k 0 0 3201k 0 --:--:-- --:--:-- --:--:-- 3201k ``` to test things further i created a manifest with the following ``` lang=puppet notice($facts) fail('bomb out') ``` If i run this with `--storeconfigs --storeconfigs_backend=puppetdb` we get the following output ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/mnt/home/jenkins-deploy/puppet_base/1/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Scope(Class[main]): {} Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` however if i run it without the config we see the facts are loaded ``` lines=10 $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/mnt/home/jenkins-deploy/puppet_base/1/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Scope(Class[main]): {agent_specified_environment => production, architecture => amd64, augeas => {version => 1.8.0}, augeasversion => 1.8.0, bios_release_date => 06/14/2019, bios_vendor => Dell Inc., bios_version => 2.2.11, blockdevice_sda_model => SSDSC2KB960G8R, blockdevice_sda_size => 960197124096, blockdevice_sda_vendor => ATA, blockdevice_sdb_model => SSDSC2KB960G8R, blockdevice_sdb_size => 960197124096, blockdevice_sdb_vendor => ATA, blockdevices => sdb,sda, boardmanufacturer => Dell Inc., boardproductname => 424242, boardserialnumber => 4242, chassistype => Rack Mount Chassis, clientcert => mw1363.eqiad.wmnet, clientnoop => false, clientversion => 5.5.10, cpu_details => {cpus => {cpu0 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu1 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu10 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu11 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu12 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu13 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu14 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu15 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu16 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu17 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu18 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu19 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu2 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu20 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu21 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu22 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu23 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu24 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu25 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu26 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu27 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu28 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu29 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu3 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu30 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu31 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu32 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu33 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu34 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu35 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu36 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu37 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu38 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu39 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu4 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu5 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu6 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu7 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu8 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu9 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpufreq => {scaling_driver => false, scaling_governor => false}, cpuidle => {scaling_driver => false, scaling_governor => false}}, scaling_driver => intel_pstate, scaling_governor => powersave, vulnerabilities => {itlb_multihit => KVM: Vulnerable, l1tf => Mitigation: PTE Inversion, mds => Mitigation: Clear CPU buffers; SMT vulnerable, meltdown => Mitigation: PTI, spec_store_bypass => Mitigation: Speculative Store Bypass disabled via prctl and seccomp, spectre_v1 => Mitigation: usercopy/swapgs barriers and __user pointer sanitization, spectre_v2 => Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling, tsx_async_abort => Mitigation: Clear CPU buffers; SMT vulnerable}}, cpu_scaling_driver => intel_pstate, cpu_scaling_governor => powersave, default_routes => {ipv4 => 10.64.48.1, ipv6 => fe80::1}, disks => {sda => {model => SSDSC2KB960G8R, size => 894.25 GiB, size_bytes => 960197124096, vendor => ATA}, sdb => {model => SSDSC2KB960G8R, size => 894.25 GiB, size_bytes => 960197124096, vendor => ATA}}, dmi => {bios => {release_date => 06/14/2019, vendor => Dell Inc., version => 2.2.11}, board => {manufacturer => Dell Inc., product => 08CYF7, serial_number => .9WYK4Z2.CNFCP0097U01F5.}, chassis => {type => Rack Mount Chassis}, manufacturer => Dell Inc., product => {name => PowerEdge R440, serial_number => 9WYK4Z2, uuid => 4C4C4544-0057-5910-804B-B9C04F345A32}}, domain => eqiad.wmnet, facterversion => 3.11.0, filesystems => ext2,ext3,ext4, fips_enabled => false, fqdn => mw1363.eqiad.wmnet, gid => root, hardwareisa => unknown, hardwaremodel => x86_64, has_ipmi => true, hostname => mw1363, id => root, identity => {gid => 0, group => root, privileged => true, uid => 0, user => root}, initsystem => systemd, interface_primary => eno1, interfaces => eno1,eno2,lo,lo:LVS, ipaddress => 10.64.48.205, ipaddress6 => 2620:0:861:107:10:64:48:205, ipaddress6_eno1 => 2620:0:861:107:10:64:48:205, ipaddress6_lo => :1, ipaddress_eno1 => 10.64.48.205, ipaddress_lo => 127.0.0.1, ipaddress_lo:LVS => 10.2.2.22, ipmi_chassis => {boot_flags => {console_redirection => BIOS-SETTING, device => NO-OVERRIDE, persistent => }, front_panel => {power_button_enabled => , reset_button_enabled => , standby_button_enabled => }, power => {restore_policy => Restore_State_AC_Apply}}, ipmi_lan => {gateway => 10.65.0.1, ipaddress => 10.65.2.42, macaddress => 4c:d9:8f:89:0b:82, netmask => 255.255.0.0}, is_pe => false, is_virtual => false, kernel => Linux, kernel_details => {release => 4.9.0-12-amd64, sysctl_settings => {net.ipv4.tcp_min_snd_mss => true}, version => #1 SMP Debian 4.9.210-1 (2020-01-20)}, kernelmajversion => 4.9, kernelrelease => 4.9.0-12-amd64, kernelversion => 4.9.0, lldp => {eno1 => {neighbor => asw2-d-eqiad, port => ge-3/0/40, vlan => vlan-1020}}, lldp_neighbors => [asw2-d-eqiad], lldp_parent => asw2-d-eqiad, load_averages => {15m => 10.81, 1m => 12.33, 5m => 11.27}, lsbdistcodename => stretch, lsbdistdescription => Debian GNU/Linux 9.12 (stretch), lsbdistid => Debian, lsbdistrelease => 9.12, lsbmajdistrelease => 9, lsbminordistrelease => 12, lvm_pv_0 => /dev/md0, lvm_pvs => 1, lvm_support => true, lvm_vg_0 => mw1363-vg, lvm_vg_mw1363-vg_pvs => /dev/md0, lvm_vgs => 1, macaddress => 4c:d9:8f:89:0b:88, macaddress_eno1 => 4c:d9:8f:89:0b:88, macaddress_eno2 => 4c:d9:8f:89:0b:89, manufacturer => Dell Inc., memory => {system => {available => 50.30 GiB, available_bytes => 54007877632, capacity => 19.34%, total => 62.36 GiB, total_bytes => 66960048128, used => 12.06 GiB, used_bytes => 12952170496}}, memoryfree => 50.30 GiB, memoryfree_mb => 51505.92578125, memorysize => 62.36 GiB, memorysize_mb => 63858.078125, mountpoints => {/ => {available => 501.91 GiB, available_bytes => 538919391232, capacity => 6.86%, device => /dev/mapper/mw1363--vg-root, filesystem => ext4, options => [rw, relatime, errors=remount-ro, data=ordered], size => 538.86 GiB, size_bytes => 578596683776, used => 36.95 GiB, used_bytes => 39677292544}, /dev/shm => {available => 31.18 GiB, available_bytes => 33480019968, capacity => 0.00%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, nodev], size => 31.18 GiB, size_bytes => 33480024064, used => 4.00 KiB, used_bytes => 4096}, /run => {available => 5.59 GiB, available_bytes => 6006710272, capacity => 10.29%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, noexec, relatime, size=6539068k, mode=755], size => 6.24 GiB, size_bytes => 6696005632, used => 657.36 MiB, used_bytes => 689295360}, /run/lock => {available => 5.00 MiB, available_bytes => 5242880, capacity => 0%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, nodev, noexec, relatime, size=5120k], size => 5.00 MiB, size_bytes => 5242880, used => 0 bytes, used_bytes => 0}, /srv => {available => 162.73 GiB, available_bytes => 174732419072, capacity => 10.67%, device => /dev/mapper/mw1363--vg-srv, filesystem => ext4, options => [rw, relatime, data=ordered], size => 182.16 GiB, size_bytes => 195593924608, used => 19.43 GiB, used_bytes => 20861505536}, /sys/fs/cgroup => {available => 31.18 GiB, available_bytes => 33480024064, capacity => 0%, device => tmpfs, filesystem => tmpfs, options => [ro, nosuid, nodev, noexec, mode=755], size => 31.18 GiB, size_bytes => 33480024064, used => 0 bytes, used_bytes => 0}, /var/lib/nginx => {available => 1023.93 MiB, available_bytes => 1073672192, capacity => 0.01%, device => tmpfs, filesystem => tmpfs, options => [rw, noatime, size=1048576k, mode=755], size => 1.00 GiB, size_bytes => 1073741824, used => 68.00 KiB, used_bytes => 69632}}, mtu_eno1 => 1500, mtu_eno2 => 1500, mtu_lo => 65536, net_driver => {eno1 => {driver => tg3, duplex => full, firmware_version => FFV21.40.9 bc 5720-v1.39, speed => 1000}, eno2 => {driver => tg3, duplex => unknown, firmware_version => FFV21.40.9 bc 5720-v1.39, speed => -1}}, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, netmask6_eno1 => ffff:ffff:ffff:ffff::, netmask6_lo => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, netmask_eno1 => 255.255.252.0, netmask_lo => 255.0.0.0, netmask_lo:LVS => 255.255.255.255, network => 10.64.48.0, network6 => 2620:0:861:107::, network6_eno1 => 2620:0:861:107::, network6_lo => :1, network_eno1 => 10.64.48.0, network_lo => 127.0.0.0, network_lo:LVS => 10.2.2.22, networking => {domain => eqiad.wmnet, fqdn => mw1363.eqiad.wmnet, hostname => mw1363, interfaces => {eno1 => {bindings => [{address => 10.64.48.205, netmask => 255.255.252.0, network => 10.64.48.0}], bindings6 => [{address => 2620:0:861:107:10:64:48:205, netmask => ffff:ffff:ffff:ffff::, network => 2620:0:861:107::}, {address => fe80::4ed9:8fff:fe89:b88, netmask => ffff:ffff:ffff:ffff::, network => fe80::}], ip => 10.64.48.205, ip6 => 2620:0:861:107:10:64:48:205, mac => 4c:d9:8f:89:0b:88, mtu => 1500, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, network => 10.64.48.0, network6 => 2620:0:861:107::}, eno2 => {mac => 4c:d9:8f:89:0b:89, mtu => 1500}, lo => {bindings => [{address => 127.0.0.1, netmask => 255.0.0.0, network => 127.0.0.0}], bindings6 => [{address => ::1, netmask => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, network => ::1}], ip => 127.0.0.1, ip6 => :1, mtu => 65536, netmask => 255.0.0.0, netmask6 => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, network => 127.0.0.0, network6 => :1}, lo:LVS => {bindings => [{address => 10.2.2.22, netmask => 255.255.255.255, network => 10.2.2.22}], ip => 10.2.2.22, netmask => 255.255.255.255, network => 10.2.2.22}}, ip => 10.64.48.205, ip6 => 2620:0:861:107:10:64:48:205, mac => 4c:d9:8f:89:0b:88, mtu => 1500, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, network => 10.64.48.0, network6 => 2620:0:861:107::, primary => eno1}, numa => {device_to_htset => {eno1 => [[0, 20], [2, 22], [4, 24], [6, 26], [8, 28], [10, 30], [12, 32], [14, 34], [16, 36], [18, 38]], eno2 => [[0, 20], [2, 22], [4, 24], [6, 26], [8, 28], [10, 30], [12, 32], [14, 34], [16, 36], [18, 38]], lo => [[0, 20], [1, 21], [2, 22], [3, 23], [4, 24], [5, 25], [6, 26], [7, 27], [8, 28], [9, 29], [10, 30], [11, 31], [12, 32], [13, 33], [14, 34], [15, 35], [16, 36], [17, 37], [18, 38], [19, 39]]}, device_to_node => {eno1 => [0], eno2 => [0], lo => [0, 1]}, nodes => [0, 1]}, operatingsystem => Debian, operatingsystemmajrelease => 9, operatingsystemrelease => 9.12, os => {architecture => amd64, distro => {codename => stretch, description => Debian GNU/Linux 9.12 (stretch), id => Debian, release => {full => 9.12, major => 9, minor => 12}}, family => Debian, hardware => x86_64, name => Debian, release => {full => 9.12, major => 9, minor => 12}, selinux => {enabled => false}}, osfamily => Debian, package_provider => apt, partitions => {/dev/mapper/mw1363--vg-placeholder => {size => 159.57 GiB, size_bytes => 171341512704}, /dev/mapper/mw1363--vg-root => {filesystem => ext4, mount => /, size => 548.47 GiB, size_bytes => 588913836032, uuid => 772dd9f2-664c-4ac6-9b81-31e6a6e1eb20}, /dev/mapper/mw1363--vg-srv => {filesystem => ext4, mount => /srv, size => 186.08 GiB, size_bytes => 199804059648, uuid => bb262acb-fe78-47ef-829b-430a4c74082d}, /dev/sda1 => {filesystem => linux_raid_member, label => mw1363:0, partuuid => 41ba7b92-01, size => 894.25 GiB, size_bytes => 960195723264, uuid => 1036e212-5b51-7d43-bf9f-d39b674279cc}, /dev/sdb1 => {filesystem => linux_raid_member, label => mw1363:0, partuuid => 1da3f988-01, size => 894.25 GiB, size_bytes => 960195723264, uuid => 1036e212-5b51-7d43-bf9f-d39b674279cc}}, path => /usr/bin:/bin:/usr/sbin:/sbin, physicalcorecount => 20, physicalprocessorcount => 2, processor0 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor1 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor10 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor11 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor12 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor13 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor14 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor15 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor16 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor17 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor18 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor19 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor2 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor20 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor21 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor22 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor23 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor24 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor25 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor26 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor27 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor28 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor29 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor3 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor30 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor31 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor32 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor33 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor34 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor35 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor36 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor37 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor38 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor39 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor4 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor5 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor6 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor7 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor8 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor9 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processorcount => 40, processors => {count => 40, isa => unknown, models => [Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz], physicalcount => 2, speed => 3.00 GHz}, productname => PowerEdge R440, puppet_config => {hostcert => /var/lib/puppet/ssl/certs/mw1363.eqiad.wmnet.pem, hostprivkey => /var/lib/puppet/ssl/private_keys/mw1363.eqiad.wmnet.pem, hostpubkey => /var/lib/puppet/ssl/public_keys/mw1363.eqiad.wmnet.pem, localcacert => /var/lib/puppet/ssl/certs/ca.pem, master => {localcacert => /var/lib/puppet/ssl/certs/ca.pem, ssldir => /var/lib/puppet/ssl}, ssldir => /var/lib/puppet/ssl}, puppet_config_dir => /etc/puppet, puppet_environmentpath => /etc/puppet/code/environments, puppet_server => puppet, puppet_vardir => /var/lib/puppet, puppetversion => 5.5.10, raid => [md], root_home => /root, ruby => {platform => x86_64-linux-gnu, sitedir => /usr/local/lib/site_ruby/2.3.0, version => 2.3.3}, rubyplatform => x86_64-linux-gnu, rubysitedir => /usr/local/lib/site_ruby/2.3.0, rubyversion => 2.3.3, selinux => false, serialnumber => 42424242, service_provider => systemd, ssh => {ecdsa => {fingerprints => {sha1 => SSHFP 3 1 a255543df7ba8104776a436e026da5898109e007, sha256 => SSHFP 3 2 ae0740d6e2e874006dfc8e978a6d432f5cdb242df7b3d63d68b36372c6629586}, key => AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLR2M0a/PGMQt9p0Kts681DcYi2pLq7if8Damrx+Sqk/MZhTAt3e+jVaEU2YDvCnfByyOWC0zriv+vMlIpBheE=, type => ecdsa-sha2-nistp256}, ed25519 => {fingerprints => {sha1 => SSHFP 4 1 0cfeeca9fb048d72289f4ca625be78551dc0c93c, sha256 => SSHFP 4 2 58e8b53761000b1f4e9969784d639da4c2335bcedb27451baf824f59c02420fc}, key => AAAAC3NzaC1lZDI1NTE5AAAAIG8NE30ToNK8YDnaBYJNJTI1E/XgqpdrCQCt2LzPVIVT, type => ssh-ed25519}, rsa => {fingerprints => {sha1 => SSHFP 1 1 f726aaf6e0a29d89e4b402842b1a522526f324a5, sha256 => SSHFP 1 2 ae9a85bb60083e74f9640fcf9cd04cb3d52f36af5ed88d99a0e04e022e95c27c}, key => AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Kr2MfD8BZrUQO3DP3MnSW8imbtSR7d6dRXsOxwhuNAdfwCcJSdiZYdpa+B7aXqWLcIA2nubQKJLEK2cqVSBk4cLZdfEFyejZdOtPQXtPN3wSP44LDwbMRZanE/8FDMX/BjqauHqMQYFmLBzPWRGU4xeA8Wq7YFd6A1fBd6CxwNxo3DRMKLoQbtyLCPCaNjOE3vwo23EiFap4zQ7oRzCxFe2hXPJfPWNnBAkY+/2Syv/A5TFXNLAIq6E1RXsidLzA562kI5yUk8LZ29a2Zzm5wWsGIZF7+TsJVZlDSfeDbOlE7KNPe8fVAtjbLO19u71K9M1oJ5eqpdZSb603OoZ9, type => ssh-rsa}}, sshecdsakey => AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLR2M0a/PGMQt9p0Kts681DcYi2pLq7if8Damrx+Sqk/MZhTAt3e+jVaEU2YDvCnfByyOWC0zriv+vMlIpBheE=, sshed25519key => AAAAC3NzaC1lZDI1NTE5AAAAIG8NE30ToNK8YDnaBYJNJTI1E/XgqpdrCQCt2LzPVIVT, sshfp_ecdsa => SSHFP 3 1 a255543df7ba8104776a436e026da5898109e007 SSHFP 3 2 ae0740d6e2e874006dfc8e978a6d432f5cdb242df7b3d63d68b36372c6629586, sshfp_ed25519 => SSHFP 4 1 0cfeeca9fb048d72289f4ca625be78551dc0c93c SSHFP 4 2 58e8b53761000b1f4e9969784d639da4c2335bcedb27451baf824f59c02420fc, sshfp_rsa => SSHFP 1 1 f726aaf6e0a29d89e4b402842b1a522526f324a5 SSHFP 1 2 ae9a85bb60083e74f9640fcf9cd04cb3d52f36af5ed88d99a0e04e022e95c27c, sshrsakey => AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Kr2MfD8BZrUQO3DP3MnSW8imbtSR7d6dRXsOxwhuNAdfwCcJSdiZYdpa+B7aXqWLcIA2nubQKJLEK2cqVSBk4cLZdfEFyejZdOtPQXtPN3wSP44LDwbMRZanE/8FDMX/BjqauHqMQYFmLBzPWRGU4xeA8Wq7YFd6A1fBd6CxwNxo3DRMKLoQbtyLCPCaNjOE3vwo23EiFap4zQ7oRzCxFe2hXPJfPWNnBAkY+/2Syv/A5TFXNLAIq6E1RXsidLzA562kI5yUk8LZ29a2Zzm5wWsGIZF7+TsJVZlDSfeDbOlE7KNPe8fVAtjbLO19u71K9M1oJ5eqpdZSb603OoZ9, system_uptime => {days => 41, hours => 1003, seconds => 3611565, uptime => 41 days}, timezone => UTC, uniqueid => 43434343, uptime => 41 days, uptime_days => 41, uptime_hours => 1003, uptime_seconds => 3611565, uuid => 4C4C4544-0057-5910-804B-B9C04F345A32, virtual => physical} Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` This issue is preventing new nodes from populating the puppetdb. One work around i have noticed is that i can compile the hosts an empty manifest i.e. `--manifests=/dev/null` and this causes enough meta data to get populated into puppetdb that further compilations using the full production catalouge work with out issue Initial run ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=$confdir/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` manifest=/dev/null ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/dev/null --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Compiled catalog for mw1363.eqiad.wmnet in environment production in 0.04 seconds { ... Empty catalouge ... } ``` another standard run ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=$confdir/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet { ... Full catalouge ... } ``` T150456 seems like it may be simlar

It seems that the puppetdb-populate script fails for hosts that don't already have data in the puppet instance of the compiler server. From testing it seems that the puppet master --compile command no longer reads the facts cache if you are using `storeconfigs_backend=puppetdb`. withthe current policuy we see this with errors like the following ``` Compiling catalog for mw1363.eqiad.wmnet [ 2020-03-26T16:35:01 ] ERROR: Compilation failed for host mw1363.eqiad.wmnet: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdbf7e2GJ/1/production/private/modules:/tmp/fill-puppetdbf7e2GJ/1/production/src/modules', '--confdir=/tmp/fill-puppetdbf7e2GJ/1/production/src', '--compile=mw1363.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30 Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/puppet_compiler/puppet.py", line 83, in compile_storeconfigs subprocess.check_call(cmd, stdout=out, stderr=err, env=env) File "/usr/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdbf7e2GJ/1/production/private/modules:/tmp/fill-puppetdbf7e2GJ/1/production/src/modules', '--confdir=/tmp/fill-puppetdbf7e2GJ/1/production/src', '--compile=mw1363.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30 Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` We can see that this server is not allready in the database with the following ``` $ curl -X GET 'http://localhost:8080/pdb/query/v4/nodes' | jq .[].certname |grep mw1363.eqiad.wmnet: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 198k 0 198k 0 0 3201k 0 --:--:-- --:--:-- --:--:-- 3201k ``` to test things further i created a manifest with the following ``` lang=puppet notice($facts) fail('bomb out') ``` If i run this with `--storeconfigs --storeconfigs_backend=puppetdb` we get the following output ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/mnt/home/jenkins-deploy/puppet_base/1/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Scope(Class[main]): {} Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` however if i run it without the config we see the facts are loaded ``` lines=10 $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/mnt/home/jenkins-deploy/puppet_base/1/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Scope(Class[main]): {agent_specified_environment => production, architecture => amd64, augeas => {version => 1.8.0}, augeasversion => 1.8.0, bios_release_date => 06/14/2019, bios_vendor => Dell Inc., bios_version => 2.2.11, blockdevice_sda_model => SSDSC2KB960G8R, blockdevice_sda_size => 960197124096, blockdevice_sda_vendor => ATA, blockdevice_sdb_model => SSDSC2KB960G8R, blockdevice_sdb_size => 960197124096, blockdevice_sdb_vendor => ATA, blockdevices => sdb,sda, boardmanufacturer => Dell Inc., boardproductname => 424242, boardserialnumber => 4242, chassistype => Rack Mount Chassis, clientcert => mw1363.eqiad.wmnet, clientnoop => false, clientversion => 5.5.10, cpu_details => {cpus => {cpu0 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu1 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu10 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu11 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu12 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu13 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu14 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu15 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu16 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu17 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu18 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu19 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu2 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu20 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu21 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu22 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu23 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu24 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu25 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu26 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu27 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu28 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu29 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu3 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu30 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu31 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu32 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu33 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu34 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu35 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu36 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu37 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu38 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu39 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu4 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu5 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu6 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu7 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu8 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu9 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpufreq => {scaling_driver => false, scaling_governor => false}, cpuidle => {scaling_driver => false, scaling_governor => false}}, scaling_driver => intel_pstate, scaling_governor => powersave, vulnerabilities => {itlb_multihit => KVM: Vulnerable, l1tf => Mitigation: PTE Inversion, mds => Mitigation: Clear CPU buffers; SMT vulnerable, meltdown => Mitigation: PTI, spec_store_bypass => Mitigation: Speculative Store Bypass disabled via prctl and seccomp, spectre_v1 => Mitigation: usercopy/swapgs barriers and __user pointer sanitization, spectre_v2 => Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling, tsx_async_abort => Mitigation: Clear CPU buffers; SMT vulnerable}}, cpu_scaling_driver => intel_pstate, cpu_scaling_governor => powersave, default_routes => {ipv4 => 10.64.48.1, ipv6 => fe80::1}, disks => {sda => {model => SSDSC2KB960G8R, size => 894.25 GiB, size_bytes => 960197124096, vendor => ATA}, sdb => {model => SSDSC2KB960G8R, size => 894.25 GiB, size_bytes => 960197124096, vendor => ATA}}, dmi => {bios => {release_date => 06/14/2019, vendor => Dell Inc., version => 2.2.11}, board => {manufacturer => Dell Inc., product => 08CYF7, serial_number => .9WYK4Z2.CNFCP0097U01F5.}, chassis => {type => Rack Mount Chassis}, manufacturer => Dell Inc., product => {name => PowerEdge R440, serial_number => 9WYK4Z2, uuid => 4C4C4544-0057-5910-804B-B9C04F345A32}}, domain => eqiad.wmnet, facterversion => 3.11.0, filesystems => ext2,ext3,ext4, fips_enabled => false, fqdn => mw1363.eqiad.wmnet, gid => root, hardwareisa => unknown, hardwaremodel => x86_64, has_ipmi => true, hostname => mw1363, id => root, identity => {gid => 0, group => root, privileged => true, uid => 0, user => root}, initsystem => systemd, interface_primary => eno1, interfaces => eno1,eno2,lo,lo:LVS, ipaddress => 10.64.48.205, ipaddress6 => 2620:0:861:107:10:64:48:205, ipaddress6_eno1 => 2620:0:861:107:10:64:48:205, ipaddress6_lo => :1, ipaddress_eno1 => 10.64.48.205, ipaddress_lo => 127.0.0.1, ipaddress_lo:LVS => 10.2.2.22, ipmi_chassis => {boot_flags => {console_redirection => BIOS-SETTING, device => NO-OVERRIDE, persistent => }, front_panel => {power_button_enabled => , reset_button_enabled => , standby_button_enabled => }, power => {restore_policy => Restore_State_AC_Apply}}, ipmi_lan => {gateway => 10.65.0.1, ipaddress => 10.65.2.42, macaddress => 4c:d9:8f:89:0b:82, netmask => 255.255.0.0}, is_pe => false, is_virtual => false, kernel => Linux, kernel_details => {release => 4.9.0-12-amd64, sysctl_settings => {net.ipv4.tcp_min_snd_mss => true}, version => #1 SMP Debian 4.9.210-1 (2020-01-20)}, kernelmajversion => 4.9, kernelrelease => 4.9.0-12-amd64, kernelversion => 4.9.0, lldp => {eno1 => {neighbor => asw2-d-eqiad, port => ge-3/0/40, vlan => vlan-1020}}, lldp_neighbors => [asw2-d-eqiad], lldp_parent => asw2-d-eqiad, load_averages => {15m => 10.81, 1m => 12.33, 5m => 11.27}, lsbdistcodename => stretch, lsbdistdescription => Debian GNU/Linux 9.12 (stretch), lsbdistid => Debian, lsbdistrelease => 9.12, lsbmajdistrelease => 9, lsbminordistrelease => 12, lvm_pv_0 => /dev/md0, lvm_pvs => 1, lvm_support => true, lvm_vg_0 => mw1363-vg, lvm_vg_mw1363-vg_pvs => /dev/md0, lvm_vgs => 1, macaddress => 4c:d9:8f:89:0b:88, macaddress_eno1 => 4c:d9:8f:89:0b:88, macaddress_eno2 => 4c:d9:8f:89:0b:89, manufacturer => Dell Inc., memory => {system => {available => 50.30 GiB, available_bytes => 54007877632, capacity => 19.34%, total => 62.36 GiB, total_bytes => 66960048128, used => 12.06 GiB, used_bytes => 12952170496}}, memoryfree => 50.30 GiB, memoryfree_mb => 51505.92578125, memorysize => 62.36 GiB, memorysize_mb => 63858.078125, mountpoints => {/ => {available => 501.91 GiB, available_bytes => 538919391232, capacity => 6.86%, device => /dev/mapper/mw1363--vg-root, filesystem => ext4, options => [rw, relatime, errors=remount-ro, data=ordered], size => 538.86 GiB, size_bytes => 578596683776, used => 36.95 GiB, used_bytes => 39677292544}, /dev/shm => {available => 31.18 GiB, available_bytes => 33480019968, capacity => 0.00%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, nodev], size => 31.18 GiB, size_bytes => 33480024064, used => 4.00 KiB, used_bytes => 4096}, /run => {available => 5.59 GiB, available_bytes => 6006710272, capacity => 10.29%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, noexec, relatime, size=6539068k, mode=755], size => 6.24 GiB, size_bytes => 6696005632, used => 657.36 MiB, used_bytes => 689295360}, /run/lock => {available => 5.00 MiB, available_bytes => 5242880, capacity => 0%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, nodev, noexec, relatime, size=5120k], size => 5.00 MiB, size_bytes => 5242880, used => 0 bytes, used_bytes => 0}, /srv => {available => 162.73 GiB, available_bytes => 174732419072, capacity => 10.67%, device => /dev/mapper/mw1363--vg-srv, filesystem => ext4, options => [rw, relatime, data=ordered], size => 182.16 GiB, size_bytes => 195593924608, used => 19.43 GiB, used_bytes => 20861505536}, /sys/fs/cgroup => {available => 31.18 GiB, available_bytes => 33480024064, capacity => 0%, device => tmpfs, filesystem => tmpfs, options => [ro, nosuid, nodev, noexec, mode=755], size => 31.18 GiB, size_bytes => 33480024064, used => 0 bytes, used_bytes => 0}, /var/lib/nginx => {available => 1023.93 MiB, available_bytes => 1073672192, capacity => 0.01%, device => tmpfs, filesystem => tmpfs, options => [rw, noatime, size=1048576k, mode=755], size => 1.00 GiB, size_bytes => 1073741824, used => 68.00 KiB, used_bytes => 69632}}, mtu_eno1 => 1500, mtu_eno2 => 1500, mtu_lo => 65536, net_driver => {eno1 => {driver => tg3, duplex => full, firmware_version => FFV21.40.9 bc 5720-v1.39, speed => 1000}, eno2 => {driver => tg3, duplex => unknown, firmware_version => FFV21.40.9 bc 5720-v1.39, speed => -1}}, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, netmask6_eno1 => ffff:ffff:ffff:ffff::, netmask6_lo => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, netmask_eno1 => 255.255.252.0, netmask_lo => 255.0.0.0, netmask_lo:LVS => 255.255.255.255, network => 10.64.48.0, network6 => 2620:0:861:107::, network6_eno1 => 2620:0:861:107::, network6_lo => :1, network_eno1 => 10.64.48.0, network_lo => 127.0.0.0, network_lo:LVS => 10.2.2.22, networking => {domain => eqiad.wmnet, fqdn => mw1363.eqiad.wmnet, hostname => mw1363, interfaces => {eno1 => {bindings => [{address => 10.64.48.205, netmask => 255.255.252.0, network => 10.64.48.0}], bindings6 => [{address => 2620:0:861:107:10:64:48:205, netmask => ffff:ffff:ffff:ffff::, network => 2620:0:861:107::}, {address => fe80::4ed9:8fff:fe89:b88, netmask => ffff:ffff:ffff:ffff::, network => fe80::}], ip => 10.64.48.205, ip6 => 2620:0:861:107:10:64:48:205, mac => 4c:d9:8f:89:0b:88, mtu => 1500, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, network => 10.64.48.0, network6 => 2620:0:861:107::}, eno2 => {mac => 4c:d9:8f:89:0b:89, mtu => 1500}, lo => {bindings => [{address => 127.0.0.1, netmask => 255.0.0.0, network => 127.0.0.0}], bindings6 => [{address => ::1, netmask => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, network => ::1}], ip => 127.0.0.1, ip6 => :1, mtu => 65536, netmask => 255.0.0.0, netmask6 => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, network => 127.0.0.0, network6 => :1}, lo:LVS => {bindings => [{address => 10.2.2.22, netmask => 255.255.255.255, network => 10.2.2.22}], ip => 10.2.2.22, netmask => 255.255.255.255, network => 10.2.2.22}}, ip => 10.64.48.205, ip6 => 2620:0:861:107:10:64:48:205, mac => 4c:d9:8f:89:0b:88, mtu => 1500, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, network => 10.64.48.0, network6 => 2620:0:861:107::, primary => eno1}, numa => {device_to_htset => {eno1 => [[0, 20], [2, 22], [4, 24], [6, 26], [8, 28], [10, 30], [12, 32], [14, 34], [16, 36], [18, 38]], eno2 => [[0, 20], [2, 22], [4, 24], [6, 26], [8, 28], [10, 30], [12, 32], [14, 34], [16, 36], [18, 38]], lo => [[0, 20], [1, 21], [2, 22], [3, 23], [4, 24], [5, 25], [6, 26], [7, 27], [8, 28], [9, 29], [10, 30], [11, 31], [12, 32], [13, 33], [14, 34], [15, 35], [16, 36], [17, 37], [18, 38], [19, 39]]}, device_to_node => {eno1 => [0], eno2 => [0], lo => [0, 1]}, nodes => [0, 1]}, operatingsystem => Debian, operatingsystemmajrelease => 9, operatingsystemrelease => 9.12, os => {architecture => amd64, distro => {codename => stretch, description => Debian GNU/Linux 9.12 (stretch), id => Debian, release => {full => 9.12, major => 9, minor => 12}}, family => Debian, hardware => x86_64, name => Debian, release => {full => 9.12, major => 9, minor => 12}, selinux => {enabled => false}}, osfamily => Debian, package_provider => apt, partitions => {/dev/mapper/mw1363--vg-placeholder => {size => 159.57 GiB, size_bytes => 171341512704}, /dev/mapper/mw1363--vg-root => {filesystem => ext4, mount => /, size => 548.47 GiB, size_bytes => 588913836032, uuid => 772dd9f2-664c-4ac6-9b81-31e6a6e1eb20}, /dev/mapper/mw1363--vg-srv => {filesystem => ext4, mount => /srv, size => 186.08 GiB, size_bytes => 199804059648, uuid => bb262acb-fe78-47ef-829b-430a4c74082d}, /dev/sda1 => {filesystem => linux_raid_member, label => mw1363:0, partuuid => 41ba7b92-01, size => 894.25 GiB, size_bytes => 960195723264, uuid => 1036e212-5b51-7d43-bf9f-d39b674279cc}, /dev/sdb1 => {filesystem => linux_raid_member, label => mw1363:0, partuuid => 1da3f988-01, size => 894.25 GiB, size_bytes => 960195723264, uuid => 1036e212-5b51-7d43-bf9f-d39b674279cc}}, path => /usr/bin:/bin:/usr/sbin:/sbin, physicalcorecount => 20, physicalprocessorcount => 2, processor0 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor1 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor10 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor11 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor12 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor13 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor14 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor15 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor16 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor17 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor18 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor19 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor2 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor20 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor21 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor22 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor23 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor24 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor25 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor26 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor27 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor28 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor29 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor3 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor30 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor31 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor32 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor33 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor34 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor35 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor36 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor37 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor38 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor39 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor4 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor5 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor6 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor7 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor8 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor9 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processorcount => 40, processors => {count => 40, isa => unknown, models => [Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz], physicalcount => 2, speed => 3.00 GHz}, productname => PowerEdge R440, puppet_config => {hostcert => /var/lib/puppet/ssl/certs/mw1363.eqiad.wmnet.pem, hostprivkey => /var/lib/puppet/ssl/private_keys/mw1363.eqiad.wmnet.pem, hostpubkey => /var/lib/puppet/ssl/public_keys/mw1363.eqiad.wmnet.pem, localcacert => /var/lib/puppet/ssl/certs/ca.pem, master => {localcacert => /var/lib/puppet/ssl/certs/ca.pem, ssldir => /var/lib/puppet/ssl}, ssldir => /var/lib/puppet/ssl}, puppet_config_dir => /etc/puppet, puppet_environmentpath => /etc/puppet/code/environments, puppet_server => puppet, puppet_vardir => /var/lib/puppet, puppetversion => 5.5.10, raid => [md], root_home => /root, ruby => {platform => x86_64-linux-gnu, sitedir => /usr/local/lib/site_ruby/2.3.0, version => 2.3.3}, rubyplatform => x86_64-linux-gnu, rubysitedir => /usr/local/lib/site_ruby/2.3.0, rubyversion => 2.3.3, selinux => false, serialnumber => 42424242, service_provider => systemd, ssh => {ecdsa => {fingerprints => {sha1 => SSHFP 3 1 a255543df7ba8104776a436e026da5898109e007, sha256 => SSHFP 3 2 ae0740d6e2e874006dfc8e978a6d432f5cdb242df7b3d63d68b36372c6629586}, key => AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLR2M0a/PGMQt9p0Kts681DcYi2pLq7if8Damrx+Sqk/MZhTAt3e+jVaEU2YDvCnfByyOWC0zriv+vMlIpBheE=, type => ecdsa-sha2-nistp256}, ed25519 => {fingerprints => {sha1 => SSHFP 4 1 0cfeeca9fb048d72289f4ca625be78551dc0c93c, sha256 => SSHFP 4 2 58e8b53761000b1f4e9969784d639da4c2335bcedb27451baf824f59c02420fc}, key => AAAAC3NzaC1lZDI1NTE5AAAAIG8NE30ToNK8YDnaBYJNJTI1E/XgqpdrCQCt2LzPVIVT, type => ssh-ed25519}, rsa => {fingerprints => {sha1 => SSHFP 1 1 f726aaf6e0a29d89e4b402842b1a522526f324a5, sha256 => SSHFP 1 2 ae9a85bb60083e74f9640fcf9cd04cb3d52f36af5ed88d99a0e04e022e95c27c}, key => AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Kr2MfD8BZrUQO3DP3MnSW8imbtSR7d6dRXsOxwhuNAdfwCcJSdiZYdpa+B7aXqWLcIA2nubQKJLEK2cqVSBk4cLZdfEFyejZdOtPQXtPN3wSP44LDwbMRZanE/8FDMX/BjqauHqMQYFmLBzPWRGU4xeA8Wq7YFd6A1fBd6CxwNxo3DRMKLoQbtyLCPCaNjOE3vwo23EiFap4zQ7oRzCxFe2hXPJfPWNnBAkY+/2Syv/A5TFXNLAIq6E1RXsidLzA562kI5yUk8LZ29a2Zzm5wWsGIZF7+TsJVZlDSfeDbOlE7KNPe8fVAtjbLO19u71K9M1oJ5eqpdZSb603OoZ9, type => ssh-rsa}}, sshecdsakey => AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLR2M0a/PGMQt9p0Kts681DcYi2pLq7if8Damrx+Sqk/MZhTAt3e+jVaEU2YDvCnfByyOWC0zriv+vMlIpBheE=, sshed25519key => AAAAC3NzaC1lZDI1NTE5AAAAIG8NE30ToNK8YDnaBYJNJTI1E/XgqpdrCQCt2LzPVIVT, sshfp_ecdsa => SSHFP 3 1 a255543df7ba8104776a436e026da5898109e007 SSHFP 3 2 ae0740d6e2e874006dfc8e978a6d432f5cdb242df7b3d63d68b36372c6629586, sshfp_ed25519 => SSHFP 4 1 0cfeeca9fb048d72289f4ca625be78551dc0c93c SSHFP 4 2 58e8b53761000b1f4e9969784d639da4c2335bcedb27451baf824f59c02420fc, sshfp_rsa => SSHFP 1 1 f726aaf6e0a29d89e4b402842b1a522526f324a5 SSHFP 1 2 ae9a85bb60083e74f9640fcf9cd04cb3d52f36af5ed88d99a0e04e022e95c27c, sshrsakey => AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Kr2MfD8BZrUQO3DP3MnSW8imbtSR7d6dRXsOxwhuNAdfwCcJSdiZYdpa+B7aXqWLcIA2nubQKJLEK2cqVSBk4cLZdfEFyejZdOtPQXtPN3wSP44LDwbMRZanE/8FDMX/BjqauHqMQYFmLBzPWRGU4xeA8Wq7YFd6A1fBd6CxwNxo3DRMKLoQbtyLCPCaNjOE3vwo23EiFap4zQ7oRzCxFe2hXPJfPWNnBAkY+/2Syv/A5TFXNLAIq6E1RXsidLzA562kI5yUk8LZ29a2Zzm5wWsGIZF7+TsJVZlDSfeDbOlE7KNPe8fVAtjbLO19u71K9M1oJ5eqpdZSb603OoZ9, system_uptime => {days => 41, hours => 1003, seconds => 3611565, uptime => 41 days}, timezone => UTC, uniqueid => 43434343, uptime => 41 days, uptime_days => 41, uptime_hours => 1003, uptime_seconds => 3611565, uuid => 4C4C4544-0057-5910-804B-B9C04F345A32, virtual => physical} Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` This issue is preventing new nodes from populating the puppetdb. One work around i have noticed is that i can compile the hosts an empty manifest i.e. `--manifests=/dev/null` and this causes enough meta data to get populated into puppetdb that further compilations using the full production catalouge work with out issue Initial run ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=$confdir/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` manifest=/dev/null ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/dev/null --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Compiled catalog for mw1363.eqiad.wmnet in environment production in 0.04 seconds { ... Empty catalouge ... } ``` another standard run ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=$confdir/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet { ... Full catalouge ... } ``` T150456 seems like it may be simlar, i have been using `jenkins-deploy@compiler1003:/mnt/home/jenkins-deploy/test.sh ` for testing

It seems that the puppetdb-populate script fails for hosts that don't already have data in the puppet instance of the compiler server. From testing it seems that the puppet master --compile command no longer reads the facts cache if you are using `storeconfigs_backend=puppetdb`. withthe current policuy we see this with errors like the following ``` Compiling catalog for mw1363.eqiad.wmnet [ 2020-03-26T16:35:01 ] ERROR: Compilation failed for host mw1363.eqiad.wmnet: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdbf7e2GJ/1/production/private/modules:/tmp/fill-puppetdbf7e2GJ/1/production/src/modules', '--confdir=/tmp/fill-puppetdbf7e2GJ/1/production/src', '--compile=mw1363.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30 Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/puppet_compiler/puppet.py", line 83, in compile_storeconfigs subprocess.check_call(cmd, stdout=out, stderr=err, env=env) File "/usr/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdbf7e2GJ/1/production/private/modules:/tmp/fill-puppetdbf7e2GJ/1/production/src/modules', '--confdir=/tmp/fill-puppetdbf7e2GJ/1/production/src', '--compile=mw1363.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30 Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdbf7e2GJ/1/production/src/manifests/realm.pp, line: 30, column: 4) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` We can see that this server is not allready in the database with the following ``` $ curl -X GET 'http://localhost:8080/pdb/query/v4/nodes' | jq .[].certname |grep mw1363.eqiad.wmnet: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 198k 0 198k 0 0 3201k 0 --:--:-- --:--:-- --:--:-- 3201k ``` to test things further i created a manifest with the following ``` lang=puppet notice($facts) fail('bomb out') ``` If i run this with `--storeconfigs --storeconfigs_backend=puppetdb` we get the following output ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/mnt/home/jenkins-deploy/puppet_base/1/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Scope(Class[main]): {} Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` however if i run it without the config we see the facts are loaded ``` lines=10 $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/mnt/home/jenkins-deploy/puppet_base/1/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Scope(Class[main]): {agent_specified_environment => production, architecture => amd64, augeas => {version => 1.8.0}, augeasversion => 1.8.0, bios_release_date => 06/14/2019, bios_vendor => Dell Inc., bios_version => 2.2.11, blockdevice_sda_model => SSDSC2KB960G8R, blockdevice_sda_size => 960197124096, blockdevice_sda_vendor => ATA, blockdevice_sdb_model => SSDSC2KB960G8R, blockdevice_sdb_size => 960197124096, blockdevice_sdb_vendor => ATA, blockdevices => sdb,sda, boardmanufacturer => Dell Inc., boardproductname => 424242, boardserialnumber => 4242, chassistype => Rack Mount Chassis, clientcert => mw1363.eqiad.wmnet, clientnoop => false, clientversion => 5.5.10, cpu_details => {cpus => {cpu0 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu1 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu10 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu11 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu12 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu13 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu14 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu15 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu16 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu17 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu18 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu19 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu2 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu20 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu21 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu22 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu23 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu24 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu25 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu26 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu27 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu28 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu29 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu3 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu30 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu31 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu32 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu33 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu34 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu35 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu36 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu37 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu38 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu39 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu4 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu5 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu6 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu7 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu8 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpu9 => {scaling_driver => intel_pstate, scaling_governor => powersave}, cpufreq => {scaling_driver => false, scaling_governor => false}, cpuidle => {scaling_driver => false, scaling_governor => false}}, scaling_driver => intel_pstate, scaling_governor => powersave, vulnerabilities => {itlb_multihit => KVM: Vulnerable, l1tf => Mitigation: PTE Inversion, mds => Mitigation: Clear CPU buffers; SMT vulnerable, meltdown => Mitigation: PTI, spec_store_bypass => Mitigation: Speculative Store Bypass disabled via prctl and seccomp, spectre_v1 => Mitigation: usercopy/swapgs barriers and __user pointer sanitization, spectre_v2 => Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling, tsx_async_abort => Mitigation: Clear CPU buffers; SMT vulnerable}}, cpu_scaling_driver => intel_pstate, cpu_scaling_governor => powersave, default_routes => {ipv4 => 10.64.48.1, ipv6 => fe80::1}, disks => {sda => {model => SSDSC2KB960G8R, size => 894.25 GiB, size_bytes => 960197124096, vendor => ATA}, sdb => {model => SSDSC2KB960G8R, size => 894.25 GiB, size_bytes => 960197124096, vendor => ATA}}, dmi => {bios => {release_date => 06/14/2019, vendor => Dell Inc., version => 2.2.11}, board => {manufacturer => Dell Inc., product => 08CYF7, serial_number => .9WYK4Z2.CNFCP0097U01F5.}, chassis => {type => Rack Mount Chassis}, manufacturer => Dell Inc., product => {name => PowerEdge R440, serial_number => 9WYK4Z2, uuid => 4C4C4544-0057-5910-804B-B9C04F345A32}}, domain => eqiad.wmnet, facterversion => 3.11.0, filesystems => ext2,ext3,ext4, fips_enabled => false, fqdn => mw1363.eqiad.wmnet, gid => root, hardwareisa => unknown, hardwaremodel => x86_64, has_ipmi => true, hostname => mw1363, id => root, identity => {gid => 0, group => root, privileged => true, uid => 0, user => root}, initsystem => systemd, interface_primary => eno1, interfaces => eno1,eno2,lo,lo:LVS, ipaddress => 10.64.48.205, ipaddress6 => 2620:0:861:107:10:64:48:205, ipaddress6_eno1 => 2620:0:861:107:10:64:48:205, ipaddress6_lo => :1, ipaddress_eno1 => 10.64.48.205, ipaddress_lo => 127.0.0.1, ipaddress_lo:LVS => 10.2.2.22, ipmi_chassis => {boot_flags => {console_redirection => BIOS-SETTING, device => NO-OVERRIDE, persistent => }, front_panel => {power_button_enabled => , reset_button_enabled => , standby_button_enabled => }, power => {restore_policy => Restore_State_AC_Apply}}, ipmi_lan => {gateway => 10.65.0.1, ipaddress => 10.65.2.42, macaddress => 4c:d9:8f:89:0b:82, netmask => 255.255.0.0}, is_pe => false, is_virtual => false, kernel => Linux, kernel_details => {release => 4.9.0-12-amd64, sysctl_settings => {net.ipv4.tcp_min_snd_mss => true}, version => #1 SMP Debian 4.9.210-1 (2020-01-20)}, kernelmajversion => 4.9, kernelrelease => 4.9.0-12-amd64, kernelversion => 4.9.0, lldp => {eno1 => {neighbor => asw2-d-eqiad, port => ge-3/0/40, vlan => vlan-1020}}, lldp_neighbors => [asw2-d-eqiad], lldp_parent => asw2-d-eqiad, load_averages => {15m => 10.81, 1m => 12.33, 5m => 11.27}, lsbdistcodename => stretch, lsbdistdescription => Debian GNU/Linux 9.12 (stretch), lsbdistid => Debian, lsbdistrelease => 9.12, lsbmajdistrelease => 9, lsbminordistrelease => 12, lvm_pv_0 => /dev/md0, lvm_pvs => 1, lvm_support => true, lvm_vg_0 => mw1363-vg, lvm_vg_mw1363-vg_pvs => /dev/md0, lvm_vgs => 1, macaddress => 4c:d9:8f:89:0b:88, macaddress_eno1 => 4c:d9:8f:89:0b:88, macaddress_eno2 => 4c:d9:8f:89:0b:89, manufacturer => Dell Inc., memory => {system => {available => 50.30 GiB, available_bytes => 54007877632, capacity => 19.34%, total => 62.36 GiB, total_bytes => 66960048128, used => 12.06 GiB, used_bytes => 12952170496}}, memoryfree => 50.30 GiB, memoryfree_mb => 51505.92578125, memorysize => 62.36 GiB, memorysize_mb => 63858.078125, mountpoints => {/ => {available => 501.91 GiB, available_bytes => 538919391232, capacity => 6.86%, device => /dev/mapper/mw1363--vg-root, filesystem => ext4, options => [rw, relatime, errors=remount-ro, data=ordered], size => 538.86 GiB, size_bytes => 578596683776, used => 36.95 GiB, used_bytes => 39677292544}, /dev/shm => {available => 31.18 GiB, available_bytes => 33480019968, capacity => 0.00%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, nodev], size => 31.18 GiB, size_bytes => 33480024064, used => 4.00 KiB, used_bytes => 4096}, /run => {available => 5.59 GiB, available_bytes => 6006710272, capacity => 10.29%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, noexec, relatime, size=6539068k, mode=755], size => 6.24 GiB, size_bytes => 6696005632, used => 657.36 MiB, used_bytes => 689295360}, /run/lock => {available => 5.00 MiB, available_bytes => 5242880, capacity => 0%, device => tmpfs, filesystem => tmpfs, options => [rw, nosuid, nodev, noexec, relatime, size=5120k], size => 5.00 MiB, size_bytes => 5242880, used => 0 bytes, used_bytes => 0}, /srv => {available => 162.73 GiB, available_bytes => 174732419072, capacity => 10.67%, device => /dev/mapper/mw1363--vg-srv, filesystem => ext4, options => [rw, relatime, data=ordered], size => 182.16 GiB, size_bytes => 195593924608, used => 19.43 GiB, used_bytes => 20861505536}, /sys/fs/cgroup => {available => 31.18 GiB, available_bytes => 33480024064, capacity => 0%, device => tmpfs, filesystem => tmpfs, options => [ro, nosuid, nodev, noexec, mode=755], size => 31.18 GiB, size_bytes => 33480024064, used => 0 bytes, used_bytes => 0}, /var/lib/nginx => {available => 1023.93 MiB, available_bytes => 1073672192, capacity => 0.01%, device => tmpfs, filesystem => tmpfs, options => [rw, noatime, size=1048576k, mode=755], size => 1.00 GiB, size_bytes => 1073741824, used => 68.00 KiB, used_bytes => 69632}}, mtu_eno1 => 1500, mtu_eno2 => 1500, mtu_lo => 65536, net_driver => {eno1 => {driver => tg3, duplex => full, firmware_version => FFV21.40.9 bc 5720-v1.39, speed => 1000}, eno2 => {driver => tg3, duplex => unknown, firmware_version => FFV21.40.9 bc 5720-v1.39, speed => -1}}, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, netmask6_eno1 => ffff:ffff:ffff:ffff::, netmask6_lo => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, netmask_eno1 => 255.255.252.0, netmask_lo => 255.0.0.0, netmask_lo:LVS => 255.255.255.255, network => 10.64.48.0, network6 => 2620:0:861:107::, network6_eno1 => 2620:0:861:107::, network6_lo => :1, network_eno1 => 10.64.48.0, network_lo => 127.0.0.0, network_lo:LVS => 10.2.2.22, networking => {domain => eqiad.wmnet, fqdn => mw1363.eqiad.wmnet, hostname => mw1363, interfaces => {eno1 => {bindings => [{address => 10.64.48.205, netmask => 255.255.252.0, network => 10.64.48.0}], bindings6 => [{address => 2620:0:861:107:10:64:48:205, netmask => ffff:ffff:ffff:ffff::, network => 2620:0:861:107::}, {address => fe80::4ed9:8fff:fe89:b88, netmask => ffff:ffff:ffff:ffff::, network => fe80::}], ip => 10.64.48.205, ip6 => 2620:0:861:107:10:64:48:205, mac => 4c:d9:8f:89:0b:88, mtu => 1500, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, network => 10.64.48.0, network6 => 2620:0:861:107::}, eno2 => {mac => 4c:d9:8f:89:0b:89, mtu => 1500}, lo => {bindings => [{address => 127.0.0.1, netmask => 255.0.0.0, network => 127.0.0.0}], bindings6 => [{address => ::1, netmask => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, network => ::1}], ip => 127.0.0.1, ip6 => :1, mtu => 65536, netmask => 255.0.0.0, netmask6 => ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, network => 127.0.0.0, network6 => :1}, lo:LVS => {bindings => [{address => 10.2.2.22, netmask => 255.255.255.255, network => 10.2.2.22}], ip => 10.2.2.22, netmask => 255.255.255.255, network => 10.2.2.22}}, ip => 10.64.48.205, ip6 => 2620:0:861:107:10:64:48:205, mac => 4c:d9:8f:89:0b:88, mtu => 1500, netmask => 255.255.252.0, netmask6 => ffff:ffff:ffff:ffff::, network => 10.64.48.0, network6 => 2620:0:861:107::, primary => eno1}, numa => {device_to_htset => {eno1 => [[0, 20], [2, 22], [4, 24], [6, 26], [8, 28], [10, 30], [12, 32], [14, 34], [16, 36], [18, 38]], eno2 => [[0, 20], [2, 22], [4, 24], [6, 26], [8, 28], [10, 30], [12, 32], [14, 34], [16, 36], [18, 38]], lo => [[0, 20], [1, 21], [2, 22], [3, 23], [4, 24], [5, 25], [6, 26], [7, 27], [8, 28], [9, 29], [10, 30], [11, 31], [12, 32], [13, 33], [14, 34], [15, 35], [16, 36], [17, 37], [18, 38], [19, 39]]}, device_to_node => {eno1 => [0], eno2 => [0], lo => [0, 1]}, nodes => [0, 1]}, operatingsystem => Debian, operatingsystemmajrelease => 9, operatingsystemrelease => 9.12, os => {architecture => amd64, distro => {codename => stretch, description => Debian GNU/Linux 9.12 (stretch), id => Debian, release => {full => 9.12, major => 9, minor => 12}}, family => Debian, hardware => x86_64, name => Debian, release => {full => 9.12, major => 9, minor => 12}, selinux => {enabled => false}}, osfamily => Debian, package_provider => apt, partitions => {/dev/mapper/mw1363--vg-placeholder => {size => 159.57 GiB, size_bytes => 171341512704}, /dev/mapper/mw1363--vg-root => {filesystem => ext4, mount => /, size => 548.47 GiB, size_bytes => 588913836032, uuid => 772dd9f2-664c-4ac6-9b81-31e6a6e1eb20}, /dev/mapper/mw1363--vg-srv => {filesystem => ext4, mount => /srv, size => 186.08 GiB, size_bytes => 199804059648, uuid => bb262acb-fe78-47ef-829b-430a4c74082d}, /dev/sda1 => {filesystem => linux_raid_member, label => mw1363:0, partuuid => 41ba7b92-01, size => 894.25 GiB, size_bytes => 960195723264, uuid => 1036e212-5b51-7d43-bf9f-d39b674279cc}, /dev/sdb1 => {filesystem => linux_raid_member, label => mw1363:0, partuuid => 1da3f988-01, size => 894.25 GiB, size_bytes => 960195723264, uuid => 1036e212-5b51-7d43-bf9f-d39b674279cc}}, path => /usr/bin:/bin:/usr/sbin:/sbin, physicalcorecount => 20, physicalprocessorcount => 2, processor0 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor1 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor10 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor11 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor12 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor13 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor14 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor15 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor16 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor17 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor18 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor19 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor2 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor20 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor21 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor22 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor23 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor24 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor25 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor26 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor27 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor28 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor29 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor3 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor30 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor31 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor32 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor33 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor34 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor35 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor36 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor37 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor38 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor39 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor4 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor5 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor6 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor7 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor8 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processor9 => Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, processorcount => 40, processors => {count => 40, isa => unknown, models => [Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz, Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz], physicalcount => 2, speed => 3.00 GHz}, productname => PowerEdge R440, puppet_config => {hostcert => /var/lib/puppet/ssl/certs/mw1363.eqiad.wmnet.pem, hostprivkey => /var/lib/puppet/ssl/private_keys/mw1363.eqiad.wmnet.pem, hostpubkey => /var/lib/puppet/ssl/public_keys/mw1363.eqiad.wmnet.pem, localcacert => /var/lib/puppet/ssl/certs/ca.pem, master => {localcacert => /var/lib/puppet/ssl/certs/ca.pem, ssldir => /var/lib/puppet/ssl}, ssldir => /var/lib/puppet/ssl}, puppet_config_dir => /etc/puppet, puppet_environmentpath => /etc/puppet/code/environments, puppet_server => puppet, puppet_vardir => /var/lib/puppet, puppetversion => 5.5.10, raid => [md], root_home => /root, ruby => {platform => x86_64-linux-gnu, sitedir => /usr/local/lib/site_ruby/2.3.0, version => 2.3.3}, rubyplatform => x86_64-linux-gnu, rubysitedir => /usr/local/lib/site_ruby/2.3.0, rubyversion => 2.3.3, selinux => false, serialnumber => 42424242, service_provider => systemd, ssh => {ecdsa => {fingerprints => {sha1 => SSHFP 3 1 a255543df7ba8104776a436e026da5898109e007, sha256 => SSHFP 3 2 ae0740d6e2e874006dfc8e978a6d432f5cdb242df7b3d63d68b36372c6629586}, key => AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLR2M0a/PGMQt9p0Kts681DcYi2pLq7if8Damrx+Sqk/MZhTAt3e+jVaEU2YDvCnfByyOWC0zriv+vMlIpBheE=, type => ecdsa-sha2-nistp256}, ed25519 => {fingerprints => {sha1 => SSHFP 4 1 0cfeeca9fb048d72289f4ca625be78551dc0c93c, sha256 => SSHFP 4 2 58e8b53761000b1f4e9969784d639da4c2335bcedb27451baf824f59c02420fc}, key => AAAAC3NzaC1lZDI1NTE5AAAAIG8NE30ToNK8YDnaBYJNJTI1E/XgqpdrCQCt2LzPVIVT, type => ssh-ed25519}, rsa => {fingerprints => {sha1 => SSHFP 1 1 f726aaf6e0a29d89e4b402842b1a522526f324a5, sha256 => SSHFP 1 2 ae9a85bb60083e74f9640fcf9cd04cb3d52f36af5ed88d99a0e04e022e95c27c}, key => AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Kr2MfD8BZrUQO3DP3MnSW8imbtSR7d6dRXsOxwhuNAdfwCcJSdiZYdpa+B7aXqWLcIA2nubQKJLEK2cqVSBk4cLZdfEFyejZdOtPQXtPN3wSP44LDwbMRZanE/8FDMX/BjqauHqMQYFmLBzPWRGU4xeA8Wq7YFd6A1fBd6CxwNxo3DRMKLoQbtyLCPCaNjOE3vwo23EiFap4zQ7oRzCxFe2hXPJfPWNnBAkY+/2Syv/A5TFXNLAIq6E1RXsidLzA562kI5yUk8LZ29a2Zzm5wWsGIZF7+TsJVZlDSfeDbOlE7KNPe8fVAtjbLO19u71K9M1oJ5eqpdZSb603OoZ9, type => ssh-rsa}}, sshecdsakey => AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLR2M0a/PGMQt9p0Kts681DcYi2pLq7if8Damrx+Sqk/MZhTAt3e+jVaEU2YDvCnfByyOWC0zriv+vMlIpBheE=, sshed25519key => AAAAC3NzaC1lZDI1NTE5AAAAIG8NE30ToNK8YDnaBYJNJTI1E/XgqpdrCQCt2LzPVIVT, sshfp_ecdsa => SSHFP 3 1 a255543df7ba8104776a436e026da5898109e007 SSHFP 3 2 ae0740d6e2e874006dfc8e978a6d432f5cdb242df7b3d63d68b36372c6629586, sshfp_ed25519 => SSHFP 4 1 0cfeeca9fb048d72289f4ca625be78551dc0c93c SSHFP 4 2 58e8b53761000b1f4e9969784d639da4c2335bcedb27451baf824f59c02420fc, sshfp_rsa => SSHFP 1 1 f726aaf6e0a29d89e4b402842b1a522526f324a5 SSHFP 1 2 ae9a85bb60083e74f9640fcf9cd04cb3d52f36af5ed88d99a0e04e022e95c27c, sshrsakey => AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Kr2MfD8BZrUQO3DP3MnSW8imbtSR7d6dRXsOxwhuNAdfwCcJSdiZYdpa+B7aXqWLcIA2nubQKJLEK2cqVSBk4cLZdfEFyejZdOtPQXtPN3wSP44LDwbMRZanE/8FDMX/BjqauHqMQYFmLBzPWRGU4xeA8Wq7YFd6A1fBd6CxwNxo3DRMKLoQbtyLCPCaNjOE3vwo23EiFap4zQ7oRzCxFe2hXPJfPWNnBAkY+/2Syv/A5TFXNLAIq6E1RXsidLzA562kI5yUk8LZ29a2Zzm5wWsGIZF7+TsJVZlDSfeDbOlE7KNPe8fVAtjbLO19u71K9M1oJ5eqpdZSb603OoZ9, system_uptime => {days => 41, hours => 1003, seconds => 3611565, uptime => 41 days}, timezone => UTC, uniqueid => 43434343, uptime => 41 days, uptime_days => 41, uptime_hours => 1003, uptime_seconds => 3611565, uuid => 4C4C4544-0057-5910-804B-B9C04F345A32, virtual => physical} Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Error while evaluating a Function Call, bomb out (file: /mnt/home/jenkins-deploy/puppet_base/1/manifests/test.pp, line: 2, column: 1) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` This issue is preventing new nodes from populating the puppetdb. One work around i have noticed is that i can compile the hosts an empty manifest i.e. `--manifests=/dev/null` and this causes enough meta data to get populated into puppetdb that further compilations using the full production catalouge work with out issue Initial run ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=$confdir/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Warning: Failed to compile catalog for node mw1363.eqiad.wmnet: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /mnt/home/jenkins-deploy/puppet_base/1/production/src/manifests/realm.pp, line: 29, column: 4) on node mw1363.eqiad.wmnet Error: Failed to compile catalog for node mw1363.eqiad.wmnet: undefined method `to_resource' for nil:NilClass ``` manifest=/dev/null ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=/dev/null --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet Notice: Compiled catalog for mw1363.eqiad.wmnet in environment production in 0.04 seconds { ... Empty catalouge ... } ``` another standard run ``` $ ./test.sh mw1363.eqiad.wmnet + BASE=/mnt/home/jenkins-deploy/puppet_base/1 + RUBYLIB=/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules/wmflib/lib + export RUBYLIB + puppet master --vardir=/var/lib/catalog-differ/puppet --manifest=$confdir/manifests --modulepath=/mnt/home/jenkins-deploy/puppet_base/1/production/private/modules:/mnt/home/jenkins-deploy/puppet_base/1/production/src/modules --confdir=/mnt/home/jenkins-deploy/puppet_base/1/production/src --yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet --environmentpath=$confdir/environments --storeconfigs --storeconfigs_backend=puppetdb --node_name_value=mw1363.eqiad.wmnet --compile=mw1363.eqiad.wmnet { ... Full catalouge ... } ``` T150456 seems like it may be simlar, i have been using `jenkins-deploy@compiler1003:/mnt/home/jenkins-deploy/test.sh ` for testing