Event Timeline
When I try to apt-install manually from the host, I get the following failure:
`
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed
Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: libx11-6 but it is not going to be installed
wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).`
Kicked off another puppet agent run on the off chance it was a first-puppet-run-failure type thing. It looks pretty borked. Probably going to try the apt --fix-broken-install command, but going to read through the output below first and see if anything obvious jumps out at me.
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: elasticsearch-oss INFO:debmonitor:Got 1 updates from dpkg hook version 3 INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. Need to get 0 B/34.2 MB of archives. After this operation, 38.4 MB of additional disk space will be used. (Reading database ... 46070 files and directories currently installed.) Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ... could not find java; set JAVA_HOME or ensure java is in PATH dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack): subprocess new pre-installation script returned error exit status 1 Errors were encountered while processing: /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) Error: /Stage[main]/Elasticsearch::Packages/Package[elasticsearch]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: elasticsearch-oss INFO:debmonitor:Got 1 updates from dpkg hook version 3 INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. Need to get 0 B/34.2 MB of archives. After this operation, 38.4 MB of additional disk space will be used. (Reading database ... 46070 files and directories currently installed.) Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ... could not find java; set JAVA_HOME or ensure java is in PATH dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack): subprocess new pre-installation script returned error exit status 1 Errors were encountered while processing: /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29) Wrapped exception: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/logstash-gelf.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33) Wrapped exception: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/json-simple.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33) Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists... Building dependency tree... Reading state information... wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch). You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Error: /Stage[main]/Profile::Elasticsearch::Cirrus/Package[wmf-elasticsearch-search-plugins]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists... Building dependency tree... Reading state information... wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch). You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). (corrective) Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: libx11-6 but it is not going to be installed wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Error: /Stage[main]/Java/Java::Package[openjdk-jdk-8]/Package[openjdk-8-jdk]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: libx11-6 but it is not going to be installed wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Notice: /Stage[main]/Logstash/Package[logstash]: Dependency Package[openjdk-8-jdk] has failures: true Warning: /Stage[main]/Logstash/Package[logstash]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/usr/local/bin/logstash-config-test]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/default/logstash]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/jvm.options]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/log4j2.properties]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/logstash.yml]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/conf.d/README]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/init/logstash.conf]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/init/logstash-web.conf]: Skipping because of failed dependencies Warning: /Stage[main]/Java/Alternatives::Java[8]/Exec[update_java_alternatives_8]: Skipping because of failed dependencies Warning: /Stage[main]/Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]: Skipping because of failed dependencies Warning: /Stage[main]/Java/Java::Cacert[wmf:Wikimedia_Internal_Root_CA]/Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]: Skipping because of failed dependencies Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Error: /Stage[main]/Elasticsearch::Curator/Package[elasticsearch-curator]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Notice: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Dependency Package[elasticsearch] has failures: true
--fix-broken says it will only install the one package, suggesting the package is the problem:
ebernhardson@elastic2051:/var/log$ apt --fix-broken install --dry-run NOTE: This is only a simulation! apt needs root privileges for real execution. Keep also in mind that locking is deactivated, so don't depend on the relevance to the real current situation! Reading package lists... Done Building dependency tree Reading state information... Done Correcting dependencies... Done The following additional packages will be installed: elasticsearch-oss The following NEW packages will be installed: elasticsearch-oss 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. Inst elasticsearch-oss (6.5.4 Wikimedia:9/stretch-wikimedia [all]) Conf elasticsearch-oss (6.5.4 Wikimedia:9/stretch-wikimedia [all]) Conf wmf-elasticsearch-search-plugins (6.5.4-7~stretch Wikimedia:9/stretch-wikimedia [all])
Checking into what happened i grepped for elasticsearch-oss in /var/log and found, among other logs:
Log started: 2022-01-13 19:23:42 (Reading database ... 46070 files and directories currently installed.) Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ... could not find java; set JAVA_HOME or ensure java is in PATH dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack): subprocess new pre-installation script returned error exit status 1 Errors were encountered while processing: /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb Log ended: 2022-01-13 19:23:42
I'm unsure why it can't find java though.
elastic2051 doesn't have java installed:
java-common/oldoldstable 0.58+deb9u1 all
compared to elastic2054:
java-common/oldoldstable,now 0.58+deb9u1 all [installed,automatic]
ryankemper@elastic2054:~$ apt-cache rdepends --installed java-common java-common Reverse Depends: openjdk-8-jre-headless openjdk-8-jre-headless
versus
ryankemper@elastic2051:~$ apt-cache rdepends --installed java-common java-common Reverse Depends:
ryankemper@elastic2051:~$ sudo run-puppet-agent Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for elastic2051.codfw.wmnet Info: Applying configuration version '(0cdcafe1ab) joal - Fix error in network_internal_flows druid job' Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: elasticsearch-oss INFO:debmonitor:Got 1 updates from dpkg hook version 3 INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. Need to get 0 B/34.2 MB of archives. After this operation, 38.4 MB of additional disk space will be used. (Reading database ... 46070 files and directories currently installed.) Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ... could not find java; set JAVA_HOME or ensure java is in PATH dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack): subprocess new pre-installation script returned error exit status 1 Errors were encountered while processing: /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) Error: /Stage[main]/Elasticsearch::Packages/Package[elasticsearch]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: elasticsearch-oss INFO:debmonitor:Got 1 updates from dpkg hook version 3 INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. Need to get 0 B/34.2 MB of archives. After this operation, 38.4 MB of additional disk space will be used. (Reading database ... 46070 files and directories currently installed.) Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ... could not find java; set JAVA_HOME or ensure java is in PATH dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack): subprocess new pre-installation script returned error exit status 1 Errors were encountered while processing: /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29) Wrapped exception: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/logstash-gelf.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33) Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33) Wrapped exception: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/json-simple.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33) Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists... Building dependency tree... Reading state information... wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch). You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Error: /Stage[main]/Profile::Elasticsearch::Cirrus/Package[wmf-elasticsearch-search-plugins]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists... Building dependency tree... Reading state information... wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch). You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). (corrective) Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: libx11-6 but it is not going to be installed wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Error: /Stage[main]/Java/Java::Package[openjdk-jdk-8]/Package[openjdk-8-jdk]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed Depends: libx11-6 but it is not going to be installed wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Notice: /Stage[main]/Logstash/Package[logstash]: Dependency Package[openjdk-8-jdk] has failures: true Warning: /Stage[main]/Logstash/Package[logstash]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/usr/local/bin/logstash-config-test]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/default/logstash]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/jvm.options]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/log4j2.properties]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/logstash.yml]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/logstash/conf.d/README]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/init/logstash.conf]: Skipping because of failed dependencies Warning: /Stage[main]/Logstash/File[/etc/init/logstash-web.conf]: Skipping because of failed dependencies Warning: /Stage[main]/Java/Alternatives::Java[8]/Exec[update_java_alternatives_8]: Skipping because of failed dependencies Warning: /Stage[main]/Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]: Skipping because of failed dependencies Warning: /Stage[main]/Java/Java::Cacert[wmf:Wikimedia_Internal_Root_CA]/Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]: Skipping because of failed dependencies Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Error: /Stage[main]/Elasticsearch::Curator/Package[elasticsearch-curator]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Notice: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Dependency Package[elasticsearch] has failures: true Warning: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/File[/srv/elasticsearch]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch.log]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_search_slowlog.log]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Service[elasticsearch]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/elasticsearch.yml]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/log4j2.properties]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/jvm.options]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/scripts]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Exec[elasticsearch-create-keystore-production-search-codfw]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/elasticsearch.keystore]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/srv/elasticsearch/production-search-codfw]: Skipping because of failed dependencies Notice: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Service[elasticsearch_6@production-search-codfw]: Dependency Package[wmf-elasticsearch-search-plugins] has failures: true Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Service[elasticsearch_6@production-search-codfw]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/elasticsearch.yml]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/log4j2.properties]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/jvm.options]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/scripts]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/Exec[elasticsearch-create-keystore-production-search-omega-codfw]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/elasticsearch.keystore]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/srv/elasticsearch/production-search-omega-codfw]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/Service[elasticsearch_6@production-search-omega-codfw]: Skipping because of failed dependencies Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-elasticsearch-exporter' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: prometheus-elasticsearch-exporter : Depends: daemon but it is not going to be installed wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Error: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Package[prometheus-elasticsearch-exporter]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-elasticsearch-exporter' returned 100: Reading package lists... Building dependency tree... Reading state information... You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: prometheus-elasticsearch-exporter : Depends: daemon but it is not going to be installed wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or elasticsearch (= 6.5.4) but it is not installable E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). Notice: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Service[prometheus-elasticsearch-exporter]: Dependency Package[prometheus-elasticsearch-exporter] has failures: true Warning: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Service[prometheus-elasticsearch-exporter]: Skipping because of failed dependencies Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33) Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33) Wrapped exception: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link Error: /Stage[main]/Profile::Logstash::Gelf_relay/Logstash::Input::Gelf[gelf_relay]/Logstash::Conf[input-gelf-gelf_relay]/File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33) Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33) Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33) Wrapped exception: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link Error: /Stage[main]/Profile::Logstash::Gelf_relay/Logstash::Output::Udp[gelf_relay]/Logstash::Conf[output-udp-gelf_relay]/File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33) Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf] has failures: true Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf] has failures: true Warning: /Stage[main]/Logstash/Service[logstash]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.service]/File[/lib/systemd/system/elasticsearch-hot-threads-log.service]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.service]/Exec[systemd daemon-reload for elasticsearch-hot-threads-log.service]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Systemd::Unit[prometheus-elasticsearch-exporter-9200]/File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Systemd::Unit[prometheus-elasticsearch-exporter-9200]/Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Service[prometheus-elasticsearch-exporter-9200]: Skipping because of failed dependencies Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]/ensure: ensure changed 'stopped' to 'running' (corrective) Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9200] Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Systemd::Unit[prometheus-elasticsearch-exporter-9400]/File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Systemd::Unit[prometheus-elasticsearch-exporter-9400]/Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Service[prometheus-elasticsearch-exporter-9400]: Skipping because of failed dependencies Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]/ensure: ensure changed 'stopped' to 'running' (corrective) Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9400] Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.timer]/File[/lib/systemd/system/elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.timer]/Exec[systemd daemon-reload for elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Service[elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies Info: Class[Profile::Elasticsearch::Cirrus]: Unscheduling all events on Class[Profile::Elasticsearch::Cirrus] Notice: Applied catalog in 22.24 seconds