Page MenuHomePhabricator
Paste P18732

elastic2051.codfw.wmnet reimage failure
ActivePublic

Authored by bking on Thu, Jan 13, 3:40 PM.
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for elastic2051.codfw.wmnet
Info: Applying configuration version '(624d5bf509) Ottomata - Add network_internal_flows to refine and druid-load'
Notice: /Stage[main]/Logstash/Package[logstash]: Dependency Package[openjdk-8-jdk] has failures: true
Notice: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Dependency Package[elasticsearch] has failures: true
Notice: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Service[elasticsearch_6@production-search-codfw]: Dependency Package[wmf-elasticsearch-search-plugins] has failures: true
Notice: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Service[prometheus-elasticsearch-exporter]: Dependency Package[prometheus-elasticsearch-exporter] has failures: true
Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf] has failures: true
Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf] has failures: true
Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9200]
Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9400]
Info: Class[Profile::Elasticsearch::Cirrus]: Unscheduling all events on Class[Profile::Elasticsearch::Cirrus]
Notice: Applied catalog in 22.07 seconds

Event Timeline

When I try to apt-install manually from the host, I get the following failure:
`
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed

Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed
Depends: libx11-6 but it is not going to be installed

wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or

elasticsearch (= 6.5.4) but it is not installable

E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).`

Kicked off another puppet agent run on the off chance it was a first-puppet-run-failure type thing. It looks pretty borked. Probably going to try the apt --fix-broken-install command, but going to read through the output below first and see if anything obvious jumps out at me.

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  elasticsearch-oss
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/34.2 MB of archives.
After this operation, 38.4 MB of additional disk space will be used.
(Reading database ... 46070 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ...
could not find java; set JAVA_HOME or ensure java is in PATH
dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error: /Stage[main]/Elasticsearch::Packages/Package[elasticsearch]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  elasticsearch-oss
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/34.2 MB of archives.
After this operation, 38.4 MB of additional disk space will be used.
(Reading database ... 46070 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ...
could not find java; set JAVA_HOME or ensure java is in PATH
dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Wrapped exception:
No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib
Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/logstash-gelf.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Wrapped exception:
No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib
Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/json-simple.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Profile::Elasticsearch::Cirrus/Package[wmf-elasticsearch-search-plugins]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). (corrective)
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: libx11-6 but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Java/Java::Package[openjdk-jdk-8]/Package[openjdk-8-jdk]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: libx11-6 but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Logstash/Package[logstash]: Dependency Package[openjdk-8-jdk] has failures: true
Warning: /Stage[main]/Logstash/Package[logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/usr/local/bin/logstash-config-test]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/default/logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/jvm.options]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/log4j2.properties]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/logstash.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/conf.d/README]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/init/logstash.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/init/logstash-web.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Alternatives::Java[8]/Exec[update_java_alternatives_8]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Java::Cacert[wmf:Wikimedia_Internal_Root_CA]/Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Elasticsearch::Curator/Package[elasticsearch-curator]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Dependency Package[elasticsearch] has failures: true

--fix-broken says it will only install the one package, suggesting the package is the problem:

ebernhardson@elastic2051:/var/log$ apt --fix-broken install --dry-run
NOTE: This is only a simulation!
      apt needs root privileges for real execution.
      Keep also in mind that locking is deactivated,
      so don't depend on the relevance to the real current situation!
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  elasticsearch-oss
The following NEW packages will be installed:
  elasticsearch-oss
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Inst elasticsearch-oss (6.5.4 Wikimedia:9/stretch-wikimedia [all])
Conf elasticsearch-oss (6.5.4 Wikimedia:9/stretch-wikimedia [all])
Conf wmf-elasticsearch-search-plugins (6.5.4-7~stretch Wikimedia:9/stretch-wikimedia [all])

Checking into what happened i grepped for elasticsearch-oss in /var/log and found, among other logs:

Log started: 2022-01-13  19:23:42
(Reading database ... 46070 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ...
could not find java; set JAVA_HOME or ensure java is in PATH
dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb
Log ended: 2022-01-13  19:23:42

I'm unsure why it can't find java though.

elastic2051 doesn't have java installed:
java-common/oldoldstable 0.58+deb9u1 all

compared to elastic2054:
java-common/oldoldstable,now 0.58+deb9u1 all [installed,automatic]

ryankemper@elastic2054:~$ apt-cache rdepends --installed java-common
java-common
Reverse Depends:
  openjdk-8-jre-headless
  openjdk-8-jre-headless

versus

ryankemper@elastic2051:~$ apt-cache rdepends --installed java-common

java-common
Reverse Depends:
ryankemper@elastic2051:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for elastic2051.codfw.wmnet
Info: Applying configuration version '(0cdcafe1ab) joal - Fix error in network_internal_flows druid job'
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  elasticsearch-oss
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/34.2 MB of archives.
After this operation, 38.4 MB of additional disk space will be used.
(Reading database ... 46070 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ...
could not find java; set JAVA_HOME or ensure java is in PATH
dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error: /Stage[main]/Elasticsearch::Packages/Package[elasticsearch]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  elasticsearch-oss
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/34.2 MB of archives.
After this operation, 38.4 MB of additional disk space will be used.
(Reading database ... 46070 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ...
could not find java; set JAVA_HOME or ensure java is in PATH
dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Wrapped exception:
No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib
Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/logstash-gelf.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Wrapped exception:
No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib
Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/json-simple.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Profile::Elasticsearch::Cirrus/Package[wmf-elasticsearch-search-plugins]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). (corrective)
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: libx11-6 but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Java/Java::Package[openjdk-jdk-8]/Package[openjdk-8-jdk]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: libx11-6 but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Logstash/Package[logstash]: Dependency Package[openjdk-8-jdk] has failures: true
Warning: /Stage[main]/Logstash/Package[logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/usr/local/bin/logstash-config-test]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/default/logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/jvm.options]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/log4j2.properties]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/logstash.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/conf.d/README]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/init/logstash.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/init/logstash-web.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Alternatives::Java[8]/Exec[update_java_alternatives_8]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Java::Cacert[wmf:Wikimedia_Internal_Root_CA]/Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Elasticsearch::Curator/Package[elasticsearch-curator]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Dependency Package[elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/srv/elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch.log]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_search_slowlog.log]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Service[elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/elasticsearch.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/log4j2.properties]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/jvm.options]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/scripts]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Exec[elasticsearch-create-keystore-production-search-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/elasticsearch.keystore]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/srv/elasticsearch/production-search-codfw]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Service[elasticsearch_6@production-search-codfw]: Dependency Package[wmf-elasticsearch-search-plugins] has failures: true
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Service[elasticsearch_6@production-search-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/elasticsearch.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/log4j2.properties]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/jvm.options]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/scripts]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/Exec[elasticsearch-create-keystore-production-search-omega-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/elasticsearch.keystore]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/srv/elasticsearch/production-search-omega-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/Service[elasticsearch_6@production-search-omega-codfw]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-elasticsearch-exporter' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 prometheus-elasticsearch-exporter : Depends: daemon but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Package[prometheus-elasticsearch-exporter]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-elasticsearch-exporter' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 prometheus-elasticsearch-exporter : Depends: daemon but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Service[prometheus-elasticsearch-exporter]: Dependency Package[prometheus-elasticsearch-exporter] has failures: true
Warning: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Service[prometheus-elasticsearch-exporter]: Skipping because of failed dependencies
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Wrapped exception:
No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link
Error: /Stage[main]/Profile::Logstash::Gelf_relay/Logstash::Input::Gelf[gelf_relay]/Logstash::Conf[input-gelf-gelf_relay]/File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Wrapped exception:
No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link
Error: /Stage[main]/Profile::Logstash::Gelf_relay/Logstash::Output::Udp[gelf_relay]/Logstash::Conf[output-udp-gelf_relay]/File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf] has failures: true
Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf] has failures: true
Warning: /Stage[main]/Logstash/Service[logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.service]/File[/lib/systemd/system/elasticsearch-hot-threads-log.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.service]/Exec[systemd daemon-reload for elasticsearch-hot-threads-log.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Systemd::Unit[prometheus-elasticsearch-exporter-9200]/File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Systemd::Unit[prometheus-elasticsearch-exporter-9200]/Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Service[prometheus-elasticsearch-exporter-9200]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9200]
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Systemd::Unit[prometheus-elasticsearch-exporter-9400]/File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Systemd::Unit[prometheus-elasticsearch-exporter-9400]/Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Service[prometheus-elasticsearch-exporter-9400]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9400]
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.timer]/File[/lib/systemd/system/elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.timer]/Exec[systemd daemon-reload for elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Service[elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies
Info: Class[Profile::Elasticsearch::Cirrus]: Unscheduling all events on Class[Profile::Elasticsearch::Cirrus]
Notice: Applied catalog in 22.24 seconds