Page MenuHomePhabricator

Replace deployment-prometheus02
Closed, ResolvedPublic

Description

tbc, see T324695: beta-prometheus.wmflabs.org 502 Bad Gateway / T306068: Cloud VPS "deployment-prep" project Stretch deprecation

Creating deployment-prometheus03 with g3.cores4.ram8.disk20 now.

Ended up creating as deployment-prometheus05 because I had to re-create the VM twice (reusing hostnames didn't seem to work..) — same spec.

Event Timeline

With role::prometheus::beta, running puppet fails to install prometheus with E: Unable to correct problems, you have held broken packages. — will take a look later unless someone else gets to it


samtar@deployment-prometheus03:~$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for deployment-prometheus03.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(39d6b74956c) root - Don't create a second disk through lvm'
Notice: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: /usr/local/sbin/make-instance-vg: lvm is not active on this host; unable to create a volume.
Error: '/usr/local/sbin/make-instance-vg '/dev/sda'' returned 1 instead of one of [0]
Error: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/make-instance-vg '/dev/sda'' returned 1 instead of one of [0] (corrective)
Notice: /Stage[main]/Profile::Labs::Lvm::Srv/Labs_lvm::Volume[second-local-disk]/Exec[create-vd-second-local-disk]: Dependency Exec[create-volume-group] has failures: true
Warning: /Stage[main]/Profile::Labs::Lvm::Srv/Labs_lvm::Volume[second-local-disk]/Exec[create-vd-second-local-disk]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Labs::Lvm::Srv/Labs_lvm::Volume[second-local-disk]/Mount[/srv]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Labs::Lvm::Srv/Labs_lvm::Volume[second-local-disk]/File[/srv]: Skipping because of failed dependencies
Warning: /Stage[main]/Prometheus/File[/srv/prometheus]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 prometheus : Depends: libjs-jquery (>= 3.5.1~) but 3.3.1~dfsg-3+deb10u1 is to be installed
E: Unable to correct problems, you have held broken packages.
Error: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/Package[prometheus]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 prometheus : Depends: libjs-jquery (>= 3.5.1~) but 3.3.1~dfsg-3+deb10u1 is to be installed
E: Unable to correct problems, you have held broken packages.
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/File[/srv/prometheus/beta]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/File[/srv/prometheus/beta/prometheus.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/File[/srv/prometheus/beta/metrics]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/File[/srv/prometheus/beta/targets]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Class_config[memcached_eqiad]/File[/srv/prometheus/beta/targets/memcached_eqiad.yaml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Class_config[mcrouter_eqiad]/File[/srv/prometheus/beta/targets/mcrouter_eqiad.yaml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Jmx_exporter_config[kafka_broker_eqiad]/File[/srv/prometheus/beta/targets/jmx_kafka_broker_beta_eqiad.yaml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Jmx_exporter_config[kafka_mirrormaker_eqiad]/File[/srv/prometheus/beta/targets/jmx_kafka_mirrormaker_beta_eqiad.yaml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/File[/srv/prometheus/beta/rules]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/File[/srv/prometheus/beta/rules/alerts_default.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Rule[rules_beta.yml]/File[/srv/prometheus/beta/rules/rules_beta.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Rule[alerts_beta.yml]/File[/srv/prometheus/beta/rules/alerts_beta.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/Exec[prometheus@beta-reload]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Labs::Lvm::Srv/Labs_lvm::Volume[second-local-disk]/Labs_lvm::Extend[/srv]/Exec[extend-vd-/srv]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/Systemd::Service[prometheus@beta]/Service[prometheus@beta]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Profile::Prometheus::Beta/Prometheus::Server[beta]/Systemd::Service[prometheus@beta]/Service[prometheus@beta]: Unscheduling refresh on Service[prometheus@beta]
Info: Prometheus::Server[beta]: Unscheduling all events on Prometheus::Server[beta]
Notice: Applied catalog in 9.99 seconds

Ended up being deployment-prometheus05 because puppet.

Change 868510 had a related patch set uploaded (by Samtar; author: Samtar):

[operations/puppet@production] deployment-prep: update prometheus host to prometheus05

https://gerrit.wikimedia.org/r/868510

Change 868510 merged by Cathal Mooney:

[operations/puppet@production] deployment-prep: update prometheus host to prometheus05

https://gerrit.wikimedia.org/r/868510