https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=restbase1004&nostatusheader
https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=restbase2002&nostatusheader
https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=restbase2005&nostatusheader
restbase1004, restbase2002, restbase2005
DPKG CRITICAL dpkg reports broken packages
CRITICAL: Puppet has 1 failures
root@restbase1001:~# puppet agent -tv | grep Warning Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install nodejs' returned 100: Reading package lists... Building dependency tree... Reading state information... The following packages were automatically installed and are no longer required: libc-ares-dev libv8-3.14-dev libv8-3.14.5 rlwrap tzdata-java Use 'apt-get autoremove' to remove them. The following extra packages will be installed: libuv1 libuv1-dev nodejs-dev Suggested packages: debhelper The following NEW packages will be installed: libuv1 libuv1-dev The following held packages will be changed: nodejs The following packages will be upgraded: nodejs nodejs-dev 2 upgraded, 2 newly installed, 0 to remove and 166 not upgraded. Need to get 4009 kB of archives. After this operation, 11.8 MB of additional disk space will be used. E: There are problems and -y was used without --force-yes Error: /Stage[main]/Packages::Nodejs/Package[nodejs]/ensure: change from held to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install nodejs' returned 100: Reading package lists... Building dependency tree... Reading state information... The following packages were automatically installed and are no longer required: libc-ares-dev libv8-3.14-dev libv8-3.14.5 rlwrap tzdata-java Use 'apt-get autoremove' to remove them. The following extra packages will be installed: libuv1 libuv1-dev nodejs-dev Suggested packages: debhelper The following NEW packages will be installed: libuv1 libuv1-dev The following held packages will be changed: nodejs The following packages will be upgraded: nodejs nodejs-dev 2 upgraded, 2 newly installed, 0 to remove and 166 not upgraded. Need to get 4009 kB of archives. After this operation, 11.8 MB of additional disk space will be used. E: There are problems and -y was used without --force-yes Warning: /Stage[main]/Restbase/Service::Node[restbase]/File[/etc/restbase]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/File[/etc/restbase/config.yaml]: Skipping because of failed dependencies Warning: /Group[restbase]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/Service::Deployment_script[restbase]/File[/usr/local/bin/restbase-deploy]: Skipping because of failed dependencies Warning: /User[restbase]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/Ferm::Service[restbase]/File[/etc/ferm/conf.d/10_restbase]: Skipping because of failed dependencies Warning: /Stage[main]/Ferm/Service[ferm]: Skipping because of failed dependencies Warning: /Package[restbase/deploy]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/File[/etc/systemd/system/restbase.service]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/File[/lib/systemd/system/restbase.service]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/Exec[systemd reload for restbase]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/Service[restbase]: Skipping because of failed dependencies Warning: /Stage[main]/Restbase/Service::Node[restbase]/Nrpe::Monitor_service[endpoints_restbase]/Nrpe::Check[check_endpoints_restbase]/File[/etc/nagios/nrpe.d/check_endpoints_restbase.cfg]: Skipping because of failed dependencies Warning: /Stage[main]/Nrpe/Service[nagios-nrpe-server]: Skipping because of failed dependencies
hi nodejs 0.10.29~dfsg-2 amd64 evented I/O for V8 javascript ii nodejs-dev 0.10.29~dfsg-2 amd64 evented I/O for V8 javascript (development files) ii nodejs-legacy 0.10.29~dfsg-2 all evented I/O for V8 javascript (legacy symlink)
i manually ran apt-get install nodejs and ran puppet again. the error disappeared.
was this update triggered by an ensure => latest or a human ?
ii nodejs 4.2.4~dfsg-1~bpo8+1 amd64 evented I/O for V8 javascript
14:45 < icinga-wm> RECOVERY - DPKG on restbase1001 is OK: All packages OK
14:46 < icinga-wm> RECOVERY - puppet last run on restbase1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
14:47 < icinga-wm> RECOVERY - DPKG on restbase2002 is OK: All packages OK
14:48 < icinga-wm> RECOVERY - DPKG on restbase2005 is OK: All packages OK
14:48 < icinga-wm> RECOVERY - puppet last run on restbase2005 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
14:49 < icinga-wm> RECOVERY - puppet last run on restbase2002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
14:50 < mutante> !log restbase1004, restbase2002, restbase2005 - manually install nodejs
14:53 < icinga-wm> RECOVERY - DPKG on restbase1004 is OK: All packages OK
@Dzahn, you basically migrated the remaining nodes to 4.2. We originally planned to do this tomorrow to be absolutely sure that 4.2 is indeed working as intended: T107762
While it sounds like RESTBase itself was not yet restarted, individual workers will pick up 4.2. We should restart RESTBase on 1001-1004 tomorrow to make sure that nodes are actually running 4.2.
In the future, we should really make sure that such alerts are acknowledged in time as a temporary, intended state to avoid misunderstandings.
This was applied inconsistently (restbase1002/1003 were still running 0.10). I downgraded 1001/1004 to 0.10 again and did NOT set dpkg's "hold" flag, as both puppet and the icinga check are too silly to handle it (and we don't really need it). We should be OK for now.
This has been a disappointing turn of events. Let's do the next upgrade a bit more coordinated, both from the technical perspective (e.g. not having those Icinga alerts in the first place and not proceeding with any upgrades while alerts are set) and far more importantly, from the communication/social perspective (e.g. @Dzahn was unaware of this upgrade being in progress — I would have let the team know during our weekly meeting, but these upgrades happened during the allhands/weekend…).
Daniel, for future reference: nodejs (like others, e.g. HHVM) is far too critical to upgrade like that.
I was the one doing this specific upgrade with @mobrovac on purpose for mobileapps and mathoid on Thursday.
Daniel, for future reference: nodejs (like others, e.g. HHVM) is far too critical to upgrade like that.
From my perspective this just popped up in Icinga as CRIT puppet and DPKG on multiple servers and when i saw the nodejs package was in status "halF-conf", i was worried things would be broken and just wanted to get the immediate issue fixed. I first expected it to be an "ensure => latest" in puppet that went wrong.
Sorry if i caused any additonal problems.