Page MenuHomePhabricator

restbase - nodejs package upgrade - puppet fail
Closed, ResolvedPublic

Event Timeline

Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn subscribed.
Dzahn triaged this task as High priority.Jan 11 2016, 10:39 PM
root@restbase1001:~# puppet agent -tv | grep Warning
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install nodejs' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libc-ares-dev libv8-3.14-dev libv8-3.14.5 rlwrap tzdata-java
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libuv1 libuv1-dev nodejs-dev
Suggested packages:
  debhelper
The following NEW packages will be installed:
  libuv1 libuv1-dev
The following held packages will be changed:
  nodejs
The following packages will be upgraded:
  nodejs nodejs-dev
2 upgraded, 2 newly installed, 0 to remove and 166 not upgraded.
Need to get 4009 kB of archives.
After this operation, 11.8 MB of additional disk space will be used.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Packages::Nodejs/Package[nodejs]/ensure: change from held to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install nodejs' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libc-ares-dev libv8-3.14-dev libv8-3.14.5 rlwrap tzdata-java
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libuv1 libuv1-dev nodejs-dev
Suggested packages:
  debhelper
The following NEW packages will be installed:
  libuv1 libuv1-dev
The following held packages will be changed:
  nodejs
The following packages will be upgraded:
  nodejs nodejs-dev
2 upgraded, 2 newly installed, 0 to remove and 166 not upgraded.
Need to get 4009 kB of archives.
After this operation, 11.8 MB of additional disk space will be used.
E: There are problems and -y was used without --force-yes
Warning: /Stage[main]/Restbase/Service::Node[restbase]/File[/etc/restbase]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/File[/etc/restbase/config.yaml]: Skipping because of failed dependencies
Warning: /Group[restbase]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/Service::Deployment_script[restbase]/File[/usr/local/bin/restbase-deploy]: Skipping because of failed dependencies
Warning: /User[restbase]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/Ferm::Service[restbase]/File[/etc/ferm/conf.d/10_restbase]: Skipping because of failed dependencies
Warning: /Stage[main]/Ferm/Service[ferm]: Skipping because of failed dependencies
Warning: /Package[restbase/deploy]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/File[/etc/systemd/system/restbase.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/File[/lib/systemd/system/restbase.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/Exec[systemd reload for restbase]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/Base::Service_unit[restbase]/Service[restbase]: Skipping because of failed dependencies
Warning: /Stage[main]/Restbase/Service::Node[restbase]/Nrpe::Monitor_service[endpoints_restbase]/Nrpe::Check[check_endpoints_restbase]/File[/etc/nagios/nrpe.d/check_endpoints_restbase.cfg]: Skipping because of failed dependencies
Warning: /Stage[main]/Nrpe/Service[nagios-nrpe-server]: Skipping because of failed dependencies
hi  nodejs                         0.10.29~dfsg-2            amd64        evented I/O for V8 javascript
ii  nodejs-dev                     0.10.29~dfsg-2            amd64        evented I/O for V8 javascript (development files)
ii  nodejs-legacy                  0.10.29~dfsg-2            all          evented I/O for V8 javascript (legacy symlink)

i manually ran apt-get install nodejs and ran puppet again. the error disappeared.

was this update triggered by an ensure => latest or a human ?

ii nodejs 4.2.4~dfsg-1~bpo8+1 amd64 evented I/O for V8 javascript


14:45 < icinga-wm> RECOVERY - DPKG on restbase1001 is OK: All packages OK
14:46 < icinga-wm> RECOVERY - puppet last run on restbase1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
14:47 < icinga-wm> RECOVERY - DPKG on restbase2002 is OK: All packages OK
14:48 < icinga-wm> RECOVERY - DPKG on restbase2005 is OK: All packages OK
14:48 < icinga-wm> RECOVERY - puppet last run on restbase2005 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
14:49 < icinga-wm> RECOVERY - puppet last run on restbase2002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
14:50 < mutante> !log restbase1004, restbase2002, restbase2005 - manually install nodejs
14:53 < icinga-wm> RECOVERY - DPKG on restbase1004 is OK: All packages OK

Dzahn lowered the priority of this task from High to Medium.Jan 11 2016, 11:01 PM

lowering priority because manual fix has been applied

@Dzahn, you basically migrated the remaining nodes to 4.2. We originally planned to do this tomorrow to be absolutely sure that 4.2 is indeed working as intended: T107762

While it sounds like RESTBase itself was not yet restarted, individual workers will pick up 4.2. We should restart RESTBase on 1001-1004 tomorrow to make sure that nodes are actually running 4.2.

In the future, we should really make sure that such alerts are acknowledged in time as a temporary, intended state to avoid misunderstandings.

faidon claimed this task.
faidon subscribed.

This was applied inconsistently (restbase1002/1003 were still running 0.10). I downgraded 1001/1004 to 0.10 again and did NOT set dpkg's "hold" flag, as both puppet and the icinga check are too silly to handle it (and we don't really need it). We should be OK for now.

This has been a disappointing turn of events. Let's do the next upgrade a bit more coordinated, both from the technical perspective (e.g. not having those Icinga alerts in the first place and not proceeding with any upgrades while alerts are set) and far more importantly, from the communication/social perspective (e.g. @Dzahn was unaware of this upgrade being in progress — I would have let the team know during our weekly meeting, but these upgrades happened during the allhands/weekend…).

Daniel, for future reference: nodejs (like others, e.g. HHVM) is far too critical to upgrade like that.

This was applied inconsistently (restbase1002/1003 were still running 0.10). I downgraded 1001/1004 to 0.10 again and did NOT set dpkg's "hold" flag, as both puppet and the icinga check are too silly to handle it (and we don't really need it). We should be OK for now.

This has been a disappointing turn of events. Let's do the next upgrade a bit more coordinated, both from the technical perspective (e.g. not having those Icinga alerts in the first place and not proceeding with any upgrades while alerts are set) and far more importantly, from the communication/social perspective (e.g. @Dzahn was unaware of this upgrade being in progress — I would have let the team know during our weekly meeting, but these upgrades happened during the allhands/weekend…).

I was the one doing this specific upgrade with @mobrovac on purpose for mobileapps and mathoid on Thursday.

Daniel, for future reference: nodejs (like others, e.g. HHVM) is far too critical to upgrade like that.

From my perspective this just popped up in Icinga as CRIT puppet and DPKG on multiple servers and when i saw the nodejs package was in status "halF-conf", i was worried things would be broken and just wanted to get the immediate issue fixed. I first expected it to be an "ensure => latest" in puppet that went wrong.
Sorry if i caused any additonal problems.