Page MenuHomePhabricator

huggle.huggle.eqiad.wmflabs does not have puppet installed
Closed, ResolvedPublic

Description

It looks like @Petrb deliberately uninstalled the Puppet agent from this instance on 2017-05-19 13:17:32.

This is dangerous for the long term health of the instance per https://wikitech.wikimedia.org/wiki/Puppet_coding#Wikimedia_Cloud_VPS

/var/log/apt/term.log
Log started: 2017-05-19  13:17:32
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 66405 files and directories currently installed.)
Removing build-essential (11.6ubuntu6) ...
Removing puppet (3.8.5-2~bpo8trusty+2) ...
 * Stopping puppet agent       ESC[271G ^MESC[265G[ OK ]
Removing puppet-common (3.8.5-2~bpo8trusty+2) ...
Removing facter (1.7.5-1ubuntu1) ...
Removing g++ (4:4.8.2-1ubuntu6) ...
Removing g++-4.8 (4.8.4-2ubuntu1~14.04.3) ...
Removing ruby-hiera (1.3.4-1~bpo8trusty+1) ...
Removing hiera (1.3.4-1~bpo8trusty+1) ...
Removing libganglia1 (3.6.0-1ubuntu2) ...
Removing libconfuse0:amd64 (2.7-4ubuntu1) ...
Removing libconfuse-common (2.7-4ubuntu1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.11) ...
Selecting previously unselected package libruby1.8.
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 64883 files and directories currently installed.)
Preparing to unpack .../libruby1.8_1.8.7.358-8ubuntu3_amd64.deb ...
Unpacking libruby1.8 (1.8.7.358-8ubuntu3) ...
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 65573 files and directories currently installed.)
Removing ruby-shadow (2.2.0-1) ...
Removing ruby-safe-yaml (1.0.1-1) ...
dpkg: ruby: dependency problems, but removing anyway as you requested:
 ruby1.9.1 depends on ruby (>= 1:1.9.3.1).

Removing ruby (1:1.9.3.4) ...
dpkg: libruby1.9.1: dependency problems, but removing anyway as you requested:
 ruby-json depends on libruby1.9.1 (>= 1.9.2.0) | libruby2.0 (>= 2.0.0); however:
  Package libruby1.9.1 is to be removed.
  Package libruby2.0 is not installed.
 ruby-augeas depends on libruby1.9.1 (>= 1.9.2.0) | libruby2.0 (>= 2.0.0); however:
  Package libruby1.9.1 is to be removed.
  Package libruby2.0 is not installed.

Removing libruby1.9.1 (1.9.3.484-2ubuntu1.2) ...
dpkg: ruby1.9.1: dependency problems, but removing anyway as you requested:
 ruby-rgen depends on ruby | ruby-interpreter; however:
  Package ruby is not installed.
  Package ruby1.8 which provides ruby is not installed.
  Package ruby-interpreter is not installed.
  Package ruby1.8 which provides ruby-interpreter is not installed.
  Package ruby1.9.1 which provides ruby-interpreter is to be removed.
 ruby-json depends on ruby | ruby-interpreter; however:
  Package ruby is not installed.
  Package ruby1.8 which provides ruby is not installed.
  Package ruby-interpreter is not installed.
  Package ruby1.8 which provides ruby-interpreter is not installed.
  Package ruby1.9.1 which provides ruby-interpreter is to be removed.
 puppet-lint depends on ruby | ruby-interpreter; however:
  Package ruby is not installed.
  Package ruby1.8 which provides ruby is not installed.
  Package ruby-interpreter is not installed.
  Package ruby1.8 which provides ruby-interpreter is not installed.
  Package ruby1.9.1 which provides ru
Removing ruby1.9.1 (1.9.3.484-2ubuntu1.2) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.11) ...
Selecting previously unselected package ruby1.8.
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 64636 files and directories currently installed.)
Preparing to unpack .../ruby1.8_1.8.7.358-8ubuntu3_amd64.deb ...
Unpacking ruby1.8 (1.8.7.358-8ubuntu3) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 64659 files and directories currently installed.)
Removing ruby-rgen (0.6.6-1) ...
Removing libstdc++5:amd64 (1:3.3.6-25ubuntu4) ...
Removing linux-tools-3.13.0-113-generic (3.13.0-113.160) ...
Removing linux-tools-3.13.0-113 (3.13.0-113.160) ...
Removing linux-tools-3.13.0-115-generic (3.13.0-115.162) ...
Removing linux-tools-3.13.0-115 (3.13.0-115.162) ...
Removing linux-tools-3.13.0-116-generic (3.13.0-116.163) ...
Removing linux-tools-3.13.0-116 (3.13.0-116.163) ...
Removing python-support (1.0.15) ...
Removing ubuntu-release-upgrader-core (1:0.220.9) ...
Removing ruby-augeas (0.5.0-2) ...
Removing ruby-json (1.8.0-1build1) ...
Removing python3-distupgrade (1:0.220.9) ...
Removing python3-update-manager (1:0.196.22) ...
Processing triggers for libc-bin (2.19-0ubuntu6.11) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Selecting previously unselected package libgettextpo0:amd64.
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 64334 files and directories currently installed.)
Preparing to unpack .../libgettextpo0_0.18.3.1-1ubuntu3_amd64.deb ...
Unpacking libgettextpo0:amd64 (0.18.3.1-1ubuntu3) ...
Selecting previously unselected package command-not-found-data.
Preparing to unpack .../command-not-found-data_0.3ubuntu12_amd64.deb ...
Unpacking command-not-found-data (0.3ubuntu12) ...
Selecting previously unselected package apache2.2-bin.
Preparing to unpack .../apache2.2-bin_2.4.7-1ubuntu4.15_amd64.deb ...
Unpacking apache2.2-bin (2.4.7-1ubuntu4.15) ...
Selecting previously unselected package html2text.
Preparing to unpack .../html2text_1.3.2a-17_amd64.deb ...
Unpacking html2text (1.3.2a-17) ...
Selecting previously unselected package libaugeas-ruby1.8.
Preparing to unpack .../libaugeas-ruby1.8_0.3.0-1.1ubuntu4_amd64.deb ...
Unpacking libaugeas-ruby1.8 (0.3.0-1.1ubuntu4) ...
Selecting previously unselected package libclass-isa-perl.
Unpacking libclass-isa-perl (0.36-5) ...
Selecting previously unselected package libshadow-ruby1.8.
Preparing to unpack .../libshadow-ruby1.8_1.4.1-8build1_amd64.deb ...
Unpacking libshadow-ruby1.8 (1.4.1-8build1) ...
Selecting previously unselected package libswitch-perl.
Preparing to unpack .../libswitch-perl_2.16-2_all.deb ...
Unpacking libswitch-perl (2.16-2) ...
Selecting previously unselected package mono-dmcs.
Preparing to unpack .../mono-dmcs_3.2.8+dfsg-4ubuntu1.1_all.deb ...
Unpacking mono-dmcs (3.2.8+dfsg-4ubuntu1.1) ...
Selecting previously unselected package mono-gmcs.
Preparing to unpack .../mono-gmcs_3.2.8+dfsg-4ubuntu1.1_all.deb ...
Unpacking mono-gmcs (3.2.8+dfsg-4ubuntu1.1) ...
Selecting previously unselected package python-boto.
Preparing to unpack .../python-boto_2.20.1-2ubuntu2_all.deb ...
Unpacking python-boto (2.20.1-2ubuntu2) ...
Selecting previously unselected package python-dbus-dev.
Preparing to unpack .../python-dbus-dev_1.2.0-2build2_all.deb ...
Unpacking python-dbus-dev (1.2.0-2build2) ...
Selecting previously unselected package python-dbus.
Preparing to unpack .../python-dbus_1.2.0-2build2_amd64.deb ...
Unpacking python-dbus (1.2.0-2build2) ...
Selecting previously unselected package python-ecdsa.
Preparing to unpack .../python-ecdsa_0.10-2_all.deb ...
Unpacking python-ecdsa (0.10-2) ...
Selecting previously unselected package python-gdbm.
Preparing to unpack .../python-gdbm_2.7.5-1ubuntu1_amd64.deb ...
Unpacking python-gdbm (2.7.5-1ubuntu1) ...
Selecting previously unselected package python-gi.
Preparing to unpack .../python-gi_3.12.0-1ubuntu1_amd64.deb ...
Unpacking python-gi (3.12.0-1ubuntu1) ...
Selecting previously unselected package python-gnupginterface.
Preparing to unpack .../python-gnupginterface_0.3.2-9.1ubuntu3_all.deb ...
Unpacking python-gnupginterface (0.3.2-9.1ubuntu3) ...
Selecting previously unselected package python-paramiko.
Preparing to unpack .../python-paramiko_1.15.2-1_all.deb ...
Unpacking python-paramiko (1.15.2-1) ...
Selecting previously unselected package python-simplejson.
Preparing to unpack .../python-simplejson_3.3.1-1ubuntu6_amd64.deb ...
Unpacking python-simplejson (3.3.1-1ubuntu6) ...
Selecting previously unselected package rlwrap.
Preparing to unpack .../rlwrap_0.37-5_amd64.deb ...
Unpacking rlwrap (0.37-5) ...
Selecting previously unselected package ttf-dejavu-core.
Preparing to unpack .../ttf-dejavu-core_2.34-1ubuntu1_all.deb ...
Unpacking ttf-dejavu-core (2.34-1ubuntu1) ...
Selecting previously unselected package libev4.
Preparing to unpack .../libev4_1%3a4.15-3_amd64.deb ...
Unpacking libev4 (1:4.15-3) ...
Selecting previously unselected package libev-dev.
Preparing to unpack .../libev-dev_1%3a4.15-3_amd64.deb ...
Unpacking libev-dev (1:4.15-3) ...
Selecting previously unselected package openjdk-6-jdk:amd64.
Preparing to unpack .../openjdk-6-jdk_6b41-1.13.13-0ubuntu0.14.04.1_amd64.deb ...
Unpacking openjdk-6-jdk:amd64 (6b41-1.13.13-0ubuntu0.14.04.1) ...
Processing triggers for mime-support (3.54ubuntu1.1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for fontconfig (2.11.0-0ubuntu4.2) ...
Setting up libruby1.8 (1.8.7.358-8ubuntu3) ...
Setting up ruby1.8 (1.8.7.358-8ubuntu3) ...
update-alternatives: using /usr/bin/ruby1.8 to provide /usr/bin/ruby (ruby) in auto mode
Setting up libgettextpo0:amd64 (0.18.3.1-1ubuntu3) ...
Setting up command-not-found-data (0.3ubuntu12) ...
Setting up apache2.2-bin (2.4.7-1ubuntu4.15) ...
Setting up html2text (1.3.2a-17) ...
Setting up libaugeas-ruby1.8 (0.3.0-1.1ubuntu4) ...
Setting up libclass-isa-perl (0.36-5) ...
Setting up libshadow-ruby1.8 (1.4.1-8build1) ...
Setting up libswitch-perl (2.16-2) ...
Setting up mono-dmcs (3.2.8+dfsg-4ubuntu1.1) ...
Setting up mono-gmcs (3.2.8+dfsg-4ubuntu1.1) ...
Setting up python-boto (2.20.1-2ubuntu2) ...
Setting up python-dbus-dev (1.2.0-2build2) ...
Setting up python-dbus (1.2.0-2build2) ...
Remove stale byte-compiled files...
Setting up python-ecdsa (0.10-2) ...
Setting up python-gdbm (2.7.5-1ubuntu1) ...
Setting up python-gi (3.12.0-1ubuntu1) ...
Setting up python-gnupginterface (0.3.2-9.1ubuntu3) ...
Setting up python-paramiko (1.15.2-1) ...
Setting up python-simplejson (3.3.1-1ubuntu6) ...
Setting up rlwrap (0.37-5) ...
update-alternatives: using /usr/bin/rlwrap to provide /usr/bin/readline-editor (readline-editor) in auto mode
Setting up ttf-dejavu-core (2.34-1ubuntu1) ...
Setting up libev4 (1:4.15-3) ...
Setting up libev-dev (1:4.15-3) ...
Setting up openjdk-6-jdk:amd64 (6b41-1.13.13-0ubuntu0.14.04.1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.11) ...
Log ended: 2017-05-19  13:17:55
$ last
root     pts/2        bastion-01.basti Tue May 30 16:43   still logged in
root     pts/2        bastion-restrict Tue May 30 16:37 - 16:39  (00:01)
petrb    pts/2        tools-bastion-02 Fri May 19 21:20 - 21:31  (00:11)
petrb    pts/2        tools-bastion-02 Fri May 19 14:23 - 16:35  (02:11)
petrb    pts/1        tools-bastion-02 Fri May 19 11:52 - 13:40  (01:48

Event Timeline

Once detached from regular Puppet runs, a virtual machine is in a state where the cloud-services-team cannot reliably support it.

Framawiki moved this task from Need discussion to Backlog on the Huggle board.

hmm, that I probably did, I have very bad experience with puppet, it happened to me couple of times that puppet completely ruined the instance and made it defunct, then I was told to just build a new one instead. Did the situation get better since?

Did the situation get better since?

I can't really say if non specific complaints from a non specific time have or have not been addressed.

I can say that any VM that is detached from applying configuration changes controlled by operations/puppet.git will eventually become unusable and a security hazard that the Cloud Services admins will be forced to shutdown. The guarantee of usability is due to things like rotation of the LDAPS x509 certificate that we recently performed because the prior certificate expired. Once an instance is detached from LDAP the authn/authz controls for allowing ssh access will fail.

bd808 triaged this task as High priority.Aug 4 2017, 4:54 PM
bd808 edited projects, added Cloud-VPS; removed Cloud-Services.

When @Andrew went through all Cloud-VPS vms today to switch them to the new project-wide puppetmaster, this instance was the only vm that is not directly or indirectly attached to puppet. We need to figure out how to either reattach it so that it stays up to date with basic Cloud Services config or build a replacement that will.

I would be glad to work with @Petrb to try and figure out what system config the project needs that is in conflict with the basic Cloud Services Puppet configuration.

We found this box has actually already become disconnected from LDAP. @bd808 added our keys to root's authorized_keys and I've figured out how to insert the new WMF CA and restart nslcd so that it allows our logins.
But this machine is being replaced anyway

Petrb claimed this task.

instance is down

instance is down

Thank you. :)