deployment-apertium01 puppet failing due to missing packages on trusty
Closed, ResolvedPublic

Description

krenair@deployment-apertium01:~$ sudo puppet agent -tv 2>&1 | grep E: | sort -d | uniq
E: Unable to locate package apertium-arg
E: Unable to locate package apertium-arg-cat
E: Unable to locate package apertium-cat
E: Unable to locate package apertium-fra
E: Unable to locate package apertium-fra-cat
E: Unable to locate package apertium-ita
E: Unable to locate package apertium-sme-nob
E: Unable to locate package apertium-spa
E: Unable to locate package apertium-spa-arg
E: Unable to locate package apertium-srd
E: Unable to locate package apertium-srd-ita
E: Unable to locate package apertium-swe
E: Unable to locate package apertium-swe-dan
E: Unable to locate package apertium-swe-nor
E: Unable to locate package giella-core
E: Unable to locate package giella-sme

We might want to do T142152 instead. T107306 already dealt with these for jessie.

This is happening due to cherry-pick of https://gerrit.wikimedia.org/r/#/c/308679/ which is for testing before deployment in Production (https://phabricator.wikimedia.org/T144588) (scheduled this week).

hashar added a comment.Oct 4 2016, 8:36 AM

deployment-apertium01 is a Trusty instance. Maybe we can move apertium to the deployment-sca* instances which are jessie? that is T142152: Move apertium to deployment-sca* hosts in Beta Cluster.

akosiaris closed this task as "Resolved".Oct 4 2016, 9:04 AM
akosiaris claimed this task.
akosiaris added a subscriber: akosiaris.

https://gerrit.wikimedia.org/r/#/c/308679/ fixes this. I 'll close as resolved

hashar reopened this task as "Open".Nov 9 2016, 10:37 AM

deployment-apertium01 (Trusty) has a bunch of missing packages again eg:

# tail -n 180 /var/log/puppet.log|egrep ^E
E: Unable to locate package apertium-swe
E: Unable to locate package apertium-swe
E: Unable to locate package giella-sme
E: Unable to locate package giella-sme
E: Unable to locate package apertium-cat
E: Unable to locate package apertium-cat
E: Unable to locate package apertium-arg
E: Unable to locate package apertium-arg
E: Unable to locate package apertium-srd
E: Unable to locate package apertium-srd
E: Unable to locate package apertium-swe-nor
E: Unable to locate package apertium-swe-nor
E: Unable to locate package apertium-spa-arg
E: Unable to locate package apertium-spa-arg
E: Unable to locate package apertium-spa
E: Unable to locate package apertium-spa
E: Unable to locate package apertium-sme-nob
E: Unable to locate package apertium-sme-nob
E: Unable to locate package apertium-ita
E: Unable to locate package apertium-ita
E: Unable to locate package apertium-swe-dan
E: Unable to locate package apertium-swe-dan
E: Unable to locate package apertium-srd-ita
E: Unable to locate package apertium-srd-ita
E: Unable to locate package apertium-fra
E: Unable to locate package apertium-fra
E: Unable to locate package apertium-arg-cat
E: Unable to locate package apertium-arg-cat
E: Unable to locate package giella-core
E: Unable to locate package giella-core
E: Unable to locate package apertium-fra-cat
E: Unable to locate package apertium-fra-cat
root@deployment-apertium01:~#

Instance IP address is 10.68.16.79

@akosiaris confirmed production moved to Jessie. So I guess we can delete deployment-apertium01 and clean up left over configuration files :}

@KartikMistry I am pretty sure we can delete that VM, could you please confirm ?

@akosiaris @hashar Yes. We can delete deployment-apertium01.

akosiaris closed this task as "Resolved".Nov 9 2016, 11:12 AM

Instance terminated.

hashar reopened this task as "Open".Nov 17 2016, 11:23 AM

deployment-apertium01 is still around and complaining. Maybe the deletion failed in wikitech/horizon?

hashar closed this task as "Resolved".Nov 17 2016, 11:26 AM

I have terminated deployment-apertium01 using Horizon.

Mentioned in SAL (#wikimedia-releng) [2016-11-17T11:34:17Z] <hasharLunch> Deleted instance deployment-apertium01 . Was Trusty and lacked packages, replaced by a Jessie one ages ago. T147210

Somehow the instance deployment-apertium01 is back!

$ uptime
 10:52:58 up 21 days, 15:25,  1 user,  load average: 0.00, 0.09, 0.10
             ^^^^^^^

And thus Shinken cames back to complain about puppet being broken: ** PROBLEM alert - deployment-apertium01/Puppet run is CRITICAL **

Mentioned in SAL (#wikimedia-releng) [2016-11-23T10:57:43Z] <hashar> Terminating deployment-apertium01 again T147210

With 21 days uptime ? I think it's just not deleted. otherwise this does not make sense.

From https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-apertium01.deployment-prep.eqiad.wmflabs

Sorry, this page was recently deleted (within the last 24 hours). The deletion and move log for the page are provided below for reference.

  • 10:57, 23 November 2016 Labslogbot (talk | contribs) deleted page Nova Resource:Deployment-apertium01.deployment-prep.eqiad.wmflabs (Resource deleted)
  • 19:10, 18 November 2016 Delete page script (talk | contribs) deleted page Nova Resource:Deployment-apertium01.deployment-prep.eqiad.wmflabs
  • 21:53, 9 November 2016 Delete page script (talk | contribs) deleted page Nova Resource:Deployment-apertium01.deployment-prep.eqiad.wmflabs
  • 23:41, 4 December 2015 Andrew Bogott (talk | contribs) deleted page Nova Resource:Deployment-apertium01.deployment-prep.eqiad.wmflabs (Cleaning up old instance pages)

Nov 9th would be Alexandros deleting it, Nov 18th and Nov 23th are me. Maybe it is worth filling a new task against OpenStack ?

Apparently it is gone for real. deployment-apertium01 hasn't reappeared and does not show up in the Horizon interface.