a ticket has been opened with Dell

You have successfully submitted request SR984761946.

the ssd is on-site, it's /dev/sda...the disk will need to be failed before I can replace. This server may need a reinstall if /dev/sda does not rebuild

@aborrero can this server be re-installed.....there is a risk that removing /dev/sda will kill the OS.

In T209029#4919182, @Cmjohnson wrote:

@aborrero can this server be re-installed.....there is a risk that removing /dev/sda will kill the OS.

I believe the server can be reinstalled right away.

Just checked:

cloudelastic1004 is a Unused spare system (spare::system)

The disk has been replaced, @aborrero the OS will need to be re-installed. Until then the raid is out of whack because I removed /dev/sda.

In T209029#4935906, @Cmjohnson wrote:

The disk has been replaced, @aborrero the OS will need to be re-installed. Until then the raid is out of whack because I removed /dev/sda.

Thanks, I will reimage it and will discuss the team how to proceed with this server.

WMCS needs discussion: what do we want to do with this server? can it live with spare::system for now?

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudelastic1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902081304_aborrero_22060_cloudelastic1004_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cloudelastic1004.eqiad.wmnet']

Of which those FAILED:

['cloudelastic1004.eqiad.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudelastic1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902081305_aborrero_22235_cloudelastic1004_eqiad_wmnet.log.

Mentioned in SAL (#wikimedia-operations) [2019-02-08T13:05:55Z] <arturo> T209029 reimaging cloudelastic1004

Completed auto-reimage of hosts:

['cloudelastic1004.eqiad.wmnet']

Of which those FAILED:

['cloudelastic1004.eqiad.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudelastic1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902081334_aborrero_27096_cloudelastic1004_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['cloudelastic1004.eqiad.wmnet']

Of which those FAILED:

['cloudelastic1004.eqiad.wmnet']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudelastic1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201902081335_aborrero_28639_cloudelastic1004_eqiad_wmnet.log.

I'm having troubles reimaging the server:

aborrero@cumin1001:~$ sudo -i wmf-auto-reimage-host -p T209029 --no-verify --no-downtime --no-reboot cloudelastic1004.eqiad.wmnet
13:35:30 | cloudelastic1004.eqiad.wmnet | REIMAGE START | To monitor the full log and cumin output:
sudo tail -F /var/log/wmf-auto-reimage/201902081335_aborrero_28639_cloudelastic1004_eqiad_wmnet.log
sudo tail -F /var/log/wmf-auto-reimage/201902081335_aborrero_28639_cloudelastic1004_eqiad_wmnet_cumin.out
IPMI Password: 
13:35:44 | cloudelastic1004.eqiad.wmnet | Removed from Puppet
13:35:44 | cloudelastic1004.eqiad.wmnet | WARNING: Unable to remove from Debmonitor, got: 404
13:35:44 | cloudelastic1004.eqiad.wmnet | Set Boot Device to pxe
13:35:45 | cloudelastic1004.eqiad.wmnet | Power cycling
13:35:45 | cloudelastic1004.eqiad.wmnet | Chassis Power Control: Cycle
13:39:45 | cloudelastic1004.eqiad.wmnet | Still waiting for reboot after 5.0 minutes
13:44:46 | cloudelastic1004.eqiad.wmnet | Still waiting for reboot after 10.0 minutes
13:49:46 | cloudelastic1004.eqiad.wmnet | Still waiting for reboot after 15.0 minutes
13:54:46 | cloudelastic1004.eqiad.wmnet | Still waiting for reboot after 20.0 minutes
13:59:47 | cloudelastic1004.eqiad.wmnet | Still waiting for reboot after 25.0 minutes
[...]

The debian installer completes, but I can't log in because apparently the first puppet run isn't completed and I can't use any login methods (ssh or direct console access).

The debian installer completes, but I can't log in because apparently the first puppet run isn't completed and I can't use any login methods (ssh or direct console access).

This should be cloudelastic1004.wikimedia.org, not cloudelastic1004.eqiad.wmnet.

Completed auto-reimage of hosts:

['cloudelastic1004.eqiad.wmnet']

Of which those FAILED:

['cloudelastic1004.eqiad.wmnet']

In T209029#4937948, @aborrero wrote:

WMCS needs discussion: what do we want to do with this server? can it live with spare::system for now?

My understanding was that the Analytics team was anxiously waiting to get access to their hypervisors. Is this one of them?

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudelastic1004.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201902081618_aborrero_73970_cloudelastic1004_wikimedia_org.log.

Completed auto-reimage of hosts:

['cloudelastic1004.wikimedia.org']

Of which those FAILED:

['cloudelastic1004.wikimedia.org']

Script wmf-auto-reimage was launched by aborrero on cumin1001.eqiad.wmnet for hosts:

cloudelastic1004.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201902081619_aborrero_74286_cloudelastic1004_wikimedia_org.log.

Completed auto-reimage of hosts:

['cloudelastic1004.wikimedia.org']

and were ALL successful.