Page MenuHomePhabricator
Create Task
Maniphest T208733

Rename labvirt1017 to cloudvirt1017, move to eqiad1
Closed, ResolvedPublic
Actions

Description

After quarry is migrated later today, this host can be moved to the new region.

Reimage + rename this server to the new naming scheme.

Timeline would be:

Details

SubjectRepoBranchLines +/-
Nova: add cloudvirt1017 to the scheduler pooloperations/puppetproduction+2 -1
cloudvps: hieradata for cloudvirt1017operations/puppetproduction+0 -0
cloudvps: rename+reimage labvirt1017 as cloudvirt1017operations/dnsmaster+5 -4
cloudvps: reimage+rename labvirt1017 as cloudvirt1017operations/puppetproduction+11 -11
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Nov 5 2018, 3:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 5 2018, 3:59 PM
aborrero claimed this task.Nov 5 2018, 6:28 PM
aborrero triaged this task as Medium priority.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added subscribers: • GTirloni, aborrero.

I will handle this task with @GTirloni (if he is interested).

• GTirloni added a comment.Nov 5 2018, 7:43 PM

Sure, let's do it!

• GTirloni updated the task description. (Show Details)Nov 7 2018, 12:59 PM
• GTirloni removed a subscriber: Aklapper.
aborrero updated the task description. (Show Details)Nov 7 2018, 1:00 PM
• GTirloni mentioned this in T208945: Relabel labvirt1017.eqiad.wmnet as cloudvirt1017.eqiad.wmnet.Nov 7 2018, 1:04 PM
• GTirloni updated the task description. (Show Details)Nov 7 2018, 1:07 PM
gerritbot subscribed.Nov 7 2018, 1:12 PM

Change 472140 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] cloudvps: reimage+rename labvirt1017 as cloudvirt1017

https://gerrit.wikimedia.org/r/472140

gerritbot added a project: Patch-For-Review.Nov 7 2018, 1:12 PM
aborrero updated the task description. (Show Details)Nov 7 2018, 1:16 PM
gerritbot added a comment.Nov 7 2018, 1:16 PM

Change 472142 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/dns@master] cloudvps: rename+reimage labvirt1017 as cloudvirt1017

https://gerrit.wikimedia.org/r/472142

aborrero updated the task description. (Show Details)Nov 7 2018, 1:20 PM
aborrero updated the task description. (Show Details)Nov 7 2018, 1:23 PM
gerritbot added a comment.Nov 7 2018, 1:48 PM

Change 472140 merged by GTirloni:
[operations/puppet@production] cloudvps: reimage+rename labvirt1017 as cloudvirt1017

https://gerrit.wikimedia.org/r/472140

gerritbot added a comment.Nov 7 2018, 1:49 PM

Change 472142 merged by GTirloni:
[operations/dns@master] cloudvps: rename+reimage labvirt1017 as cloudvirt1017

https://gerrit.wikimedia.org/r/472142

Stashbot subscribed.Nov 7 2018, 1:49 PM

Mentioned in SAL (#wikimedia-cloud) [2018-11-07T13:49:39Z] <arturo> T208733 moving labvirt1017 from main deployment to eqiad1 and renaming it to cloudvirt1017

aborrero updated the task description. (Show Details)Nov 7 2018, 1:53 PM
ops-monitoring-bot subscribed.Nov 7 2018, 2:11 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

labvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071411_gtirloni_21761_labvirt1017_eqiad_wmnet.log.

• GTirloni added a comment.Nov 7 2018, 2:13 PM

wmf-auto-reimage-host -p T208733 --rename cloudvirt1017.eqiad.wmnet --rename-mgmt cloudvirt1017.mgmt.eqiad.wmnet labvirt1017.eqiad.wmnet labvirt1017.mgmt.eqiad.wmnet

ops-monitoring-bot added a comment.Nov 7 2018, 2:21 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 2:22 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

labvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071422_gtirloni_24260_labvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 2:22 PM

Completed auto-reimage of hosts:

['labvirt1017.eqiad.wmnet']

Of which those FAILED:

['labvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 2:29 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

labvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071429_gtirloni_25942_labvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 2:48 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 3:49 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071549_gtirloni_14020_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 3:49 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 3:49 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071549_gtirloni_14176_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 4:05 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 4:15 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071615_gtirloni_21777_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 4:15 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 4:21 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071621_gtirloni_23337_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 4:22 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 4:22 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071622_gtirloni_24537_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 4:38 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 4:39 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071639_gtirloni_29369_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 4:39 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 5:14 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071713_gtirloni_7282_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 5:14 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 7 2018, 5:15 PM

Script wmf-auto-reimage was launched by gtirloni on neodymium.eqiad.wmnet for hosts:

cloudvirt1017.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201811071714_gtirloni_7523_cloudvirt1017_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 7 2018, 5:31 PM

Completed auto-reimage of hosts:

['cloudvirt1017.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1017.eqiad.wmnet']
• GTirloni added a comment.EditedNov 7 2018, 5:35 PM

Faced too many issues with the reimage script. This is where it's last failing:

# wmf-auto-reimage-host --no-downtime -p T208733 cloudvirt1017.eqiad.wmnet cloudvirt1017.mgmt.eqiad.wmnet
17:14:55 | cloudvirt1017.eqiad.wmnet | REIMAGE START | To monitor the full log and cumin output:
sudo tail -F /var/log/wmf-auto-reimage/201811071714_gtirloni_7523_cloudvirt1017_eqiad_wmnet.log
sudo tail -F /var/log/wmf-auto-reimage/201811071714_gtirloni_7523_cloudvirt1017_eqiad_wmnet_cumin.out
IPMI Password: 
17:15:03 | cloudvirt1017.eqiad.wmnet | Validated host
17:15:09 | cloudvirt1017.eqiad.wmnet | Removed from Puppet
17:15:09 | cloudvirt1017.eqiad.wmnet | WARNING: Unable to remove from Debmonitor, got: 404
17:15:09 | cloudvirt1017.eqiad.wmnet | Set Boot Device to pxe
17:15:09 | cloudvirt1017.eqiad.wmnet | Power cycling
17:15:09 | cloudvirt1017.eqiad.wmnet | Chassis Power Control: Cycle
17:19:16 | cloudvirt1017.eqiad.wmnet | Still waiting for reboot after 5.0 minutes
17:19:16 | cloudvirt1017.eqiad.wmnet | Uptime checked
17:19:16 | cloudvirt1017.eqiad.wmnet | Host up (Debian installer)
17:23:46 | cloudvirt1017.eqiad.wmnet | Still waiting for reboot after 5.0 minutes
17:28:47 | cloudvirt1017.eqiad.wmnet | Still waiting for reboot after 10.0 minutes
17:30:52 | cloudvirt1017.eqiad.wmnet | Uptime checked
17:30:52 | cloudvirt1017.eqiad.wmnet | Host up
17:30:57 | cloudvirt1017.eqiad.wmnet | Puppet CSR generated, fingerprint is: A6:AC:55:2B:EC:FD:0F:4D:FA:8E:ED:72:3B:DA:E1:4D:9D:FA:43:EC:D1:15:E1:94:81:93:97:38:39:E3:CA:82
17:30:57 | cloudvirt1017.eqiad.wmnet | Polling until a Puppet sign request appears
17:31:01 | cloudvirt1017.eqiad.wmnet | Signed Puppet cert
17:31:03 | cloudvirt1017.eqiad.wmnet | Validated host
17:31:03 | cloudvirt1017.eqiad.wmnet | Scheduled delayed downtime on Icinga
17:31:03 | cloudvirt1017.eqiad.wmnet | Started first puppet run (sit back, relax, and enjoy the wait)
17:31:03 | cloudvirt1017.eqiad.wmnet | Sleeping for 120 seconds
17:31:22 | cloudvirt1017.eqiad.wmnet | Unable to run wmf-auto-reimage-host: Failed to puppet_first_run
17:31:22 | cloudvirt1017.eqiad.wmnet | REIMAGE END | retcode=2
17:33:03 | cloudvirt1017.eqiad.wmnet | Running Puppet on the Icinga server
17:37:17 | einsteinium.wikimedia.org | Puppet run completed
ERROR:cumin.transports.clustershell.AsyncEventHandler:100.0% (1/1) of nodes failed to execute command 'icinga-downtime ...ab_task=T208733'': einsteinium.wikimedia.org
CRITICAL:cumin.transports.clustershell.AsyncEventHandler:0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
17:37:18 | cloudvirt1017.eqiad.wmnet | Unable to run wmf-downtime-host: Failed to icinga_downtime
ERROR:wmf-downtime-host:Unable to run wmf-downtime-host
Traceback (most recent call last):
  File "/usr/local/sbin/wmf-downtime-host", line 67, in main
    lib.icinga_downtime(args.host, user, args.phab_task_id, title='wmf-downtime-host')
  File "/usr/local/lib/python3.4/dist-packages/wmf_auto_reimage_lib.py", line 536, in icinga_downtime
    run_cumin('icinga_downtime', icinga_host, [command])
  File "/usr/local/lib/python3.4/dist-packages/wmf_auto_reimage_lib.py", line 469, in run_cumin
    raise RuntimeError('Failed to {label}'.format(label=label))
RuntimeError: Failed to icinga_downtime
gerritbot added a comment.Nov 7 2018, 6:15 PM

Change 472223 had a related patch set uploaded (by GTirloni; owner: GTirloni):
[operations/puppet@production] cloudvps: hieradata for cloudvirt1017

https://gerrit.wikimedia.org/r/472223

gerritbot added a comment.Nov 7 2018, 6:17 PM

Change 472223 merged by GTirloni:
[operations/puppet@production] cloudvps: hieradata for cloudvirt1017

https://gerrit.wikimedia.org/r/472223

• GTirloni updated the task description. (Show Details)Nov 7 2018, 6:33 PM
• GTirloni added a comment.Nov 7 2018, 7:58 PM

cloudvirt1017 is up & running now by using the old-school method (manually make server boot through PXE and use install-console).

gerritbot added a comment.Nov 7 2018, 8:43 PM

Change 472253 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: add cloudvirt1017 to the scheduler pool

https://gerrit.wikimedia.org/r/472253

gerritbot added a comment.Nov 8 2018, 10:33 AM

Change 472253 merged by GTirloni:
[operations/puppet@production] Nova: add cloudvirt1017 to the scheduler pool

https://gerrit.wikimedia.org/r/472253

• GTirloni added a comment.Nov 8 2018, 10:43 AM

VMs are being successfully spawned on cloudvirt1017 (2 tests VMs and one from a user project so far).

• GTirloni updated the task description. (Show Details)Nov 8 2018, 11:18 AM
aborrero updated the task description. (Show Details)Nov 12 2018, 12:57 PM
aborrero moved this task from Doing to Blocked on the cloud-services-team (Kanban) board.

All pending stuff is blocked by T208945.

aborrero changed the task status from Open to Stalled.Nov 12 2018, 1:08 PM
• Cmjohnson closed subtask T208945: Relabel labvirt1017.eqiad.wmnet as cloudvirt1017.eqiad.wmnet as Resolved.Nov 26 2018, 5:20 PM
Andrew closed this task as Resolved.Nov 26 2018, 7:26 PM
Andrew updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits