rack/setup/install labvirt102[12]
Open, NormalPublic

Description

This task will track the racking and setup of labvirt102[12].eqiad.wmnet. These are essentially the same as the rest of our labvirts, using the same row (B) and same vlan/os as the rest of labvirt servers.

Special Note: Please ensure the virtualization options for the CPUs on these hosts are ENABLED, this is the opposite of the majority of the fleet.

Racking Proposal: These MUST be in row B of eqiad, with other labvirts.

labvirt1021:

  • - receive in system on procurement task T178937
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, labs vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

labvirt1022:

  • - receive in system on procurement task T178937
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, labs vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation
RobH created this task.Tue, Jan 2, 5:21 PM
RobH triaged this task as Normal priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Jan 2, 5:21 PM
Cmjohnson moved this task from Backlog to Being worked on on the ops-eqiad board.Wed, Jan 3, 4:06 PM

Change 401793 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns labvirt1021/22

https://gerrit.wikimedia.org/r/401793

Change 401793 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns labvirt1021/22

https://gerrit.wikimedia.org/r/401793

Cmjohnson updated the task description. (Show Details)Wed, Jan 3, 9:54 PM

These seem pretty close, any chance they are on the agenda for early next week? We are looking at a potential resource crunch for CPU and these would be heartwarming to have ready to go.

Change 402361 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns labvirt1021/22

https://gerrit.wikimedia.org/r/402361

Change 402361 merged by Cmjohnson:
[operations/dns@master] Adding production dns labvirt1021/22

https://gerrit.wikimedia.org/r/402361

Change 402417 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Add new labvirts to netboot and dhcpd

https://gerrit.wikimedia.org/r/402417

Change 402417 merged by Cmjohnson:
[operations/puppet@production] Add new labvirts to netboot and dhcpd

https://gerrit.wikimedia.org/r/402417

Cmjohnson updated the task description. (Show Details)Mon, Jan 8, 8:24 PM

These are hitting the install server but not receiving the image. Chasemp or robh can you take a look at this please. They were received with 10G Nics that I turned off and set the 1GB Nic to pxe. Please verify that everything looks okay.

chasemp added a subscriber: Andrew.EditedTue, Jan 9, 5:06 PM

@Andrew said he would have a minute to take a look and that this sounded vaguely familiar

Andrew added a comment.EditedWed, Jan 10, 10:52 PM

OK -- on 1021 in System Setup:Device Settings I see one NIC with four ports:

Integrated NIC 1 Port 1: Intel(R) Ethernet 10G 4P X520/I350 rNDC -            
 24:6E:96:8D:B3:A0                                                            
Integrated NIC 1 Port 2: Intel(R) Ethernet 10G 4P X520/I350 rNDC -            
 24:6E:96:8D:B3:A2                                                            
Integrated NIC 1 Port 3: Intel(R) Gigabit 4P X520/I350 rNDC -                 
 24:6E:96:8D:B3:A4                                                            
Integrated NIC 1 Port 4: Intel(R) Gigabit 4P X520/I350 rNDC -                 
 24:6E:96:8D:B3:A5

Is that everything, or is there also an external NIC?

What I would expect is for 1 and 2 to be disabled, and for 3 and 4 to show that they're connected (3 for eth0 and 4 for eth1). That's not how it looks to me, which makes me think I'm missing something.

In System Bios I see "Integrated Network Card 1" which is shown as 'enabled'. I assume that's the same nic 1 that I'm seeing in Device Settings... I don't so far see how to selectively disable ports on that one card without disabling the whole card.

Andrew added a subscriber: Joe.Mon, Jan 15, 6:09 PM

I powered these off for the moment, just to cut down on dhcp noise.

Note from T184909 that these were spamming teh DHCP server

I've seen this on install1002 via syslog, both servers seem to send multiple DHCPDISCOVER requests every minute, to which the dhcp server answers correctly.

We should take a hard look at what is going on there, something seems deifnitely wrong.