Page MenuHomePhabricator

Q3:rack/setup/install frdata1003, frmx1002, frqueue100[5-6]
Closed, ResolvedPublic

Description

This task will track the racking, setup, and OS installation of frdata1003, frmx1002, frqueue100[5-6]

Hostname / Racking / Installation Details

Hostnames: What are the hostnames, and have you updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions ?
Racking Proposal: Where should these systems be racked? Can they share with any existing systems or should they avoid any other systems sharing their rack or row? Use https://fault-tolerance.toolforge.org/map to optimize this placement.
Networking Setup: # of Connections:1/2 - Speed:1G/10G. - VLAN:Private/Public/Other(Specify) :
OS Distro: Bookworm (default unless otherwise specified)
Boot Method: Legacy BIOS or UEFI. Please note UEFI must have partman updates applied in advance of setup and is currently in pilot program: https://wikitech.wikimedia.org/wiki/UEFI_Boot
Sub-team Technical Contact: Who should our on-sites contact with any questions involving system racking and setup?

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

frdata1003
  • Receive in system on procurement task T412559 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
frmx1002
  • Receive in system on procurement task T412559 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
frqueue1005
  • Receive in system on procurement task T412559 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
frqueue1006
  • Receive in system on procurement task T412559 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Other Assignee
Jclark-ctr

Event Timeline

Jhancock.wm mentioned this in Unknown Object (Task).Feb 2 2026, 9:28 PM

frqueue1005
1st CableID: 230304500128
Port 26

2nd CableID: 239394599180
Port 26

frqueue1006
1st CableID: 230304500102
port 27

2nd CableID: 230304500108
port 27

(Power cables have been plugged in, but servers haven't been turned on)

the Fundraising Server Provision for Dual-Switch Uplinks script in netbox is it setup to only look for old codfw currently reached out to Cathal by irc. No new fundraising servers in eqiad are showing up under devices as option

Recommended changing the tenant to "Fundraising Tech". Once that was set in netbox, the scripts could see those units.

Jclark-ctr updated the task description. (Show Details)
Jclark-ctr updated Other Assignee, added: Jclark-ctr.
Jclark-ctr added subscribers: Jgreen, Jclark-ctr.

@Jgreen Hey jeff these are setup and reachable. password should be set to default dell password

These have all been updated to the frack management password.

Working on frqueue1005:

  • disabled the "embedded" NICs
  • set serial port address: COM2
  • set console redirection after boot: enabled
  • switched boot method to UEFI
  • confirmed that first "integrated" NIC pxeboot is enabled

Additional/new:

  • in BIOS / Network Settings enable UEFI Networking
  • in BIOS / Network Settings enabled "PXE Device1"
Booting from PXE Device 1: Integrated NIC 1 Port 1 Partition 1

>>Start PXE over IPv4.
  PXE-E18: Server response timeout.
Boot Failed: PXE Device 1: Integrated NIC 1 Port 1 Partition 1

No boot device available or Operating System detected. 
Please ensure a compatible bootable media is available.

No DHCP request reaches the server...

frmx1002's management interface isn't accessible, doesn't respond to ping

@Jclark-ctr I finally had a chance to look into the frmx1002/frdata1003 issue. I think the servers are in the correct rack, but I think they were put in the wrong vlan, so they'll need port and IP changes. These two servers should be in frack-listenerdmz not frack-fundraising.

frqueue1005/frqueue1006 are up and running

@Jclark-ctr I finally had a chance to look into the frmx1002/frdata1003 issue. I think the servers are in the correct rack, but I think they were put in the wrong vlan, so they'll need port and IP changes. These two servers should be in frack-listenerdmz not frack-fundraising.

@cmooney Could you assist with this next week?

@Jgreen Is there a way to determine which VLAN should be assigned to each server? The available options I see are:

fundraising
administration
payments
listenerdmz
bastion

Also, would you be able to include this information in future fundraising tickets T418928

@cmooney Could you assist with this next week?

Done now, I've moved the frmx1002 and frdata1003 ports into frack-listenerdmz1-e15-eqiad and assigned new IPs for them from the appropriate subnet.

@Jgreen Is there a way to determine which VLAN should be assigned to each server? The available options I see are:

Yes we need this information up front to know where to rack the servers. Servers for the funraising vlan (frack-fundraising1-e16-eqiad) need to go in rack E16, all others need to go in rack E15. Following the patch we merged earlier in the week Netbox will prevent us trying to add one to a vlan incompatible with the rack, but by then we've usually racked the server so we want to catch it earlier.

@Jgreen if you can select "Other(Specify)" for the vlan option in this form and then list the specific one that should hopefully make sure we stay ahead here. Thanks.

@Jclark-ctr I finally had a chance to look into the frmx1002/frdata1003 issue. I think the servers are in the correct rack, but I think they were put in the wrong vlan, so they'll need port and IP changes. These two servers should be in frack-listenerdmz not frack-fundraising.

@cmooney Could you assist with this next week?

@Jgreen Is there a way to determine which VLAN should be assigned to each server? The available options I see are:

fundraising
administration
payments
listenerdmz
bastion

The only way currently is I guess is to look at the rack/install task for the host it replaces, but that's a pain. I'll try to come up with something better.

Also, would you be able to include this information in future fundraising tickets T418928

Absolutely. I missed the window for this task, I didn't even spot it until you reassigned to me.

I'm not sure how this works for other teams., but would it make sense to assign the install task to me (or whoever signed off on the procurement) when it's created?

All four are switched to UEFI and built.