Page MenuHomePhabricator
Create Task
Maniphest T242265

rack/setup/install frlog2001.frack.codfw.wmnet
Closed, ResolvedPublic
Actions

Description

  • - receive in system on procurement task T237440
  • - rack system in frack
  • - bios/drac/serial setup/testing
  • - raid6 of all 12 SSDs created
  • - mgmt dns entries added for hostname (fqdn is frlog2001.mgmt.frack.eqiad.wmnet)
  • - network port setup (description, enable, vlan) - attach two network ports, one to each asw, ports assigned to frack-codfw-fundraising vlan.
    • end on-site specific steps - handoff to Fundraising for remainder as they use their own puppet master.
  • - production dns entries added (ip 10.195.0.76, fqdn frlog2001.frack.eqiad.wmnet)
  • - OS installation
  • - interface bond0 configured
  • - host state in netbox set to staged
  • - clone data from heka/frlog1001
  • - firewall policy updates
  • - switch logging configuration for codfw cluster
  • - reconfigure archive_sync script and exchange ssh keys
  • - configure icinga monitoring
  • - service implementer changes from 'staged' status to 'active' status in netbox'

DRAC Setup Info

The drac network and login settings are different on frack than production.
Please setup the network with the following details:

IP:
Netmask:
Gateway:

The root password for drac is different. Please check the password store, and if you still cannot access it, use the production idrac password and ping @RobH (or @Jgreen) to login and change it to the frack idrac password.

Details

SubjectRepoBranchLines +/-
adjust nsca_frack.cfg to monitor frlog2001, replacing bellatrixoperations/puppetproduction+2 -2
DNS: Add mgmt and production DNS for frlog2001operations/dnsmaster+6 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jgreen created this task.Jan 8 2020, 7:15 PM
Jgreen added a subtask: Unknown Object (Task).
Jgreen updated the task description. (Show Details)Jan 8 2020, 9:11 PM
Jgreen updated the task description. (Show Details)
Jgreen added a project: ops-codfw.
Restricted Application added a project: SRE. · View Herald TranscriptJan 8 2020, 9:12 PM
Dzahn triaged this task as Medium priority.Jan 8 2020, 9:26 PM
Jgreen added a subtask: Restricted Task.Jan 8 2020, 9:50 PM
wiki_willy assigned this task to Papaul.Jan 8 2020, 10:29 PM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.Jan 9 2020, 12:48 AM
Jgreen closed subtask Restricted Task as Resolved.Jan 14 2020, 2:41 PM
Papaul updated the task description. (Show Details)Jan 14 2020, 6:54 PM
Papaul updated the task description. (Show Details)Jan 14 2020, 9:13 PM
Papaul added a comment.Jan 14 2020, 9:40 PM
papaul@fasw-c-codfw# show | compare 
[edit interfaces interface-range disabled]
-    member ge-0/0/21;
-    member ge-1/0/21;
[edit interfaces interface-range vlan-administration]
     member "ge-[0-1]/0/16" { ... }
+    member "ge-[0-1]/0/21";
[edit interfaces]
+   ge-0/0/21 {
+       description frlog2001:eth0;
+   }
+   ge-1/0/21 {
+       description frlog2001:eth1;
+   }
papaul@fasw-c-codfw# run show interfaces ge-[0-1]/0/21 descriptions 
Interface       Admin Link Description
ge-0/0/21       up    up   frlog2001:eth0
ge-1/0/21       up    up   frlog2001:eth1
Papaul updated the task description. (Show Details)Jan 14 2020, 9:41 PM
gerritbot added a comment.Jan 15 2020, 5:26 PM

Change 565067 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for frlog2001

https://gerrit.wikimedia.org/r/565067

gerritbot added a project: Patch-For-Review.Jan 15 2020, 5:26 PM
gerritbot added a comment.Jan 15 2020, 7:52 PM

Change 565067 merged by Papaul:
[operations/dns@master] DNS: Add mgmt and production DNS for frlog2001

https://gerrit.wikimedia.org/r/565067

Papaul updated the task description. (Show Details)Jan 15 2020, 8:11 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 15 2020, 8:11 PM
Papaul reassigned this task from Papaul to Jgreen.Jan 15 2020, 8:12 PM
Papaul subscribed.

@Jgreen All yours let me know if you have any questions

Jgreen reassigned this task from Jgreen to Papaul.Jan 16 2020, 3:32 PM

@Papaul I'm getting an error at pxeboot, looks like the cable is not connected or the network port is off? Can you take a look?

Booting from BRCM MBA Slot 1700 v214.0.230.0

Broadcom UNDI PXE-2.1 v214.0.230.0
Copyright (C) 2000-2019 Broadcom Limited
Copyright (C) 1997-2000 Intel Corporation
All rights reserved.
PXE-E61: Media test failure, check cable
PXE-M0F: Exiting Broadcom PXE ROM.

Jgreen added a comment.Jan 16 2020, 5:21 PM

Note we have the same Debian vs unused 10G NIC problem documented here T242481. We're waiting for information from Dell on how to disable the 10G ports in BIOS, meanwhile there's a pxeboot/preseed workaround in T242481.

Papaul reassigned this task from Papaul to Jgreen.Jan 17 2020, 2:21 PM

@Jgreen no good update from Dell

Jgreen added a comment.Jan 17 2020, 2:43 PM
In T242265#5812899, @Papaul wrote:

@Jgreen no good update from Dell

Thanks for looking into it. No problem, I was able to use the fix the SRE team figured out from T242481 and got it imaged.

Jgreen updated the task description. (Show Details)Jan 17 2020, 3:25 PM
Jgreen updated the task description. (Show Details)Jan 21 2020, 6:43 PM
RobH unsubscribed.Jan 21 2020, 7:24 PM
Jgreen added a subtask: Restricted Task.Jan 22 2020, 3:48 PM
ayounsi closed subtask Restricted Task as Resolved.Jan 22 2020, 4:13 PM
gerritbot added a comment.Jan 22 2020, 8:49 PM

Change 566597 had a related patch set uploaded (by Jgreen; owner: Jgreen):
[operations/puppet@production] adjust nsca_frack.cfg to monitor frlog2001, replacing bellatrix

https://gerrit.wikimedia.org/r/566597

gerritbot added a project: Patch-For-Review.Jan 22 2020, 8:49 PM

Change 566597 merged by Jgreen:
[operations/puppet@production] adjust nsca_frack.cfg to monitor frlog2001, replacing bellatrix

https://gerrit.wikimedia.org/r/566597

Jgreen closed this task as Resolved.Jan 22 2020, 8:59 PM
Jgreen updated the task description. (Show Details)
Jgreen moved this task from Triage to Done on the fundraising-tech-ops board.
Jgreen added a subscriber: RobH.
Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 9:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL