Page MenuHomePhabricator
Create Task
Maniphest T266481

(Need By: TBD) rack/setup/install payments100[5-8]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of payments100[5-8]

Hostname / Racking / Installation Details

Hostnames: payments100[5-8]
Racking Proposal: frack
Networking/Subnet/VLAN/IP: connect both 1GBE ports, one per frack asw

  • payments1005.frack.eqiad.wmnet/frack-payments-eqiad/10.64.40.9
  • payments1006.frack.eqiad.wmnet/frack-payments-eqiad/10.64.40.10
  • payments1007.frack.eqiad.wmnet/frack-payments-eqiad/10.64.40.11
  • payments1008.frack.eqiad.wmnet/frack-payments-eqiad/10.64.40.12

Partitioning/Raid:
OS Distro:

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

  • payments1005:
    • - receive in system on procurement task T265064 & in coupa
    • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
    • - bios/drac/serial setup/testing
    • - mgmt dns entries added for both asset tag and hostname
    • - network port setup (description, enable, vlan)
    • end on-site specific steps
    • - DRAC password fixed
    • - DRAC console redirection after boot fixed
    • - production dns entries added
    • - No OS installation, hand off to fundraising to install - fundraising to set to active in netbox when they complete installation.
  • payments1006:
    • - receive in system on procurement task T265064 & in coupa
    • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
    • - bios/drac/serial setup/testing
    • - mgmt dns entries added for both asset tag and hostname
    • - network port setup (description, enable, vlan)
    • end on-site specific steps
    • - DRAC password fixed
    • - DRAC console redirection after boot fixed
    • - production dns entries added
    • - No OS installation, hand off to fundraising to install - fundraising to set to active in netbox when they complete installation.
  • payments1007:
    • - receive in system on procurement task T265064 & in coupa
    • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
    • - bios/drac/serial setup/testing
    • - mgmt dns entries added for both asset tag and hostname
    • - network port setup (description, enable, vlan)
    • end on-site specific steps
    • - DRAC password fixed
    • - DRAC console redirection after boot fixed
    • - production dns entries added
    • - No OS installation, hand off to fundraising to install - fundraising to set to active in netbox when they complete installation.
  • payments1008:
    • - receive in system on procurement task T265064 & in coupa
    • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
    • - bios/drac/serial setup/testing
    • - mgmt dns entries added for both asset tag and hostname
    • - network port setup (description, enable, vlan)
    • end on-site specific steps
    • - DRAC password fixed
    • - DRAC console redirection after boot fixed
    • - production dns entries added
    • - No OS installation, hand off to fundraising to install - fundraising to set to active in netbox when they complete installation.

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
Add new payments hosts to monitoringoperations/puppetproduction+52 -0
add A/PTR records for payments100[5-8].frack.eqiad.wmnetoperations/dnsmaster+7 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Oct 26 2020, 3:37 PM
Restricted Application added a project: SRE. · View Herald TranscriptOct 26 2020, 3:37 PM
RobH updated the task description. (Show Details)Oct 26 2020, 3:37 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.Oct 26 2020, 3:42 PM
Jgreen added a parent task: T257244: 2020-2021 fundraising hardware refresh and capex.Oct 29 2020, 8:01 PM
wiki_willy assigned this task to • Cmjohnson.Dec 4 2020, 9:24 PM
wiki_willy subscribed.

Arrived Dec 3

wiki_willy added a subscriber: Jgreen.Jan 25 2021, 8:50 PM

Hi @Jgreen - it looks like we're running a bit tight on space in the Fundraising rack. In order for us to rack the servers for this install, do you have 1-2 existing servers that can be decommissioned in eqiad? Thanks, Willy

Jclark-ctr updated the task description. (Show Details)Jan 25 2021, 8:53 PM
Jgreen added a comment.Jan 26 2021, 4:25 PM
In T266481#6775156, @wiki_willy wrote:

Hi @Jgreen - it looks like we're running a bit tight on space in the Fundraising rack. In order for us to rack the servers for this install, do you have 1-2 existing servers that can be decommissioned in eqiad? Thanks, Willy

There are a couple DC Ops tasks in the works that will lead to removing old hardware:

T271739: decommission frdb1001.frack.eqiad.wmnet
T266365: (Need By: TBD) rack/setup/install frqueue100[34]

Let's do those first, and then do the payments servers two at a time?

wiki_willy added a subscriber: Jclark-ctr.Jan 26 2021, 4:28 PM

Thanks @Jgreen (cc'ing @Jclark-ctr as a fyi)

In T266481#6777535, @Jgreen wrote:
In T266481#6775156, @wiki_willy wrote:

Hi @Jgreen - it looks like we're running a bit tight on space in the Fundraising rack. In order for us to rack the servers for this install, do you have 1-2 existing servers that can be decommissioned in eqiad? Thanks, Willy

There are a couple DC Ops tasks in the works that will lead to removing old hardware:

T271739: decommission frdb1001.frack.eqiad.wmnet
T266365: (Need By: TBD) rack/setup/install frqueue100[34]

Let's do those first, and then do the payments servers two at a time?

Jgreen added a comment.Feb 12 2021, 9:01 PM

@wiki_willy @Jclark-ctr we're done with frqueue1002 and can be decommed and removed T274671: decommission frqueue1002.frack.eqiad.wmnet. When you're ready start on payments boxes, we can also shut down payments1004 which is a standby box and not in active service.

wiki_willy reassigned this task from • Cmjohnson to Jclark-ctr.Feb 12 2021, 9:02 PM
wiki_willy added a subscriber: • Cmjohnson.
wiki_willy added a comment.Feb 18 2021, 5:41 PM

Thanks @Jgreen, do you have a decom task for payments1004 as well?

In T266481#6827394, @Jgreen wrote:

@wiki_willy @Jclark-ctr we're done with frqueue1002 and can be decommed and removed T274671: decommission frqueue1002.frack.eqiad.wmnet. When you're ready start on payments boxes, we can also shut down payments1004 which is a standby box and not in active service.

Jclark-ctr updated the task description. (Show Details)Feb 24 2021, 11:57 PM
Jclark-ctr added a comment.Feb 25 2021, 12:12 AM

@Cmjohnson
4 Servers racked updated netbox. servers use same ports in both switches
payments1005 U2. P30
payments1006 U26. P28
payments1007. U14. P30
payments1008. U28. P29

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Feb 25 2021, 12:12 AM
Volans subscribed.Feb 25 2021, 8:27 AM
In T266481#6827394, @Jgreen wrote:

@wiki_willy @Jclark-ctr we're done with frqueue1002 and can be decommed and removed T274671: decommission frqueue1002.frack.eqiad.wmnet. When you're ready start on payments boxes, we can also shut down payments1004 which is a standby box and not in active service.

I see frqueue1002 was decomm'ed but the sre.dns.netbox cookbook was not run for its mgmt records, so Icinga started alerting because of:

Uncommitted DNS changes in Netbox on netbox1001 is CRITICAL: Netbox has uncommitted DNS changes
https://wikitech.wikimedia.org/wiki/Monitoring/Netbox_DNS_uncommitted_changes

I'm running the cookbook now to clear the mgmt records and the alert.
Please make sure that the documentation reflects that the cookbook must be run also when decommissioning frack hosts because the management records are automatically managed.

Jclark-ctr added a comment.Feb 25 2021, 8:31 AM

Thanks. I was mistaken that network dns was done by fundraising

Jgreen updated the task description. (Show Details)Mar 11 2021, 1:55 PM
• Cmjohnson added a comment.Mar 12 2021, 3:23 PM

updated port locations

1005 28
1006 29
1007 30
1008 31

• Cmjohnson updated the task description. (Show Details)Mar 15 2021, 6:27 PM
• Cmjohnson updated the task description. (Show Details)Mar 18 2021, 5:37 PM
• Cmjohnson reassigned this task from • Cmjohnson to Jgreen.Mar 18 2021, 5:39 PM
• Cmjohnson edited projects, added fundraising-tech-ops; removed ops-eqiad, DC-Ops.

Assigning this to @Jgreen to complete the installs. All the on-site work has been completed, network ports are set up and enabled so please install quickly. Please resolve this task once completed and let me know if you need anything.
Removing ops-eqiad and DC Ops tag, adding fundraising tech-ops.

Jgreen moved this task from Triage to In Progress on the fundraising-tech-ops board.Mar 18 2021, 7:06 PM
gerritbot added a comment.Mar 18 2021, 7:12 PM

Change 673342 had a related patch set uploaded (by Jgreen; owner: Jgreen):
[operations/dns@master] add A/PTR records for payments100[5-8].frack.eqiad.wmnet

https://gerrit.wikimedia.org/r/673342

gerritbot added a project: Patch-For-Review.Mar 18 2021, 7:12 PM
gerritbot added a comment.Mar 18 2021, 7:15 PM

Change 673342 merged by Jgreen:
[operations/dns@master] add A/PTR records for payments100[5-8].frack.eqiad.wmnet

https://gerrit.wikimedia.org/r/673342

Jgreen updated the task description. (Show Details)Mar 18 2021, 7:20 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 18 2021, 8:10 PM
Jgreen added subtasks: Unknown Object (Task), T278250: (Need By: TBD) install second SSD into payments100[5-8].Mar 31 2021, 6:45 PM
Jgreen updated the task description. (Show Details)Apr 2 2021, 2:14 PM
Jgreen updated the task description. (Show Details)
Jgreen added a comment.Apr 2 2021, 2:20 PM

@Cmjohnson @Jclark-ctr Can you take a look at the serial settings for payments1006? Console redirection isn't working at all. I tried to fix it using racadm but although it seems to accept the settings there's still no DRAC access on com2. Maybe there's something else going on?

Jgreen reassigned this task from Jgreen to • Cmjohnson.Apr 2 2021, 4:02 PM
Jgreen moved this task from In Progress to Blocked on the fundraising-tech-ops board.Apr 2 2021, 6:38 PM
Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Apr 16 2021, 6:05 PM
Dwisehaupt added projects: DC-Ops, ops-eqiad.Apr 19 2021, 7:09 PM
Dwisehaupt subscribed.

Re-adding the ops tags to get payments1006 back on the radar for the console redirection.

Dwisehaupt removed projects: ops-eqiad, DC-Ops.Apr 19 2021, 7:13 PM

Backing that out and removing the ops tags as @Jgreen created T280527 this morning specifically about the console issue.

Jclark-ctr closed subtask T280527: payments1006.frack.eqiad.wmnet DRAC no console output as Resolved.Apr 21 2021, 9:06 PM
RhinosF1 updated the task description. (Show Details)Apr 21 2021, 9:08 PM
Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptApr 21 2021, 9:08 PM
Dwisehaupt updated the task description. (Show Details)Apr 23 2021, 5:44 PM
gerritbot added a comment.Apr 23 2021, 6:24 PM

Change 682186 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/puppet@production] Add new payments hosts to monitoring

https://gerrit.wikimedia.org/r/682186

gerritbot added a project: Patch-For-Review.Apr 23 2021, 6:24 PM
Dwisehaupt updated the task description. (Show Details)Apr 23 2021, 7:02 PM
Jclark-ctr closed subtask T278250: (Need By: TBD) install second SSD into payments100[5-8] as Resolved.Apr 30 2021, 7:48 PM
gerritbot added a comment.May 13 2021, 4:22 PM

Change 682186 merged by Jgreen:

[operations/puppet@production] Add new payments hosts to monitoring

https://gerrit.wikimedia.org/r/682186

Maintenance_bot removed a project: Patch-For-Review.May 13 2021, 5:10 PM
Jgreen closed this task as Resolved.May 17 2021, 3:43 PM
Jgreen moved this task from Blocked to Done on the fundraising-tech-ops board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL