Page MenuHomePhabricator

rack/setup/install frnetmon1001.frack.eqiad.wmnet
Closed, ResolvedPublic

Description

This task will track the racking, setup, and installation of a new netsec host (replacing bismuth) in eqiad with the hostname of frnetmon1001.

Please note that this is a fundraising host, and doesn't follow all the normal steps of a production host.

  • - receive in system on procurement task T230082
  • - rack system in fundraising rack & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag (wmf5253) and hostname (10.64.40.207)
  • - network port setup (description, enable, vlan) - 1G port to each fasw (@Cmjohnson is familiar with this setup)
  • end on-site specific steps - may want to hand off to @Jgreen for the remainder.
  • - production dns entries added (10.64.40.67)
  • - OS installation
  • - calling into frack puppet
  • - host state in netbox set to staged (please don't forget this)
  • - pushed into service
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Event Timeline

RobH triaged this task as Medium priority.Sep 5 2019, 6:21 PM
RobH created this task.
RobH renamed this task from rack/setup/install new eqiad netsec server to rack/setup/install frnetmon1001.Sep 5 2019, 6:21 PM
RobH added a parent task: Unknown Object (Task).
RobH updated the task description. (Show Details)

Change 541362 had a related patch set uploaded (by Jgreen; owner: Jgreen):
[operations/dns@master] add frban1001 and frnetmon1001

https://gerrit.wikimedia.org/r/541362

Change 541362 merged by Jgreen:
[operations/dns@master] add frban1001 and frnetmon1001

https://gerrit.wikimedia.org/r/541362

Jgreen closed subtask Restricted Task as Resolved.Oct 11 2019, 5:14 PM
Jgreen renamed this task from rack/setup/install frnetmon1001 to rack/setup/install frnetmon1001.frack.eqiad.wmnet.Nov 15 2019, 6:54 PM
wiki_willy renamed this task from rack/setup/install frnetmon1001.frack.eqiad.wmnet to (No Need By Date Provided) rack/setup/install frnetmon1001.frack.eqiad.wmnet.Nov 22 2019, 9:01 PM
Jgreen renamed this task from (No Need By Date Provided) rack/setup/install frnetmon1001.frack.eqiad.wmnet to (ASAP) rack/setup/install frnetmon1001.frack.eqiad.wmnet.Jan 7 2020, 9:32 PM
Jgreen raised the priority of this task from Medium to High.
Jgreen updated the task description. (Show Details)
Jgreen updated the task description. (Show Details)

@Cmjohnson this hasn't moved since November, does it just need network port setup?

Moving task back to the untriaged pool

eth0 c1a port 6
eth1 c1b port 6

RobH renamed this task from (ASAP) rack/setup/install frnetmon1001.frack.eqiad.wmnet to (Need by: ASAP) rack/setup/install frnetmon1001.frack.eqiad.wmnet.Feb 24 2020, 9:10 PM

@Cmjohnson @Jclark-ctr could someone do the final bits on this task so we can finish the deploy? This host arrived nearly 6 months ago!

So I'm not familiar with the frack vlans and bonding setup for interfaces. However, if someone can point out a server this should duplicate in vlan/network settings, I can attempt to duplicate it for this system.

There are a number of vlans on the fsaw: frack-DMZ1-c-eqiad, frack-administration1-c-eqiad, frack-bastion1-c-eqiad (doubt this one), frack-fundraising1-c-eqiad, frack-listenerdmz1-c-eqiad, frack-management1-c-eqiad, frack-payments1-c-eqiad

I've gone ahead and labeled port 6 for this server, so its clearly defined on the switch. However, I did not remove it from the disabled group, as there is no vlan to put it in until I know which one.

Ok, updated the switch (Thanks Arzhel) and put things into the admin vlan to match dns that was already setup.

This should be good to from the network side.

Jgreen lowered the priority of this task from High to Medium.Mar 2 2020, 7:40 PM
Jgreen moved this task from Watching to In Progress on the fundraising-tech-ops board.

I'm not able to access the management IP, can you check it out?

jgreen@frbast1001:~$ dig frnetmon1001.mgmt.frack.eqiad.wmnet

; <<>> DiG 9.10.3-P4-Debian <<>> frnetmon1001.mgmt.frack.eqiad.wmnet
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34842
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;frnetmon1001.mgmt.frack.eqiad.wmnet. IN A

;; ANSWER SECTION:
frnetmon1001.mgmt.frack.eqiad.wmnet. 3600 IN A 10.64.40.207

;; Query time: 0 msec
;; SERVER: 10.3.0.1#53(10.3.0.1)
;; WHEN: Mon Mar 02 21:35:16 UTC 2020
;; MSG SIZE rcvd: 80

jgreen@frbast1001:~$ ping 10.64.40.207
PING 10.64.40.207 (10.64.40.207) 56(84) bytes of data.
^C

  • 10.64.40.207 ping statistics ---

2 packets transmitted, 0 received, 100% packet loss, time 1000ms

Jgreen removed Jgreen as the assignee of this task.Mar 2 2020, 9:39 PM

@Cmjohnson @Jclark-ctr can someone take a look at the management interface situation? It's not accessible via network as far as I can tell.

Jgreen renamed this task from (Need by: ASAP) rack/setup/install frnetmon1001.frack.eqiad.wmnet to rack/setup/install frnetmon1001.frack.eqiad.wmnet.Mar 4 2020, 6:59 PM
Jgreen moved this task from In Progress to Up Next on the fundraising-tech-ops board.

Management interface network config had some issues, fixed now.

Cmjohnson removed a project: ops-eqiad.

Removing ops-eqiad tag and assigning to @Jgreen

  • enabled console redirection after boot

Updates to iptables rulesets completed. frnetmon1001 added to configs where bismuth is present.

Migrated nessus code, config and reports from bismuth to frnetmon1001 using the instructions here: https://community.tenable.com/s/article/Migrating-Nessus-to-new-Server-Linux

Only change is that the setting of the proxy bits couldn't be done 'secure' until the server was registered so I ran them without the --secure flag for now.

Running a test internal PCI scan and credentialed audit scan to verify functionality.

Internal PCI scan and credentialed audit scan run as expected.

Need to add to icinga once T247855 refactor is done.

Nessus package removed from bismuth and /opt/nessus removed. the tar.gz backup that was used during the transfer is still present on the host, just in case.

Change 582912 had a related patch set uploaded (by Dwisehaupt; owner: Dwisehaupt):
[operations/puppet@production] Add frnetmon1001 as a host to check

https://gerrit.wikimedia.org/r/582912

Change 582912 merged by Jgreen:
[operations/puppet@production] Add frnetmon1001 as a host to check

https://gerrit.wikimedia.org/r/582912

Monitoring enabled. Just need to update netbox and we can close this out.

Jgreen updated the task description. (Show Details)
Jgreen moved this task from In Progress to Done on the fundraising-tech-ops board.