Page MenuHomePhabricator
Create Task
Maniphest T166180

rack/setup/install netmon2001
Closed, ResolvedPublic
Actions

Description

This task will track the receiving, racking, and installation of netmon2001, ordered on T161807.

Racking Plan: This is the only network monitoring system in codfw. It uses 1Gbps networking, so place in a 1Gbit networking rack with the most space available and power overhead. It can basically go in any 1Gbps rack other than frack.

  • - receive in system on procurement task T161807.
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - production dns entries added (external subnet)
  • - network port setup (description, enable, external vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

also see: T159756 (setup netmon1002 the eqiad equivalent)

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedDzahnT166180 rack/setup/install netmon2001

Event Timeline

RobH created this task.May 23 2017, 9:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2017, 9:04 PM
RobH added a project: ops-codfw.May 23 2017, 9:04 PM
gerritbot subscribed.Jun 27 2017, 12:23 AM

Change 361606 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: allow rsync of data from netmon1001 to netmon1002

https://gerrit.wikimedia.org/r/361606

gerritbot added a project: Patch-For-Review.Jun 27 2017, 12:23 AM
Dzahn subscribed.Jun 27 2017, 12:26 AM

oops, wrong ticket, disregard last comment

faidon moved this task from Inbox to In progress on the observability board.Jul 10 2017, 12:36 PM
Papaul updated the task description. (Show Details)Jul 10 2017, 3:30 PM
gerritbot added a comment.Jul 10 2017, 5:50 PM

Change 364260 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for netmon2001

https://gerrit.wikimedia.org/r/364260

Papaul updated the task description. (Show Details)Jul 10 2017, 5:52 PM
gerritbot added a comment.Jul 10 2017, 6:13 PM

Change 364260 merged by Dzahn:
[operations/dns@master] DNS: Add mgmt and production DNS for netmon2001

https://gerrit.wikimedia.org/r/364260

gerritbot added a comment.Jul 11 2017, 12:43 AM

Change 364353 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address for netmon2001

https://gerrit.wikimedia.org/r/364353

gerritbot added a comment.Jul 11 2017, 1:21 AM

Change 364353 merged by Dzahn:
[operations/puppet@production] DHCP: Add MAC address for netmon2001

https://gerrit.wikimedia.org/r/364353

Papaul added a comment.Jul 11 2017, 3:04 PM

@RobH can you please setup network port for netmon2001? Thanks

asw-d-codfw:ge-5/0/23

RobH closed this task as Resolved.Jul 11 2017, 4:09 PM
robh@asw-d-codfw# show | compare  
[edit interfaces interface-range vlan-public1-d-codfw]
     member ge-1/0/13 { ... }
+    member ge-5/0/23;
[edit interfaces]
+   ge-5/0/23 {
+       description netmon2001;
+       enable;
+   }

Done and merged live on switch stack.

RobH updated the task description. (Show Details)Jul 11 2017, 4:09 PM
Dzahn reopened this task as Open.Jul 11 2017, 5:07 PM

Thanks Papaul, Rob, i'm gonna reopen this and take it to continue with OS install and adding services.

Dzahn claimed this task.Jul 11 2017, 5:07 PM
Dzahn added a subscriber: Papaul.
Dzahn reassigned this task from Dzahn to Papaul.Jul 11 2017, 5:10 PM
RobH added a comment.Jul 11 2017, 5:11 PM

I did not mean to resolve the task, my bad!

Dzahn added a comment.Jul 11 2017, 5:12 PM

Do you know which partman recipe is the right one?

RobH claimed this task.Jul 11 2017, 5:17 PM

I'm going to claim this for install, so @Papaul can work on other onsite tasks =]

gerritbot added a comment.Jul 11 2017, 5:30 PM

Change 364478 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] settting netmon2001 install params

https://gerrit.wikimedia.org/r/364478

gerritbot added a comment.Jul 11 2017, 5:32 PM

Change 364478 merged by RobH:
[operations/puppet@production] settting netmon2001 install params

https://gerrit.wikimedia.org/r/364478

gerritbot added a comment.Jul 11 2017, 5:52 PM

Change 364481 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] correcting netmon2001 dns entries

https://gerrit.wikimedia.org/r/364481

gerritbot added a comment.Jul 11 2017, 5:54 PM

Change 364481 merged by RobH:
[operations/dns@master] correcting netmon2001 dns entries

https://gerrit.wikimedia.org/r/364481

gerritbot added a comment.Jul 11 2017, 5:58 PM

Change 364482 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting netmon2001 install params

https://gerrit.wikimedia.org/r/364482

gerritbot added a comment.Jul 11 2017, 6:10 PM

Change 364482 merged by RobH:
[operations/puppet@production] setting netmon2001 install params

https://gerrit.wikimedia.org/r/364482

gerritbot added a comment.Jul 11 2017, 6:30 PM

Change 364489 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] fixing ordering of servers

https://gerrit.wikimedia.org/r/364489

gerritbot added a comment.Jul 11 2017, 6:31 PM

Change 364489 merged by RobH:
[operations/puppet@production] fixing ordering of servers

https://gerrit.wikimedia.org/r/364489

RobH reassigned this task from RobH to Dzahn.Jul 11 2017, 7:37 PM
RobH updated the task description. (Show Details)
RobH removed projects: Patch-For-Review, ops-codfw.

Assigned to @Dzahn for service implemetnation. I've assumed it goes to him, since he is handling the stretch service updates for netmon1002.

gerritbot added a comment.Jul 11 2017, 9:07 PM

Change 364585 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add netmon2001 to site, equal to netmon1002

https://gerrit.wikimedia.org/r/364585

gerritbot added a project: Patch-For-Review.Jul 11 2017, 9:07 PM
gerritbot added a comment.Jul 11 2017, 10:18 PM

Change 364585 merged by Dzahn:
[operations/puppet@production] add netmon2001 to site, equal to netmon1002

https://gerrit.wikimedia.org/r/364585

gerritbot added a comment.Jul 11 2017, 10:26 PM

Change 364613 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] netmon: disable Letsencrypt on netmon2001

https://gerrit.wikimedia.org/r/364613

gerritbot added a comment.Jul 11 2017, 10:29 PM

Change 364613 merged by Dzahn:
[operations/puppet@production] netmon: disable Letsencrypt on netmon2001

https://gerrit.wikimedia.org/r/364613

gerritbot added a comment.Jul 11 2017, 11:32 PM

Change 364620 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] rancid: add rsync::quickdatacopy to sync /var/lib/rancid

https://gerrit.wikimedia.org/r/364620

gerritbot added a comment.Jul 11 2017, 11:34 PM

Change 364620 merged by Dzahn:
[operations/puppet@production] rancid: add rsync::quickdatacopy to sync /var/lib/rancid

https://gerrit.wikimedia.org/r/364620

gerritbot added a comment.Jul 12 2017, 12:04 AM

Change 364624 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] rancid: disable fully automatic rsyncing of app data

https://gerrit.wikimedia.org/r/364624

gerritbot added a comment.Jul 12 2017, 12:06 AM

Change 364624 merged by Dzahn:
[operations/puppet@production] rancid: disable fully automatic rsyncing of app data

https://gerrit.wikimedia.org/r/364624

gerritbot added a comment.Jul 12 2017, 1:19 AM

Change 364629 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] rancid/netmon: add active_server parameter to DC-switch

https://gerrit.wikimedia.org/r/364629

gerritbot added a comment.Jul 12 2017, 1:49 AM

Change 364629 merged by Dzahn:
[operations/puppet@production] rancid/netmon: add active_server parameter to DC-switch

https://gerrit.wikimedia.org/r/364629

gerritbot added a comment.Jul 12 2017, 3:03 AM

Change 364641 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add IPv6 records for netmon2001

https://gerrit.wikimedia.org/r/364641

gerritbot added a comment.Jul 12 2017, 3:41 AM

Change 364641 merged by Dzahn:
[operations/dns@master] add IPv6 records for netmon2001

https://gerrit.wikimedia.org/r/364641

Dzahn updated the task description. (Show Details)Jul 12 2017, 4:29 AM
Dzahn mentioned this in T159756: setup netmon1002.wikimedia.org.
Stashbot subscribed.Jul 14 2017, 4:21 AM

Mentioned in SAL (#wikimedia-operations) [2017-07-14T04:21:35Z] <mutante> netmon1002/netmon2001 - change UID/GID for rancid to universal 445/445, use find -exec to chown existing files, for unmessy data syncing, define UID on wikitech page UID (T166180)

gerritbot added a comment.Jul 18 2017, 3:05 AM

Change 365890 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cache::misc: add director for netmon2001

https://gerrit.wikimedia.org/r/365890

gerritbot added a comment.Jul 18 2017, 3:10 AM

Change 365890 merged by Dzahn:
[operations/puppet@production] cache::misc: add director for netmon2001

https://gerrit.wikimedia.org/r/365890

gerritbot added a comment.Jul 18 2017, 3:18 AM

Change 365892 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: switch backend from netmon1002 to netmon2001

https://gerrit.wikimedia.org/r/365892

gerritbot added a comment.Jul 18 2017, 3:32 AM

Change 365893 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: sync data to netmon2001, use quickdatacopy, in role

https://gerrit.wikimedia.org/r/365893

gerritbot added a comment.Jul 18 2017, 4:13 AM

Change 365893 merged by Dzahn:
[operations/puppet@production] smokeping: sync data to netmon2001, use quickdatacopy, in role

https://gerrit.wikimedia.org/r/365893

gerritbot added a comment.Jul 18 2017, 5:39 PM

Change 365892 merged by Dzahn:
[operations/puppet@production] smokeping: switch backend from netmon1002 to netmon2001

https://gerrit.wikimedia.org/r/365892

gerritbot added a comment.Jul 18 2017, 5:54 PM

Change 366012 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] rancid: switch active server to netmon2001

https://gerrit.wikimedia.org/r/366012

gerritbot added a comment.Jul 18 2017, 5:58 PM

Change 366012 merged by Dzahn:
[operations/puppet@production] rancid: switch active server to netmon2001

https://gerrit.wikimedia.org/r/366012

Dzahn mentioned this in T171018: decom netmon1001.Jul 19 2017, 12:48 AM
Dzahn added a comment.Jul 19 2017, 1:50 AM

netmon2001 is up and running.

netmon1002 and netmon2001 use identical roles:

1791 node /^netmon(1002|2001)\.wikimedia\.org$/ {
1792     role(network::monitor, librenms, rancid, smokeping)

smokeping is currently on netmon2001 (behind cache::misc, switch cache backend in Hiera)

rancid is on currently netmon2001 (switch controlled by "netmon_server" setting in common.yaml in Hiera)

librenms is currently on netmon1002 (switch in DNS)

Dzahn closed this task as Resolved.Jul 21 2017, 10:12 PM
Dzahn updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits