Page MenuHomePhabricator
Create Task
Maniphest T315523

Put netmon2002 in service
Closed, ResolvedPublic
Actions

Description

Host was racked in T313867 and needs to be put in production.

Prerequisites:

  • Apply the role::netmon to the netmon2002 host #854624.
  • Set netmon2002 as a passive server host #854625
  • Confirm the netmon2002 host has DB privileges
    • Open firewall port for the LibreNMS database #854951
    • Connect to the LibreNMS database
  • Determine what (if any) data should to be carried over from netmon2001 to netmon2002 (Also relevant for T322695.)
  • Sync RRDs in /srv/librenms/rrd/
  • Sync rancid in /var/lib/rancid/core and /var/lib/rancid/GIT
  • Add the netmon2002 host to the alertmanager API rw #854974
  • Add the netmon2002 instance as a ganeti rapi node #865707
  • Add the netmon2002 as a LibreNMS scap deploy target #865705

codfw failover:

  • Set netmon2002 the main instance in codfw #865711

Post deployment validation:

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+4 -5netmon: Set netmon2002 the main instance in codfw
operations/puppetproduction+2 -0netmon: Add the netmon2002 instance as a ganeti rapi node.
operations/puppetproduction+1 -0netmon: Add the netmon2002 as a LibreNMS scap deploy target
operations/puppetproduction+6 -2netmon: Open LibreNMS port for netmon2002.
operations/puppetproduction+1 -0netmon: Add netmon2002 to the alertmanager rw api
operations/puppetproduction+1 -0netmon: Put the netmon2002 as passive server
operations/dnsmaster+2 -2Lower the TTL for netbox for the migration.
operations/puppetproduction+1 -12netmon: Add regex to match all the netmon instances.
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolvedandrea.denisseT315523 Put netmon2002 in service
 Resolvedandrea.denisseT322695 Decommission netmon2001
 Resolvedandrea.denisseT322321 Decommission netmon1002

Event Timeline

fgiunchedi created this task.Aug 18 2022, 7:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2022, 7:34 AM
andrea.denisse claimed this task.Nov 8 2022, 8:50 PM
andrea.denisse updated the task description. (Show Details)Nov 8 2022, 9:04 PM
andrea.denisse updated the task description. (Show Details)Nov 8 2022, 11:29 PM
andrea.denisse updated the task description. (Show Details)Nov 8 2022, 11:50 PM
andrea.denisse updated the task description. (Show Details)Nov 9 2022, 6:23 AM
gerritbot added a comment.Nov 9 2022, 8:04 AM

Change 854625 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] netmon: Put the netmon2002 as passive server

https://gerrit.wikimedia.org/r/854625

gerritbot added a project: Patch-For-Review.Nov 9 2022, 8:04 AM
gerritbot added a comment.Nov 9 2022, 8:40 AM

Change 854624 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] netmon: Add regex to match all the netmon instances.

https://gerrit.wikimedia.org/r/854624

gerritbot added a comment.Nov 9 2022, 9:01 AM

Change 854951 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] netmon: Open LibreNMS port for netmon2002.

https://gerrit.wikimedia.org/r/854951

andrea.denisse updated the task description. (Show Details)Nov 9 2022, 9:18 AM
andrea.denisse updated the task description. (Show Details)Nov 9 2022, 9:39 AM
andrea.denisse updated the task description. (Show Details)Nov 9 2022, 9:48 AM
gerritbot added a comment.Nov 9 2022, 9:49 AM

Change 854624 merged by Andrea Denisse:

[operations/puppet@production] netmon: Add regex to match all the netmon instances.

https://gerrit.wikimedia.org/r/854624

gerritbot added a comment.Nov 9 2022, 10:30 AM

Change 854974 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] netmon: Add netmon2002 to the alertmanager rw api

https://gerrit.wikimedia.org/r/854974

andrea.denisse updated the task description. (Show Details)Nov 9 2022, 10:35 AM
andrea.denisse updated the task description. (Show Details)Nov 12 2022, 8:28 PM
andrea.denisse updated the task description. (Show Details)Nov 12 2022, 11:51 PM
gerritbot added a comment.Nov 12 2022, 11:59 PM

Change 856065 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/dns@master] Lower the TTL for netbox for the migration.

https://gerrit.wikimedia.org/r/856065

andrea.denisse updated the task description. (Show Details)Nov 13 2022, 12:04 AM
Volans added a subscriber: Volans.Nov 14 2022, 9:35 AM

Lower LibreNMS and Netbox TTL in preparation for the new instance

@fgiunchedi @andrea.denisse What has Netbox to do with the netmon work? Is that a typo? I've commented on the related CR too.

andrea.denisse updated the task description. (Show Details)Nov 16 2022, 7:16 AM
gerritbot added a comment.Nov 16 2022, 9:00 PM

Change 856065 abandoned by Andrea Denisse:

[operations/dns@master] Lower the TTL for netbox for the migration.

Reason:

The patch is not required for the netmon2002 migration.

https://gerrit.wikimedia.org/r/856065

gerritbot added a comment.Nov 17 2022, 12:41 AM

Change 854625 merged by Andrea Denisse:

[operations/puppet@production] netmon: Put the netmon2002 as passive server

https://gerrit.wikimedia.org/r/854625

andrea.denisse added a subtask: T322321: Decommission netmon1002.Nov 17 2022, 8:46 PM
andrea.denisse updated the task description. (Show Details)Nov 28 2022, 8:52 PM
gerritbot added a comment.Nov 29 2022, 7:06 PM

Change 854974 merged by Andrea Denisse:

[operations/puppet@production] netmon: Add netmon2002 to the alertmanager rw api

https://gerrit.wikimedia.org/r/854974

lmata moved this task from Inbox to Up next on the SRE Observability (FY2022/2023-Q3) board.Nov 30 2022, 6:40 PM
gerritbot added a comment.Dec 2 2022, 2:48 PM

Change 854951 merged by Andrea Denisse:

[operations/puppet@production] netmon: Open LibreNMS port for netmon2002.

https://gerrit.wikimedia.org/r/854951

andrea.denisse updated the task description. (Show Details)Dec 2 2022, 2:49 PM
andrea.denisse updated the task description. (Show Details)
andrea.denisse updated the task description. (Show Details)Dec 7 2022, 4:25 PM
andrea.denisse updated the task description. (Show Details)Dec 7 2022, 4:30 PM
gerritbot added a comment.Dec 7 2022, 4:54 PM

Change 865705 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] netmon: Add the netmon2002 as a LibreNMS scap deploy target

https://gerrit.wikimedia.org/r/865705

gerritbot added a comment.Dec 7 2022, 5:00 PM

Change 865707 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] netmon: Add the netmon2002 instance as a ganeti rapi node.

https://gerrit.wikimedia.org/r/865707

gerritbot added a comment.Dec 7 2022, 5:14 PM

Change 865711 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] netmon: Set netmon2002 the main instance in codfw

https://gerrit.wikimedia.org/r/865711

gerritbot added a comment.Dec 7 2022, 5:40 PM

Change 865705 merged by Andrea Denisse:

[operations/puppet@production] netmon: Add the netmon2002 as a LibreNMS scap deploy target

https://gerrit.wikimedia.org/r/865705

gerritbot added a comment.Dec 8 2022, 5:20 AM

Change 865707 merged by Andrea Denisse:

[operations/puppet@production] netmon: Add the netmon2002 instance as a ganeti rapi node.

https://gerrit.wikimedia.org/r/865707

gerritbot added a comment.Dec 8 2022, 5:25 AM

Change 865711 merged by Andrea Denisse:

[operations/puppet@production] netmon: Set netmon2002 the main instance in codfw

https://gerrit.wikimedia.org/r/865711

andrea.denisse updated the task description. (Show Details)Dec 8 2022, 8:05 PM
andrea.denisse updated the task description. (Show Details)
andrea.denisse closed subtask T322321: Decommission netmon1002 as Resolved.Dec 15 2022, 11:14 PM
andrea.denisse closed this task as Resolved.Dec 15 2022, 11:38 PM
andrea.denisse updated the task description. (Show Details)
andrea.denisse closed subtask T322695: Decommission netmon2001 as Resolved.Dec 16 2022, 12:07 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL