Page MenuHomePhabricator
Create Task
Maniphest T196484

rack/setup/install graphite1004
Closed, ResolvedPublic
Actions

Description

This task will track the receiving, racking, setup, and installation of graphite1004.

Racking Proposal: This new graphite host should be located in different rack from any existing graphite hosts in the site. graphite100[1-3] are located in C4, B1, & A3, do not place this host in those racks. Place it in any other 1G racks.

graphite1004:

  • - receive in system on procurement task T194862
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -4graphite: remove old graphite hardware from receiving metrics
operations/puppetproduction+5 -0statsd_proxy: add socket_receive_bufsize parameter
operations/debs/statsd-proxywmf_v0.0.10+41 -4add socket_bufsize option to make SO_RCVBUF tunable
operations/puppetproduction+4 -0graphite: backup graphite/django sqlite database
operations/puppetproduction+2 -0graphite: add interface::rps settings to graphite hosts
operations/puppetproduction+0 -2hieradata: remove old graphite hardware from graphite-web cluster
operations/puppetproduction+5 -0role: alias static path on graphite
operations/puppetproduction+2 -2Use graphite2003 when codfw graphite is enabled
operations/puppetproduction+2 -2Move read traffic to graphite1004
operations/puppetproduction+1 -0graphite: add graphite1004 to cluster_servers
operations/puppetproduction+1 -0graphite: add graphite1004 to cluster_servers
operations/puppetproduction+3 -3graphite: move alerting to graphite1004
operations/puppetproduction+4 -2graphite: add queue_depth and batch_size options to carbon-c-relay
operations/dnsmaster+1 -1update graphite-in to use graphite1004
operations/dnsmaster+3 -3lower TTL for graphite CNAMEs before failover
operations/dnsmaster+1 -1move statsd cname to graphite1004
operations/dnsmaster+4 -4Switch statsd/carbon to graphite1004
operations/dnsmaster+4 -4Shift carbon/statsd write traffic to graphite1004
operations/puppetproduction+1 -1diamond: send metrics to graphite1004
operations/puppetproduction+5 -0calico: allow statsd traffic to graphite1004
operations/puppetproduction+10 -0graphite: install dummy carbonate.conf
operations/puppetproduction+3 -0graphite: mirror metric traffic to graphite1004
operations/puppetproduction+1 -1graphite: fix graphite-manage stretch command
operations/puppetproduction+1 -1Assign graphite1004 its role
operations/puppetproduction+10 -0setting graphite1004 install params
operations/dnsmaster+87 -82Adding mgmt/production dns graphite1004
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedfgiunchediT196484 rack/setup/install graphite1004
 ResolvedayounsiT202846 Update ACLs for newer graphite hosts

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fgiunchedi added a project: observability.Jul 9 2018, 2:15 PM
fgiunchedi moved this task from Inbox to Externally blocked on the observability board.
fgiunchedi added a comment.Jul 23 2018, 3:41 PM

@Cmjohnson what's the status for graphite1004 ?

Cmjohnson added a comment.Jul 23 2018, 3:50 PM

@fgiunchedi I am currently working through 14 racking tasks...the CP's are the highest priority. I am not sure where the graphite falls in the priority list but I am working through the. I am set to do the dbxproxy servers next.

fgiunchedi added a comment.Jul 24 2018, 8:47 AM

Thanks for the update @Cmjohnson, not particularly urgent but it would be nice to have graphite1004 before the end of the quarter

gerritbot added a subscriber: gerritbot.Jul 31 2018, 7:57 PM

Change 449529 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt/production dns graphite1004

https://gerrit.wikimedia.org/r/449529

gerritbot added a project: Patch-For-Review.Jul 31 2018, 7:57 PM

Change 449529 merged by Cmjohnson:
[operations/dns@master] Adding mgmt/production dns graphite1004

https://gerrit.wikimedia.org/r/449529

Cmjohnson updated the task description. (Show Details)Jul 31 2018, 8:01 PM
gerritbot added a comment.Aug 1 2018, 1:07 PM

Change 449715 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding graphite1004 mgmt/production dns

https://gerrit.wikimedia.org/r/449715

Cmjohnson reassigned this task from Cmjohnson to RobH.Aug 2 2018, 7:19 PM
Cmjohnson updated the task description. (Show Details)
Cmjohnson moved this task from Racking Tasks to Blocked on the ops-eqiad board.

This servers is ready for install, assigning to @RobH for help with installation

gerritbot added a comment.Aug 3 2018, 6:13 PM

Change 450252 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting graphite1004 install params

https://gerrit.wikimedia.org/r/450252

gerritbot added a comment.Aug 3 2018, 6:16 PM

Change 450252 merged by RobH:
[operations/puppet@production] setting graphite1004 install params

https://gerrit.wikimedia.org/r/450252

RobH reassigned this task from RobH to fgiunchedi.Aug 3 2018, 6:27 PM
RobH removed projects: Patch-For-Review, ops-eqiad.

Please note I set this to role spare, since I wasn't sure if setting it to any other role may produce logging spam/traffic/alerts to the other graphite hosts. When in doubt, go for the smaller impact role choice before service implementation.

I've assigned this to @fgiunchedi for service implementation.

RobH updated the task description. (Show Details)Aug 3 2018, 6:27 PM
fgiunchedi added a project: User-fgiunchedi.Aug 6 2018, 2:13 PM

Thanks @RobH ! Yeah role spare makes sense in this case.

gerritbot added a comment.Aug 13 2018, 10:37 PM

Change 452576 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Assign graphite1004 its role

https://gerrit.wikimedia.org/r/452576

gerritbot added a project: Patch-For-Review.Aug 13 2018, 10:37 PM
gerritbot added a comment.Aug 13 2018, 10:40 PM

Change 452576 merged by Filippo Giunchedi:
[operations/puppet@production] Assign graphite1004 its role

https://gerrit.wikimedia.org/r/452576

gerritbot added a comment.Aug 13 2018, 10:59 PM

Change 452578 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: fix graphite-manage stretch command

https://gerrit.wikimedia.org/r/452578

gerritbot added a comment.Aug 13 2018, 10:59 PM

Change 452579 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: mirror metric traffic to graphite1004

https://gerrit.wikimedia.org/r/452579

gerritbot added a comment.Aug 13 2018, 11:03 PM

Change 452578 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: fix graphite-manage stretch command

https://gerrit.wikimedia.org/r/452578

gerritbot added a comment.Aug 13 2018, 11:22 PM

Change 452579 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: mirror metric traffic to graphite1004

https://gerrit.wikimedia.org/r/452579

Stashbot added a subscriber: Stashbot.Aug 13 2018, 11:24 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-13T23:24:33Z] <godog> restart carbon-c-relay on graphite1001 to mirror traffic to graphite1004 - T196484

fgiunchedi moved this task from Externally blocked to In progress on the observability board.Aug 20 2018, 3:25 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Aug 21 2018, 12:56 AM
Stashbot added a comment.Aug 22 2018, 4:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-22T16:58:26Z] <godog> start backfilling metrics from graphite1001 into graphite1004 - T196484

gerritbot added a comment.Aug 22 2018, 9:33 PM

Change 454695 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: install dummy carbonate.conf

https://gerrit.wikimedia.org/r/454695

gerritbot added a comment.Aug 22 2018, 9:41 PM

Change 454695 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: install dummy carbonate.conf

https://gerrit.wikimedia.org/r/454695

gerritbot added a comment.Aug 23 2018, 6:19 PM

Change 454872 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Shift carbon/statsd write traffic to graphite1004

https://gerrit.wikimedia.org/r/454872

gerritbot added a comment.Aug 23 2018, 6:24 PM

Change 454874 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] diamond: send metrics to graphite1004

https://gerrit.wikimedia.org/r/454874

gerritbot added a comment.Aug 23 2018, 6:24 PM

Change 454875 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: add graphite1004 to cluster_servers

https://gerrit.wikimedia.org/r/454875

gerritbot added a comment.Aug 23 2018, 6:24 PM

Change 454876 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] varnish: move to graphite1004

https://gerrit.wikimedia.org/r/454876

gerritbot added a comment.Aug 23 2018, 6:24 PM

Change 454877 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: move alerting to graphite1004

https://gerrit.wikimedia.org/r/454877

gerritbot added a comment.Aug 23 2018, 6:24 PM

Change 454878 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] calico: allow statsd traffic to graphite1004

https://gerrit.wikimedia.org/r/454878

gerritbot added a comment.Aug 24 2018, 7:37 AM

Change 454878 merged by Alexandros Kosiaris:
[operations/puppet@production] calico: allow statsd traffic to graphite1004

https://gerrit.wikimedia.org/r/454878

ayounsi closed subtask T202846: Update ACLs for newer graphite hosts as Resolved.Aug 27 2018, 7:36 PM
gerritbot added a comment.Aug 28 2018, 8:31 AM

Change 454874 merged by Filippo Giunchedi:
[operations/puppet@production] diamond: send metrics to graphite1004

https://gerrit.wikimedia.org/r/454874

gerritbot added a comment.Aug 28 2018, 10:38 AM

Change 454872 abandoned by Filippo Giunchedi:
Shift carbon/statsd write traffic to graphite1004

Reason:
Superceded

https://gerrit.wikimedia.org/r/454872

gerritbot added a comment.Aug 28 2018, 10:38 AM

Change 455808 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Switch statsd/carbon to graphite1004

https://gerrit.wikimedia.org/r/455808

gerritbot added a comment.Aug 29 2018, 9:02 AM

Change 455808 merged by Filippo Giunchedi:
[operations/dns@master] Switch statsd/carbon to graphite1004

https://gerrit.wikimedia.org/r/455808

Stashbot added a comment.Aug 29 2018, 9:04 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-29T09:04:43Z] <godog> switch statsd and carbon CNAMEs to graphite1004 - T196484

fgiunchedi added a comment.Aug 29 2018, 9:58 AM
In T196484#4541406, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2018-08-29T09:04:43Z] <godog> switch statsd and carbon CNAMEs to graphite1004 - T196484

This got reverted, due to the fact that carbon-c-relay frontend started dropping metrics. Stretch has a new version (2.5) compared to what we run on jessie (1.11) so we'll need to investigate what changed and if the newer version in buster (3.2) might help.

fgiunchedi moved this task from Doing to Up next on the User-fgiunchedi board.Oct 3 2018, 1:16 PM
fgiunchedi moved this task from In progress to Up next on the observability board.Oct 15 2018, 2:16 PM
fgiunchedi mentioned this in T207040: Graphite1001 disk usage at 96%.Oct 15 2018, 2:57 PM
gerritbot added a comment.Oct 17 2018, 3:50 PM

Change 468009 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/dns@master] move statsd cname to graphite1004

https://gerrit.wikimedia.org/r/468009

gerritbot added a comment.Oct 17 2018, 6:16 PM

Change 468009 merged by Cwhite:
[operations/dns@master] move statsd cname to graphite1004

https://gerrit.wikimedia.org/r/468009

fgiunchedi mentioned this in T101141: UDP rcvbuferrors and inerrors on graphite hosts.Oct 18 2018, 12:59 PM
gerritbot added a comment.Oct 18 2018, 6:57 PM

Change 468388 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] graphite: add interface::rps settings to graphite hosts

https://gerrit.wikimedia.org/r/468388

Stashbot added a comment.Oct 18 2018, 7:11 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-18T19:11:28Z] <shdubsh> upping ring buffer size on graphite1004 in an attempt to mitigate dropped packets at the interface -- T196484

Stashbot added a comment.Oct 19 2018, 7:05 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-19T07:05:58Z] <godog> bump /proc/sys/net/core/rmem_default temporarily to 1MB and bounce statsd-proxy statsite-instances on graphite1004 - T196484

Stashbot added a comment.Oct 19 2018, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-19T07:50:21Z] <godog> bump /proc/sys/net/core/rmem_default temporarily to 2MB and bounce statsd-proxy statsite-instances on graphite1004 - T196484

fgiunchedi added a comment.Oct 19 2018, 8:38 AM

Setting a 2MB socket receive buffer has helped getting errors down to ~0, unfortunately statsd-proxy nor statsite support setting SO_RCVBUF socket option via configuration, so I did this to temporarily set the buffer to 2MB and then back to its default:

echo 2097152 > /proc/sys/net/core/rmem_default
systemctl restart statsd-proxy statsite-instances
echo 212992 > /proc/sys/net/core/rmem_default

There's also ~1 packet/s dropped by the NIC which we'll need to tackle too

Stashbot added a comment.Oct 19 2018, 9:37 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-19T09:37:05Z] <godog> bump /proc/sys/net/core/rmem_default temporarily to 6MB and bounce statsd-proxy statsite-instances on graphite1004 - T196484

fgiunchedi added a comment.Oct 19 2018, 9:41 AM

Tried 6MB per thread now: we're ingesting about 30MB/s of udp traffic, with 4 statsd-proxy threads each should be able to buffer its share of bandwidth (7.5MB/s) for ~1s

fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Oct 19 2018, 3:13 PM
fgiunchedi moved this task from Up next to In progress on the observability board.Oct 29 2018, 1:59 PM
gerritbot added a comment.Oct 29 2018, 3:16 PM

Change 470410 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/dns@master] update graphite-in to use graphite1004

https://gerrit.wikimedia.org/r/470410

gerritbot added a comment.Oct 29 2018, 10:55 PM

Change 470512 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/debs/statsd-proxy@wmf_v0.0.10] add socket_bufsize option to make SO_RCVBUF tunable

https://gerrit.wikimedia.org/r/470512

gerritbot added a comment.Oct 30 2018, 9:43 AM

Change 470557 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] lower TTL for graphite CNAMEs before failover

https://gerrit.wikimedia.org/r/470557

gerritbot added a comment.Oct 30 2018, 9:44 AM

Change 470557 merged by Filippo Giunchedi:
[operations/dns@master] lower TTL for graphite CNAMEs before failover

https://gerrit.wikimedia.org/r/470557

gerritbot added a comment.Oct 30 2018, 4:22 PM

Change 470410 merged by Cwhite:
[operations/dns@master] update graphite-in to use graphite1004

https://gerrit.wikimedia.org/r/470410

Stashbot added a comment.Oct 30 2018, 4:27 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-30T16:27:11Z] <shdubsh> updated graphite-in cname to graphite1004 - T196484

gerritbot added a comment.Oct 30 2018, 7:08 PM

Change 470659 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] graphite: add queue_depth and batch_size options to carbon-c-relay

https://gerrit.wikimedia.org/r/470659

gerritbot added a comment.Oct 31 2018, 3:30 PM

Change 470659 merged by Cwhite:
[operations/puppet@production] graphite: add queue_depth and batch_size options to carbon-c-relay

https://gerrit.wikimedia.org/r/470659

gerritbot added a comment.Oct 31 2018, 5:07 PM

Change 454877 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: move alerting to graphite1004

https://gerrit.wikimedia.org/r/454877

gerritbot added a comment.Nov 5 2018, 1:55 PM

Change 468388 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: add interface::rps settings to graphite hosts

https://gerrit.wikimedia.org/r/468388

gerritbot added a comment.Nov 5 2018, 6:43 PM

Change 454875 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: add graphite1004 to cluster_servers

https://gerrit.wikimedia.org/r/454875

Stashbot added a comment.Nov 5 2018, 6:44 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-05T18:44:42Z] <godog> pool graphite1004 for reads - T196484

gerritbot added a comment.Nov 6 2018, 2:15 PM

Change 471965 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: add graphite1004 to cluster_servers

https://gerrit.wikimedia.org/r/471965

gerritbot added a comment.Nov 6 2018, 2:20 PM

Change 471965 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: add graphite1004 to cluster_servers

https://gerrit.wikimedia.org/r/471965

gerritbot added a comment.Nov 6 2018, 3:18 PM

Change 454876 merged by Filippo Giunchedi:
[operations/puppet@production] Move read traffic to graphite1004

https://gerrit.wikimedia.org/r/454876

gerritbot added a comment.Nov 6 2018, 3:44 PM

Change 471986 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: remove old graphite hardware from graphite-web cluster

https://gerrit.wikimedia.org/r/471986

gerritbot added a comment.Nov 6 2018, 3:44 PM

Change 471987 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: remove old graphite hardware from receiving metrics

https://gerrit.wikimedia.org/r/471987

gerritbot added a comment.Nov 6 2018, 3:44 PM

Change 471988 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Use graphite2003 when codfw graphite is enabled

https://gerrit.wikimedia.org/r/471988

gerritbot added a comment.Nov 6 2018, 3:55 PM

Change 471988 merged by Filippo Giunchedi:
[operations/puppet@production] Use graphite2003 when codfw graphite is enabled

https://gerrit.wikimedia.org/r/471988

gerritbot added a comment.Nov 6 2018, 4:07 PM

Change 471994 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] role: alias static path on graphite

https://gerrit.wikimedia.org/r/471994

fgiunchedi mentioned this in T166173: Upgrade graphite from 0.9.x to 1.x.Nov 6 2018, 4:18 PM
gerritbot added a comment.Nov 6 2018, 4:19 PM

Change 471994 merged by Cwhite:
[operations/puppet@production] role: alias static path on graphite

https://gerrit.wikimedia.org/r/471994

gerritbot added a comment.Nov 6 2018, 4:26 PM

Change 471986 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: remove old graphite hardware from graphite-web cluster

https://gerrit.wikimedia.org/r/471986

gerritbot added a comment.Nov 6 2018, 4:31 PM

Change 472002 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] graphite: backup graphite/django sqlite database

https://gerrit.wikimedia.org/r/472002

gerritbot added a comment.Nov 7 2018, 3:08 PM

Change 472002 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: backup graphite/django sqlite database

https://gerrit.wikimedia.org/r/472002

gerritbot added a comment.Nov 7 2018, 9:52 PM

Change 470512 merged by Cwhite:
[operations/debs/statsd-proxy@wmf_v0.0.10] add socket_bufsize option to make SO_RCVBUF tunable

https://gerrit.wikimedia.org/r/470512

gerritbot added a comment.Nov 8 2018, 4:09 PM

Change 472489 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] statsd_proxy: add socket_receive_bufsize parameter

https://gerrit.wikimedia.org/r/472489

gerritbot added a comment.Nov 8 2018, 5:48 PM

Change 472489 merged by Cwhite:
[operations/puppet@production] statsd_proxy: add socket_receive_bufsize parameter

https://gerrit.wikimedia.org/r/472489

gerritbot added a comment.Nov 13 2018, 10:33 AM

Change 471987 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: remove old graphite hardware from receiving metrics

https://gerrit.wikimedia.org/r/471987

fgiunchedi closed this task as Resolved.Nov 13 2018, 1:24 PM

graphite1004 is fully in service, followup for decom is T209357: Return graphite100[13] to spares pool (or decom)

fgiunchedi updated the task description. (Show Details)Nov 13 2018, 1:24 PM
CDanis mentioned this in T211747: Graphite generates a lot of 502 in Grafana.Dec 18 2018, 4:50 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL