Page MenuHomePhabricator
Create Task
Maniphest T85451

scale graphite deployment (tracking)
Closed, InvalidPublic
Actions

Description

tracking graphite scaling work using this ticket, see also https://wikitech.wikimedia.org/wiki/Graphite/Scaling for a general overview of the options and http://etherpad.wikimedia.org/p/graphitetodo for a scratchpad/ramblings.

currently on the plate:

future plans:

  • expand the setup beyond 1+1 machine using graphite clustering
    • route/cluster metrics with carbon-c-relay and carbon consistent hashing
    • use carbonate to rebalance metric data around also using the same consistent hashing
  • consider hybrid caching with bcache (i.e. spinning disks + block caching on ssd)

Details

SubjectRepoBranchLines +/-
graphite: add realserver classoperations/puppetproduction+8 -0
svc: add graphite LVS addressesoperations/dnsmaster+4 -0
lvs: add graphite serviceoperations/puppetproduction+63 -0
graphite: introduce local carbon-c-relay daemonoperations/puppetproduction+64 -0
graphite: add 'big_users' route and clusteroperations/puppetproduction+26 -14
graphite: add cluster_servers graphite-web settingoperations/puppetproduction+11 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Dec 29 2014, 2:23 PM
fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added projects: acl*sre-team, Grafana.
fgiunchedi subscribed.
fgiunchedi updated the task description. (Show Details)Dec 29 2014, 2:27 PM
fgiunchedi set Security to None.
fgiunchedi claimed this task.Dec 29 2014, 2:38 PM
mark subscribed.Jan 5 2015, 11:37 AM

Let's update this ticket with sub tasks / blockers for the tasks mentioned below. Some of them are in old RT links of course, but let's change that into the newer Phab tickets...

fgiunchedi added a comment.Jan 6 2015, 9:32 AM

yep, created the subtasks now to migrate to new hardware

fgiunchedi triaged this task as High priority.Jan 12 2015, 4:46 PM
fgiunchedi closed subtask T85907: acquire graphite hardware in codfw and eqiad as Resolved.Feb 24 2015, 3:34 PM
fgiunchedi closed subtask T85909: migrate graphite to new hardware as Resolved.Feb 24 2015, 3:39 PM
fgiunchedi reopened subtask T90111: replace txstatsd as Open.Mar 25 2015, 11:10 AM
• GWicke mentioned this in T78514: Detailed cassandra monitoring: metrics and dashboards done, need to set up alerts.Mar 25 2015, 8:24 PM
• GWicke subscribed.
fgiunchedi closed subtask T90111: replace txstatsd as Resolved.Apr 16 2015, 8:49 AM
fgiunchedi mentioned this in T89857: scale statsd reporting/aggregation (plan).Apr 16 2015, 8:54 AM
fgiunchedi changed the status of subtask T89857: scale statsd reporting/aggregation (plan) from Open to Stalled.
• GWicke added a comment.Apr 17 2015, 8:15 PM

@fgiunchedi, what is the plan for increasing SSD space? If we plan to continue using a single graphite node per DC, can we order larger SSDs and replace the current ones to have some headroom for more metrics?

• GWicke mentioned this in T95703: enable statsd extended counters.Apr 17 2015, 11:18 PM
RobH subscribed.EditedApr 20 2015, 3:14 PM

Host uranlum has two SSDs installed, and the R610 planform can usually support 4 2.5" disks installed.

If you guys wanted to upgrade/add/replace the disks, please create a hardware-request task linked to this, requesting such.

Thanks.

RobH added a comment.Apr 20 2015, 3:23 PM

So, I looked at uranium, not graphite1001. graphite1001 is an R420 with 4 disks already installed.

So this host actually CANNOT add additional SSDs without removing existing disks.

robh@graphite1001:~$ sudo lshw -class disk

*-disk                  
     description: ATA Disk
     product: INTEL SSDSC2BB60
     physical id: 0.0.0
     bus info: scsi@0:0.0.0
     logical name: /dev/sda
     version: D201
     serial: PHWL4316007M600TGN
     size: 558GiB (600GB)
     capabilities: partitioned partitioned:dos
     configuration: ansiversion=5 sectorsize=4096 signature=0004e927
*-disk
     description: ATA Disk
     product: INTEL SSDSC2BB60
     physical id: 0.0.0
     bus info: scsi@1:0.0.0
     logical name: /dev/sdb
     version: D201
     serial: PHWL43160035600TGN
     size: 558GiB (600GB)
     capabilities: partitioned partitioned:dos
     configuration: ansiversion=5 sectorsize=4096 signature=000559e0
*-disk
     description: ATA Disk
     product: INTEL SSDSC2BB60
     physical id: 0.0.0
     bus info: scsi@2:0.0.0
     logical name: /dev/sdc
     version: D201
     serial: PHWL43160086600TGN
     size: 558GiB (600GB)
     capabilities: partitioned partitioned:dos
     configuration: ansiversion=5 sectorsize=4096 signature=0005b39f
*-disk
     description: ATA Disk
     product: INTEL SSDSC2BB60
     physical id: 0.0.0
     bus info: scsi@3:0.0.0
     logical name: /dev/sdd
     version: D201
     serial: PHWL4316001S600TGN
     size: 558GiB (600GB)
     capabilities: partitioned partitioned:dos
     configuration: ansiversion=5 sectorsize=4096 signature=00060dff
fgiunchedi closed subtask T90591: backfill metrics from tungsten to graphite1001 as Resolved.Apr 27 2015, 10:26 AM
• GWicke added a comment.May 4 2015, 9:12 PM

So, should we

a) get a new box with more / bigger SSDs (most 2.5" cases have space for 8 SSDs), or
b) replace the existing SSDs with bigger ones?

fgiunchedi closed subtask T85908: replicate metric traffic in eqiad and codfw as Resolved.Jun 16 2015, 1:15 AM
fgiunchedi mentioned this in T113733: column family cassandra metrics size.Sep 30 2015, 3:01 PM
• GWicke added a comment.Sep 30 2015, 10:19 PM

@fgiunchedi, sorry if I'm naggy on this, but I'm still wondering what the plan is for scaling graphite storage, especially with codfw and new services adding even more metrics.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptSep 30 2015, 10:19 PM
• MZMcBride subscribed.Oct 7 2015, 5:38 AM
Addshore subscribed.Nov 13 2015, 9:43 PM
• GWicke added a subscriber: EBernhardson.Jan 27 2016, 12:51 AM
• GWicke added a project: Blocked-on-Operations.Jan 27 2016, 1:35 AM
• GWicke added a comment.EditedJan 27 2016, 1:49 AM

FYI, there are some upcoming changes in Services that will use more disk space for metrics:

  • We are about to split RESTBase metrics by request type (internal, internal update, external), which will triple the number of metrics produced by RESTBase.
  • The ongoing conversion of Cassandra nodes to a multi-instance setup will increase the number of Cassandra metrics by a factor of 3-5.
• GWicke added a project: Services.Jan 27 2016, 6:32 PM
Addshore added a comment.Feb 1 2016, 5:27 PM
In T85451#1258826, @GWicke wrote:

So, should we

a) get a new box with more / bigger SSDs (most 2.5" cases have space for 8 SSDs), or
b) replace the existing SSDs with bigger ones?

I support something along these lines happening!

Addshore awarded a token.Feb 1 2016, 5:27 PM
Addshore mentioned this in T125408: Regularly & Automatically backup WMDE metrics stored in graphite.Feb 1 2016, 5:43 PM
EBernhardson added a comment.Feb 1 2016, 6:11 PM

I also support this change, collecting stats in graphite has shown to be quite powerful. Removing limitations around stats collection should be quite helpful.

Addshore added a project: WMDE-Analytics-Engineering.Feb 3 2016, 12:45 PM
Addshore moved this task from Incoming to Watching / External on the WMDE-Analytics-Engineering board.
intracer subscribed.Feb 24 2016, 9:16 AM
gerritbot subscribed.Mar 15 2016, 10:54 AM

Change 277490 had a related patch set uploaded (by Filippo Giunchedi):
graphite: add 'big_users' route and cluster

https://gerrit.wikimedia.org/r/277490

gerritbot added a project: Patch-For-Review.Mar 15 2016, 10:54 AM
RobH changed the status of subtask T126253: additional graphite machines request, 1x per DC from Open to Stalled.Mar 24 2016, 8:25 PM
gerritbot added a comment.Apr 5 2016, 11:03 AM

Change 281631 had a related patch set uploaded (by Filippo Giunchedi):
graphite: add cluster_servers graphite-web setting

https://gerrit.wikimedia.org/r/281631

• GWicke mentioned this in T132771: Cleanup Graphite Cassandra metrics.Apr 15 2016, 5:42 PM
RobH closed subtask T126253: additional graphite machines request, 1x per DC as Resolved.Apr 21 2016, 7:10 PM
• GWicke added a parent task: T134016: RESTBase Cassandra cluster: Increase instance count to 3.May 4 2016, 3:13 PM
Danny_B added a project: Tracking-Neverending.May 5 2016, 7:12 PM
fgiunchedi created subtask T134889: put additional graphite machines in service.May 10 2016, 2:45 PM
gerritbot added a comment.May 10 2016, 3:13 PM

Change 281631 merged by Filippo Giunchedi:
graphite: add cluster_servers graphite-web setting

https://gerrit.wikimedia.org/r/281631

fgiunchedi mentioned this in rOPUP5d59ef23fc34: graphite: add cluster_servers graphite-web setting.May 10 2016, 3:16 PM
gerritbot added a comment.May 12 2016, 10:03 AM

Change 277490 merged by Filippo Giunchedi:
graphite: add 'big_users' route and cluster

https://gerrit.wikimedia.org/r/277490

fgiunchedi mentioned this in rOPUP590524fadb93: graphite: add 'big_users' route and cluster.May 12 2016, 10:05 AM
gerritbot added a comment.May 18 2016, 4:02 PM

Change 289440 had a related patch set uploaded (by Filippo Giunchedi):
graphite: introduce local carbon-c-relay daemon

https://gerrit.wikimedia.org/r/289440

fgiunchedi added a comment.May 18 2016, 4:38 PM

the patch at https://gerrit.wikimedia.org/r/289440 add a local carbon-c-relay to be used for submitting graphite metrics on localhost. The local daemon provides load-balancing, failover and batching across graphite "frontends" in eqiad (graphite1001 and graphite1003 ATM).

Going forward the plan would be to add codfw cluster with graphite machines (graphite2001 and graphite2002) and turn off the current mirroring eqiad -> codfw that's happening on graphite1001.
This would simplify datacenter failover for graphite (not statsd) considerably since each machine would push metrics to both datacenters, transparently to clients which always write to localhost:2003.

gerritbot added a comment.May 19 2016, 11:45 AM

Change 289635 had a related patch set uploaded (by Filippo Giunchedi):
svc: add graphite LVS addresses

https://gerrit.wikimedia.org/r/289635

gerritbot added a comment.May 19 2016, 11:48 AM

Change 289636 had a related patch set uploaded (by Filippo Giunchedi):
lvs: add graphite service

https://gerrit.wikimedia.org/r/289636

gerritbot added a comment.May 19 2016, 11:48 AM

Change 289637 had a related patch set uploaded (by Filippo Giunchedi):
graphite: add realserver class

https://gerrit.wikimedia.org/r/289637

fgiunchedi added a comment.May 19 2016, 1:43 PM

the next series of patches uses LVS instead to perform load-balancing among graphite machines, clients would need to be pointed to graphite.svc.<site>.wmnet and each site mirrors to the other

gerritbot added a comment.May 27 2016, 1:06 PM

Change 289440 abandoned by Filippo Giunchedi:
graphite: introduce local carbon-c-relay daemon

Reason:
abandoning in favor of LVS in Ibac0711dc

https://gerrit.wikimedia.org/r/289440

fgiunchedi mentioned this in rOPUPe40bc4a061b9: lvs: add graphite service.Jun 17 2016, 6:08 PM
fgiunchedi mentioned this in rOPUP306d29c1b528: graphite: add realserver class.
fgiunchedi mentioned this in rOPUP716a15ea9b32: graphite: introduce local carbon-c-relay daemon.
fgiunchedi mentioned this in rOPUP237da99e812d: graphite: add cluster_servers graphite-web setting.
fgiunchedi mentioned this in rOPUP484862c2475c: graphite: add 'big_users' route and cluster.
• Pchelolo moved this task from Backlog to watching on the Services board.Oct 12 2016, 6:49 PM
• Pchelolo edited projects, added Services (watching); removed Services.
fgiunchedi closed subtask T134889: put additional graphite machines in service as Resolved.Mar 1 2017, 2:28 PM
Addshore unsubscribed.Mar 1 2017, 4:59 PM
Dzahn subscribed.May 1 2017, 5:25 PM

graphite1003 is alerting in Icinga because of disk space 99% used on /var/lib/carbon

but because that's 1.6T , 99% means still 20G left.

/dev/mapper/graphite1003--vg-carbon 1.6T 1.6T 20G 99% /var/lib/carbon

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=graphite1003&service=Disk+space

fgiunchedi added a subscriber: Eevans.May 2 2017, 8:43 AM

@Dzahn indeed, and there's ~500G left on the vg still. I'll debug this with @Eevans but I suspect it is related to T163936: Latency metrics missing

Volans removed a project: Blocked-on-Operations.Dec 18 2017, 10:47 AM
gerritbot added a comment.Mar 2 2018, 2:59 PM

Change 289636 abandoned by Filippo Giunchedi:
lvs: add graphite service

Reason:
Not needed

https://gerrit.wikimedia.org/r/289636

gerritbot added a comment.Nov 16 2018, 10:29 AM

Change 289635 abandoned by Filippo Giunchedi:
svc: add graphite LVS addresses

Reason:
Not doing this for now

https://gerrit.wikimedia.org/r/289635

gerritbot added a comment.Nov 16 2018, 10:29 AM

Change 289637 abandoned by Filippo Giunchedi:
graphite: add realserver class

Reason:
Not doing this for now

https://gerrit.wikimedia.org/r/289637

• mobrovac added a project: Platform Team Legacy (Watching / External).Dec 20 2018, 12:02 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 7:57 PM
fgiunchedi closed this task as Invalid.Aug 13 2019, 1:07 PM
fgiunchedi closed subtask T89857: scale statsd reporting/aggregation (plan) as Invalid.

We're sunsetting Graphite (e.g. T228380: Tech debt: sunsetting of Graphite (part 1) ) so resolving as invalid

fgiunchedi closed subtask T86316: graphite clustering plan as Declined.Mar 2 2020, 7:58 AM
fgiunchedi closed subtask T102575: document graphite failover/backfill procedures as Resolved.Oct 22 2021, 12:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL