Move coal from graphite#001 nodes to webperf#001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Mar 1 2017, 5:56 PM

Description

At the moment coal runs on graphite machines, though this is not strictly needed and they can run on a dedicated machine (baremetal or VM)

Weed out coal puppet module dependencies from graphite module
Not done: ~~Backup /var/lib/coal (or /var/lib/carbon/whisper) in bacula.~~
Migrate coal's retention config to graphite in puppet. – https://gerrit.wikimedia.org/r/427945
Update coal to write to graphite1001 via its line protocol instead of to disk directly. – https://gerrit.wikimedia.org/r/427664
Move coal processor from graphite1001/graphite2001 to webperf1001/webperf2001 - https://gerrit.wikimedia.org/r/#/c/429252
Move coal web from graphite1001/graphite2001 to webperfx001
Remove coal-web away from graphitex001

Details

Subject	Repo	Branch	Lines +/-
performance website: remove from graphite hosts	operations/puppet	production	+2 -4
coal: don't deploy to graphite	performance/coal	master	+3 -4
performance website: allow traffic	operations/puppet	production	+7 -0
performance.wikimedia.org: serve from webperfX001	operations/puppet	production	+5 -1
coal-web: needs proxy_http module as well	operations/puppet	production	+1 -1
performance::site: load apache mod proxy	operations/puppet	production	+1 -1
performance::site: require libapache2-mod-uwsgi	operations/puppet	production	+3 -0
coal: require requests module; deploy to webperf	operations/puppet	production	+10 -2
coal: require requests module; deploy to webperf	operations/puppet	production	+2 -1
coal_web: Use the graphite API to fetch data instead of whisper	performance/coal	master	+65 -16
coal: scap config is new to me...	performance/coal	master	+3 -1
coal: deploy to webperf machines as well	performance/coal	master	+5 -2
coal: incorporate etcd to test for master datacenter	performance/coal	master	+83 -2
coal: convert to using graphite instead of writing to whisper	performance/coal	master	+113 -60

Related Objects
Search...

Status	Assigned	Task
Resolved	• dpifke	T158837 Consolidate performance website and related software
Resolved	• Imarlier	T159354 Move coal from graphite#001 nodes to webperf#001
Resolved	Dzahn	T194283 Clean up graphite1001.eqiad.wmnet, now that coal has been moved

Event Timeline

fgiunchedi created this task.Mar 1 2017, 5:56 PM

Ottomata assigned this task to Krinkle.Mar 6 2017, 6:55 PM

Ottomata triaged this task as Medium priority.

Krinkle mentioned this in T158837: Consolidate performance website and related software.Mar 6 2017, 7:37 PM

• Gilles moved this task from Inbox, needs triage to Backlog: Maintenance, non-prioritized on the Performance-Team board.Mar 9 2017, 9:55 PM

fgiunchedi updated the task description. (Show Details)Mar 27 2017, 10:40 AM

fgiunchedi mentioned this in T123728: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie).Mar 27 2017, 5:19 PM

The current thinking at T158837 might result in the removal of the coal and coal-web services. Shelving this task for now.

Krinkle added a subtask: T175087: Create a navtiming processor for Prometheus.Sep 12 2017, 9:46 AM

Krinkle moved this task from Backlog: Maintenance, non-prioritized to Blocked (old) on the Performance-Team board.

Change 427664 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[performance/coal@master] coal: convert to using graphite instead of writing to whisper

https://gerrit.wikimedia.org/r/427664

gerritbot added a project: Patch-For-Review.Apr 19 2018, 1:53 PM

@fgiunchedi Mentioned in #wikimedia-perf that he thought he remembered there being a reason why submitting metrics via graphite wouldn't work.

First, here's a demonstration that it does. Red lines are several different coal metrics, still being written directly to whisper files by coal, just as they have been. Blue lines are those same metrics, being written by a second coal process to a separate key, via graphite (coal2.* instead of coal.*):

As you can see, they line up perfectly. Manual examination of the whisper files shows that the same values are being written by both processes, at the same timestamps.

This works because of two changes that were made during the recent refactors:

We only commit an offset in Kafka once all 5 5-minute periods that include that offset have been processed and submitted.
We align data boundaries to the 0-second mark (eg, each data boundary has a timestamp such that ts % 60 == 0)

Neither of these were previously true. ZeroMQ, the message broker that was being used, doesn't retain data. And the prior incarnation of coal aligned its data boundaries based on startup time (and restarted several dozen times a day due to various ZeroMQ timeouts).

Carbon, meanwhile, does align its data boundaries to the same %60 that we're using at the moment, which means we're submitting data with a timestamp that matches the timestamp that carbon is going to write. (Not true previously, when the timestamp would have been adjusted by carbon to the appropriate boundary). Additionally, carbon will write the last value received for any given period, and the prior version of coal could have sent mulitple, different values for a given metric within the same window (especially if multiple instances were running). This is no longer the case.

Thanks @Imarlier for the explanation and insight! Makes sense to me, the other thing I suggest checking is coal's whisper files aggregation/retetion periods vs graphite's defaults (modules/role/manifests/graphite/base.pp) and see if that needs adjusting when new coal whisper files are created.

Change 427664 merged by jenkins-bot:
[performance/coal@master] coal: convert to using graphite instead of writing to whisper

https://gerrit.wikimedia.org/r/427664

Krinkle changed the task status from Stalled to Open.Apr 23 2018, 4:40 PM

Krinkle removed a subtask: T175087: Create a navtiming processor for Prometheus.

Krinkle moved this task from Blocked (old) to To-do: Goals, prioritized next 4 Quarters on the Performance-Team board.

Change 427958 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[performance/coal@master] coal: incorporate etcd to test for master datacenter

https://gerrit.wikimedia.org/r/427958

Krinkle renamed this task from Move coal from graphite machine(s) to Move coal from graphite#001 nodes to webperf#001.Apr 24 2018, 4:16 PM

Krinkle raised the priority of this task from Low to Medium.

Krinkle updated the task description. (Show Details)

Krinkle updated the task description. (Show Details)Apr 24 2018, 4:19 PM

Change 427958 merged by jenkins-bot:
[performance/coal@master] coal: incorporate etcd to test for master datacenter

https://gerrit.wikimedia.org/r/427958

@Imarlier I landed it as-is.

Nevermind about using the /etc/wikimedia-cluster file (puppet), usage). Passing it as parameter from Puppet is probably as good or better that using that file. They both come from the same source and other services use the ::realm variable for this purpose as well. In Git, the only use of the /etc/ file is actually MediaWiki PHP, which is that way because it can't easily get values from Puppet. But for pretty much anything else, the parameter works well.

• Imarlier claimed this task.Apr 30 2018, 7:46 PM

• Imarlier moved this task from To-do: Goals, prioritized next 4 Quarters to Doing (old) on the Performance-Team board.

• Imarlier updated the task description. (Show Details)May 2 2018, 6:33 PM

Change 430601 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[performance/coal@master] coal: deploy to webperf machines as well

https://gerrit.wikimedia.org/r/430601

Change 430601 merged by jenkins-bot:
[performance/coal@master] coal: deploy to webperf machines as well

https://gerrit.wikimedia.org/r/430601

Change 430605 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[performance/coal@master] coal: scap config is new to me...

https://gerrit.wikimedia.org/r/430605

Change 430605 merged by jenkins-bot:
[performance/coal@master] coal: scap config is new to me...

https://gerrit.wikimedia.org/r/430605

Change 430625 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[performance/coal@master] coal_web: Use the graphite API to fetch data instead of whisper

https://gerrit.wikimedia.org/r/430625

Krinkle mentioned this in T193570: Why did first paint/fully loaded time drop (in a good way!) on mobile?.May 3 2018, 8:41 PM

Change 430625 merged by jenkins-bot:
[performance/coal@master] coal_web: Use the graphite API to fetch data instead of whisper

https://gerrit.wikimedia.org/r/430625

Change 431583 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] coal: require requests module; deploy to webperf

https://gerrit.wikimedia.org/r/431583

Ops - would appreciate a merge on the patch above ^

Change 431583 merged by Alexandros Kosiaris:
[operations/puppet@production] coal: require requests module; deploy to webperf

https://gerrit.wikimedia.org/r/431583

Change 431615 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] coal: require requests module; deploy to webperf

https://gerrit.wikimedia.org/r/431615

Change 431615 merged by Dzahn:
[operations/puppet@production] coal: require requests module; deploy to webperf

https://gerrit.wikimedia.org/r/431615

Change 431636 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] performance::site: require libapache2-mod-uwsgi

https://gerrit.wikimedia.org/r/431636

Change 431636 merged by Dzahn:
[operations/puppet@production] performance::site: require libapache2-mod-uwsgi

https://gerrit.wikimedia.org/r/431636

Change 431638 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] performance::site: load apache mod proxy

https://gerrit.wikimedia.org/r/431638

Change 431638 merged by Dzahn:
[operations/puppet@production] performance::site: load apache mod proxy

https://gerrit.wikimedia.org/r/431638

Change 431644 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] coal-web: needs proxy_http module as well

https://gerrit.wikimedia.org/r/431644

Change 431644 merged by Dzahn:
[operations/puppet@production] coal-web: needs proxy_http module as well

https://gerrit.wikimedia.org/r/431644

Change 431659 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] performance.wikimedia.org: serve from webperfX001

https://gerrit.wikimedia.org/r/431659

Change 431659 merged by Dzahn:
[operations/puppet@production] performance.wikimedia.org: serve from webperfX001

https://gerrit.wikimedia.org/r/431659

Change 431779 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] performance website: allow traffic

https://gerrit.wikimedia.org/r/431779

Change 431779 merged by Dzahn:
[operations/puppet@production] performance website: allow traffic

https://gerrit.wikimedia.org/r/431779

Change 431792 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[operations/puppet@production] performance website: remove from graphite hosts

https://gerrit.wikimedia.org/r/431792

• Imarlier updated the task description. (Show Details)May 8 2018, 6:50 PM

Change 431792 merged by Dzahn:
[operations/puppet@production] performance website: remove from graphite hosts

https://gerrit.wikimedia.org/r/431792

Change 432095 had a related patch set uploaded (by Imarlier; owner: Imarlier):
[performance/coal@master] coal: don't deploy to graphite

https://gerrit.wikimedia.org/r/432095

• Imarlier added a subtask: T194283: Clean up graphite1001.eqiad.wmnet, now that coal has been moved.May 9 2018, 3:46 PM

Change 432095 merged by jenkins-bot:
[performance/coal@master] coal: don't deploy to graphite

https://gerrit.wikimedia.org/r/432095

Dzahn closed subtask T194283: Clean up graphite1001.eqiad.wmnet, now that coal has been moved as Resolved.May 9 2018, 7:59 PM

• Imarlier closed this task as Resolved.May 9 2018, 8:00 PM

Krinkle mentioned this in T175087: Create a navtiming processor for Prometheus.May 10 2018, 12:50 PM

Krinkle updated the task description. (Show Details)

Also ticking off the "backup files in bacula" checkbox, because we now use the regular carbon storage, which at some point between between 2015 and now has been added to the backup process (despite being considerably larger than the subset of coal metrics).

Resource: puppet:/modules/profile/manifests/backup/director.pp#L145

In T159354#4197075, @Krinkle wrote:

Also ticking off the "backup files in bacula" checkbox, because we now use the regular carbon storage, which at some point between between 2015 and now has been added to the backup process (despite being considerably larger than the subset of coal metrics).

Resource: puppet:/modules/profile/manifests/backup/director.pp#L145

Carbon files are not backed up in bacula (too many files). The part in puppet is the fileset declaration, said fileset is never actually applied/used to graphite role with backup::set and thus never backed up.