Setup automated topk wide row reporting
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	Eevans
	Oct 4 2016, 9:24 PM

Description

T94121 tracks the larger problem of making high growth revision histories more tractable. In the meantime, the remedy has been to reactively (manually) identify these partitions (typically after a query triggers an OOM), remove them, and (optionally) blacklist the title. Until a more permanent solution is in place, we should automate the aforementioned procedure, and provide early (earlier) warning of impending issues.

This issue proposes to create a script that periodically (weekly, tentatively) queries for compaction generated wide-row warnings in logstash, summarizes them, and publishes the results via email to the Services list.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	RESTBase-Cassandra: Add the topk reporter	operations/puppet	production	+773 -0
	logstash: let maintenance hosts connect to elasticsearch	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Eevans	T106619 investigate G1GC pause times
Duplicate	None	T94121 Understand and solve wide row issues for frequently edited and re-rendered pages
Declined	• mobrovac	T147366 Setup automated topk wide row reporting

Event Timeline

To summarize an IRC conversation with @bd808 and @greg, terbium would seem to be the best candidate to generate this report from.

Eevans moved this task from Backlog to In-Progress on the Cassandra board.Oct 4 2016, 9:26 PM

Eevans updated the task description. (Show Details)Oct 4 2016, 10:25 PM

Change 314772 had a related patch set uploaded (by Eevans):
Update firewall to allow terbium access to elasticsearch

https://gerrit.wikimedia.org/r/314772

gerritbot added a project: Patch-For-Review.Oct 7 2016, 9:44 PM

Change 314772 merged by Dzahn:
logstash: let maintenance hosts connect to elasticsearch

https://gerrit.wikimedia.org/r/314772

• GWicke added a project: Services.Oct 12 2016, 11:22 PM

A first-stab at this can be found here: https://github.com/eevans/services-adhoc-reports/blob/master/report-topk-partion-sizes. It's still a bit rough, but WFM; Review welcome.

I've sent one report out to services@wikimedia.org already (the result of a manual run on Friday), and I've since set it up to run via cron from terbium on Sunday evenings.

NOTE: This is currently running from a Git checkout in my home directory, and is kicked off by an entry in my crontab. This should be considered temporary; Long-term this isn't acceptable. Pending review/feedback this should be properly deployed, documented, and Puppetized.

Eevans moved this task from In-Progress to Blocked on the Cassandra board.Oct 18 2016, 11:59 PM

@Eevans should we close this or are you planning more work here?

In T147366#2882716, @mobrovac wrote:

@Eevans should we close this or are you planning more work here?

Well, there was some question as to our level of commitment to this; Ideally we'd solve the underlying issues, and save ourselves the noise, but that may still be a ways off.

Currently it's running out of my home directory and crontab on terbium, if it's something we plan to rely on for a while, then the code should probably be reviewed, moved over, and its invocation Puppetized.

@mobrovac wdyt?

In T147366#2882791, @Eevans wrote:

In T147366#2882716, @mobrovac wrote:

@Eevans should we close this or are you planning more work here?

Well, there was some question as to our level of commitment to this; Ideally we'd solve the underlying issues, and save ourselves the noise, but that may still be a ways off.

Currently it's running out of my home directory and crontab on terbium, if it's something we plan to rely on for a while, then the code should probably be reviewed, moved over, and its invocation Puppetized.

@mobrovac wdyt?