Services 2017/18 Q1 goal: Start gradual roll-out of Cassandra 3 & new schema to resolve storage scaling issues and OOM errors.
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• GWicke
	Jul 6 2017, 10:42 PM

Description

Annual plan:

Program 7: Smart tools for better data
Outcome 2: access to Wikimedia content and data with scalable APIs
Objective 1: Revision storage scaling

Objective for this quarter

Start gradual roll-out of Cassandra 3 & new schema to resolve storage scaling issues and OOM errors.

Details

	Subject	Repo	Branch	Lines +/-
	restbase-ng: Collect type=Table (not ColumnFamily)	operations/puppet	production	+21 -21

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Eevans	T169936 Services 2017/18 Q1 goal: Start gradual roll-out of Cassandra 3 & new schema to resolve storage scaling issues and OOM errors.
Resolved	Eevans	T169938 End of July milestone: Initial implementation & start of testing
Resolved	• Pchelolo	T170901 Expose default table TTL
Resolved	• Pchelolo	T170902 Don't add the TID by default in storage module
Resolved	Eevans	T171104 Reinitialize dev cluster on new hardware (restbase-dev100[4-6])
Resolved	Eevans	T169939 End of August milestone: Cassandra 3 cluster in production
Declined	Eevans	T172149 Consider a lower virtual node count
Resolved	fgiunchedi	T174392 Disk errors: restbase1010.eqiad.wmnet
Resolved	Eevans	T169940 End of September milestone: Start migration of production use cases.
Resolved	• mobrovac	T172224 Consider partitioning local_group_wikipedia even more
Resolved	Eevans	T172384 OOM exceptions in dev environment
Resolved	Eevans	T171772 Prometheus metrics storage for RESTBase dev environment
Resolved	fgiunchedi	T173490 Provision prometheus instance for cassandra/services metrics collection
Resolved	Eevans	T175284 Create parent directories for JBOD data_directories and e.g. commitlog directories

Event Timeline

• GWicke created this task.Jul 6 2017, 10:42 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 6 2017, 10:42 PM

• GWicke created subtask T169938: End of July milestone: Initial implementation & start of testing.Jul 6 2017, 10:47 PM

• GWicke created subtask T169939: End of August milestone: Cassandra 3 cluster in production.

• GWicke created subtask T169940: End of September milestone: Start migration of production use cases..Jul 6 2017, 10:52 PM

• GWicke changed the status of subtask T169939: End of August milestone: Cassandra 3 cluster in production from Open to Stalled.

Eevans triaged this task as Medium priority.Jul 31 2017, 7:15 PM

Eevans closed subtask T169938: End of July milestone: Initial implementation & start of testing as Resolved.Jul 31 2017, 7:44 PM

Eevans changed the status of subtask T169939: End of August milestone: Cassandra 3 cluster in production from Stalled to Open.Jul 31 2017, 7:55 PM

• mobrovac added a subtask: T172224: Consider partitioning local_group_wikipedia even more.Aug 1 2017, 9:21 PM

Eevans created subtask T172384: OOM exceptions in dev environment.Aug 3 2017, 1:40 PM

• mobrovac added a project: Cassandra.Aug 4 2017, 9:49 PM

Eevans added a subtask: T171772: Prometheus metrics storage for RESTBase dev environment.Sep 7 2017, 3:26 PM

Eevans added a subtask: T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories.Sep 7 2017, 3:34 PM

Eevans changed the status of subtask T169940: End of September milestone: Start migration of production use cases. from Stalled to Open.Sep 7 2017, 7:04 PM

Mentioned in SAL (#wikimedia-operations) [2017-09-13T13:30:44Z] <urandom> T169936: Converting RESTBase dev environment to size-tiered compaction

• mobrovac closed subtask T172224: Consider partitioning local_group_wikipedia even more as Resolved.Sep 20 2017, 11:41 AM

Eevans closed subtask T169939: End of August milestone: Cassandra 3 cluster in production as Resolved.Oct 2 2017, 3:27 PM

Eevans closed subtask T169940: End of September milestone: Start migration of production use cases. as Resolved.

Eevans closed subtask T171772: Prometheus metrics storage for RESTBase dev environment as Resolved.Oct 2 2017, 3:47 PM

Eevans closed subtask T172384: OOM exceptions in dev environment as Resolved.Oct 2 2017, 3:50 PM

Eevans mentioned this in T170276: Test/evaluate JBOD support.Oct 2 2017, 7:34 PM

Change 381884 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] restbase-ng: Collect type=Table (not ColumnFamily)

https://gerrit.wikimedia.org/r/381884

gerritbot added a project: Patch-For-Review.Oct 2 2017, 9:52 PM

Change 381884 merged by Dzahn:
[operations/puppet@production] restbase-ng: Collect type=Table (not ColumnFamily)

https://gerrit.wikimedia.org/r/381884

Q1 has come to an end, this goal was achieved; Closing this issue as resolved.

Eevans closed subtask T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories as Resolved.Jun 11 2018, 3:23 PM