Eevans (Eric Evans)
Senior Software Engineer

Projects (13)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Feb 27 2015, 10:47 PM (151 w, 1 d)
Availability
Available
IRC Nick
urandom
LDAP User
Eevans
MediaWiki User
Unknown

Recent Activity

Yesterday

Eevans updated the task description for T184100: Reprovision legacy Cassandra nodes into new cluster.
Sat, Jan 20, 11:19 PM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal
Eevans updated the task description for T184100: Reprovision legacy Cassandra nodes into new cluster.
Sat, Jan 20, 3:31 AM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal

Wed, Jan 17

Eevans claimed T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories.
Wed, Jan 17, 3:48 PM · Patch-For-Review, User-Eevans, Services (next), Cassandra
Eevans added a project to T185089: Alert UNKNOWN: RESTBase Cassandra highest tombstones scanned: User-Eevans.
Wed, Jan 17, 3:24 PM · User-Eevans, RESTBase-Cassandra, RESTBase, Cassandra, Services (doing), Goal

Tue, Jan 16

Eevans added a comment to T184100: Reprovision legacy Cassandra nodes into new cluster.

It seems indeed pretty obvious that with HBA mode on there's a performance improvement in terms of latency. I think it makes sense at this point to turn HBA on for the HPs left to be reimagined and proceed. This of course begs the question of what to do with the machines with HBA off, which we'll need to reimage at some point (and gather more data between the two configurations in the mean time)

Tue, Jan 16, 5:54 PM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal
Eevans moved T184100: Reprovision legacy Cassandra nodes into new cluster from Backlog to In-Progress on the User-Eevans board.
Tue, Jan 16, 5:32 PM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal
Eevans added a comment to T184100: Reprovision legacy Cassandra nodes into new cluster.

Here are the nodes of rack a for the previous 2 days. 1007 is a Dell, and 1010 and 1011 are HPs (where the latter has been configured in HBA mode).

Tue, Jan 16, 4:15 PM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal

Thu, Jan 11

Eevans updated subscribers of T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories.
Thu, Jan 11, 5:48 PM · Patch-For-Review, User-Eevans, Services (next), Cassandra
Eevans added a comment to T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories.

Here is some more detail:

Thu, Jan 11, 5:44 PM · Patch-For-Review, User-Eevans, Services (next), Cassandra
Eevans added a comment to T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories.

From a discussion on IRC:

Thu, Jan 11, 5:02 PM · Patch-For-Review, User-Eevans, Services (next), Cassandra
Eevans updated the title for P6579 IRC from untitled to IRC.
Thu, Jan 11, 5:01 PM
Eevans created P6579 IRC.
Thu, Jan 11, 5:01 PM

Wed, Jan 10

Eevans added a project to T184100: Reprovision legacy Cassandra nodes into new cluster: User-Eevans.
Wed, Jan 10, 6:41 PM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal

Tue, Jan 9

Eevans added a comment to T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.

Per our team meeting earlier today: These branches represent significant, non-backward compatible change, so we will publish them each as 1.0.0

Tue, Jan 9, 7:24 PM · Services (done), RESTBase-Cassandra, RESTBase
Eevans added a comment to T184100: Reprovision legacy Cassandra nodes into new cluster.

Next steps before starting with the reimages:

  • Depool the machines above from LVS (cassandra 3 restbase nodes now run restbase)
  • Stop puppet, downtime in icinga
  • Stop restbase
  • Make sure no clients are accessing cassandra on those nodes
  • Stop cassandra
Tue, Jan 9, 5:12 PM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal
Eevans awarded T184110: Set up RESTBase on Cassandra 3 nodes a 100 token.
Tue, Jan 9, 2:51 PM · Services (done), User-mobrovac, Operations, RESTBase

Mon, Jan 8

Eevans added a comment to T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.

Ok, bikeshedding of versions notwithstanding, pull requests are open for rb-m-t-spec, rb-m-t-sqlite, and rb-m-t-cassandra, to adjust versions and naming.

Mon, Jan 8, 10:06 PM · Services (done), RESTBase-Cassandra, RESTBase
Eevans updated the task description for T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.
Mon, Jan 8, 9:55 PM · Services (done), RESTBase-Cassandra, RESTBase
Eevans updated the task description for T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.
Mon, Jan 8, 9:52 PM · Services (done), RESTBase-Cassandra, RESTBase
Eevans added a comment to T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.

For the rb-m-t-* packages, all looks good, except the version bumps: let's bump the minor versions, not just the patch level, as that will prevent automatic upgrades which in this case could break clients. So let's have versions 0.3.0, 0.2.0 and 0.13.0 respectively.

With the exception of rb-m-t-sqlite, none of these were ever published, and each (again, rb-m-t-sqlite notwithstanding) got a version bump at the time of branching; Do we want to bump them a second time?

Mon, Jan 8, 9:26 PM · Services (done), RESTBase-Cassandra, RESTBase
Eevans added a comment to T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.

For the rb-m-t-* packages, all looks good, except the version bumps: let's bump the minor versions, not just the patch level, as that will prevent automatic upgrades which in this case could break clients. So let's have versions 0.3.0, 0.2.0 and 0.13.0 respectively.

Mon, Jan 8, 9:14 PM · Services (done), RESTBase-Cassandra, RESTBase
Eevans added a comment to T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.

Ok, Sanity Check™.

Mon, Jan 8, 5:12 PM · Services (done), RESTBase-Cassandra, RESTBase
Eevans claimed T184199: Discontinue the Cassandra, Sqlite and Spec -ng packages.
Mon, Jan 8, 4:25 PM · Services (done), RESTBase-Cassandra, RESTBase

Wed, Jan 3

Eevans closed T179422: Reshape RESTBase Cassandra clusters as Resolved.

Some cleanups remain, but as we are set to begin a new round of bootstraps, it makes sense to discontinue this effort until they are done; Closing this issue as complete

Wed, Jan 3, 6:07 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans closed T179422: Reshape RESTBase Cassandra clusters, a subtask of T179417: Migrate Parsoid from legacy to new storage, as Resolved.
Wed, Jan 3, 6:07 PM · Services (done), Parsoid, RESTBase-Cassandra, RESTBase
Eevans triaged T184100: Reprovision legacy Cassandra nodes into new cluster as Normal priority.
Wed, Jan 3, 6:05 PM · User-fgiunchedi, User-Eevans, Patch-For-Review, Services (next), Cassandra, Goal
Eevans added a comment to T183745: FY17/18 Q3 Program 7 Services Goal: Full migration to Cassandra 3.

All non-mathoid keyspaces have been successfully dropped:

Wed, Jan 3, 5:41 PM · RESTBase-Cassandra, RESTBase, Cassandra, Services (doing), Goal
Eevans added a comment to T183745: FY17/18 Q3 Program 7 Services Goal: Full migration to Cassandra 3.

I've double-checked that none of these keyspaces are receiving any reads/writes.

Wed, Jan 3, 4:00 PM · RESTBase-Cassandra, RESTBase, Cassandra, Services (doing), Goal

Tue, Jan 2

Eevans added a comment to T179416: Program 7 Outcome 2 Objective 1, Q2: Develop a scalable and cost-effective storage solution for backing the REST API.

Assuming that the only remaining use-case on the legacy Cassandra cluster is mathoid, I propose the following to drop every remaining (non-system) keyspace that is NOT in service of mathoid.

Tue, Jan 2, 11:23 PM · Services (done), Goal, Cassandra, Epic, RESTBase-Cassandra, RESTBase
Eevans updated the language for P6516 Masterwork From Distant Lands from autodetect to yaml.
Tue, Jan 2, 11:18 PM
Eevans edited P6516 Masterwork From Distant Lands.
Tue, Jan 2, 11:18 PM
Eevans moved T178839: New upstream jvm-tools from Blocked to In-Progress on the User-Eevans board.
Tue, Jan 2, 5:37 PM · Services (doing), User-Eevans, User-fgiunchedi, Operations

Dec 20 2017

Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 20 2017, 6:15 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)

Dec 19 2017

Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 19 2017, 2:40 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 19 2017, 2:36 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)

Dec 15 2017

Eevans lowered the priority of T177650: Cassandra compaction throughput rate limiting from Normal to Low.
Dec 15 2017, 9:13 PM · User-Eevans, Cassandra, Services (doing)
Eevans updated subscribers of T177650: Cassandra compaction throughput rate limiting.

An additional data point (that makes matters somewhat worse), is that there are two metrics that one would assume should agree:

org.apache.cassandra.metrics<type=Compaction, name=BytesCompacted><>Count

org.apache.cassandra.metrics<type=Table, keyspace={keyspace}, scope={table}, name=CompactionBytesWritten><>Count

My assumption would be that the latter, summed for all tables, would equal the former, but this is not the case.

Dec 15 2017, 9:13 PM · User-Eevans, Cassandra, Services (doing)

Dec 14 2017

Eevans added a comment to T177650: Cassandra compaction throughput rate limiting.

Should this discrepancy be followed up with upstream?

Dec 14 2017, 2:43 PM · User-Eevans, Cassandra, Services (doing)

Dec 13 2017

Eevans added a comment to T177650: Cassandra compaction throughput rate limiting.

An additional data point (that makes matters somewhat worse), is that there are two metrics that one would assume should agree:

Dec 13 2017, 9:52 PM · User-Eevans, Cassandra, Services (doing)
Eevans moved T177650: Cassandra compaction throughput rate limiting from Backlog to Next on the User-Eevans board.
Dec 13 2017, 9:40 PM · User-Eevans, Cassandra, Services (doing)
Eevans added a project to T177650: Cassandra compaction throughput rate limiting: User-Eevans.
Dec 13 2017, 9:39 PM · User-Eevans, Cassandra, Services (doing)
Eevans updated subscribers of T177650: Cassandra compaction throughput rate limiting.
Dec 13 2017, 9:39 PM · User-Eevans, Cassandra, Services (doing)
Eevans added a project to T137419: Investigate aberrant disk read throughput in Cassandra (affects 2.2.x and 3.x): User-Eevans.
Dec 13 2017, 8:25 PM · User-Eevans, Patch-For-Review, Services (later), Cassandra
Eevans closed T159161: User-defined compaction work-around for tombstone GC as Resolved.

The move to the new storage strategy has made operationalizing this work-around unnecessary. Closing.

Dec 13 2017, 8:23 PM · Cassandra, Services (blocked)
Eevans closed T160759: Cassandra OOMs as Resolved.

We're no longer experiencing chronic OOM exceptions, and so should no longer need a ticket to document them. Closing.

Dec 13 2017, 8:21 PM · Services (doing), Cassandra
Eevans updated subscribers of T178839: New upstream jvm-tools.

[ ... ]

  1. Added https://github.com/aragozin/jvm-tools.git as a Git remote
  2. Merged jvmtool-umbrella-pom-0.8.1 (the upstream tag) onto master
  3. Tagged upstream/0.8.1 -> jvmtool-umbrella-pom-0.8.1
  4. Updated debian/ accordingly

Obviously, the result doesn't lend itself well to a Gerrit changeset for review.

Dec 13 2017, 8:14 PM · Services (doing), User-Eevans, User-fgiunchedi, Operations
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 13 2017, 7:49 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 13 2017, 7:45 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans moved T181964: Clean metrics for restbase erroneus legacy tables from cassandra 3 cluster from In-Progress to Blocked on the User-Eevans board.
Dec 13 2017, 4:37 PM · Services (doing), RESTBase, Cassandra, User-Eevans
Eevans added a comment to T181964: Clean metrics for restbase erroneus legacy tables from cassandra 3 cluster.

These keyspaces are definitely gone at this point, and the schema is in total agreement. On the basis that anything being sent to Graphite should also show up in Prometheus, I spot checked a couple of machines. restbase2002-{b,c} both have table-specific metrics (oddly only those pertaining to CAS), but restbase1010 does not.

Dec 13 2017, 4:36 PM · Services (doing), RESTBase, Cassandra, User-Eevans
Eevans moved T177664: Cassandra races the JVM to create a heapdump from Backlog to Blocked on the User-Eevans board.
Dec 13 2017, 3:38 PM · User-Eevans, Services (blocked), Cassandra
Eevans edited projects for T177664: Cassandra races the JVM to create a heapdump, added: Services (blocked), User-Eevans; removed Services (watching).
Dec 13 2017, 3:37 PM · User-Eevans, Services (blocked), Cassandra
Eevans added a comment to T177664: Cassandra races the JVM to create a heapdump.

This has been resolved upstream, and will be included in 3.11.2.

Dec 13 2017, 3:37 PM · User-Eevans, Services (blocked), Cassandra
Eevans updated subscribers of T158691: Cassandra query tracing.
Dec 13 2017, 3:33 PM · Services (designing), RESTBase, Cassandra
Eevans moved T179422: Reshape RESTBase Cassandra clusters from In-Progress to Blocked on the User-Eevans board.
Dec 13 2017, 3:31 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)

Dec 11 2017

Eevans added a comment to T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.

More data:

Dec 11 2017, 10:21 PM · User-Eevans, Services (doing), Cassandra
Eevans claimed T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.

We now have ~3 days of metrics since Smart Path was disabled. It almost looks like it could be a tiny bit less spike-y, but I wouldn't swear to it; I don't think this had any/much effect.

Dec 11 2017, 7:55 PM · User-Eevans, Services (doing), Cassandra
Eevans awarded T178690: Better organization for ops grafana dashboards a Love token.
Dec 11 2017, 7:27 PM · User-fgiunchedi, monitoring, Operations
Eevans edited projects for T164865: Prototype and test range delete-based current revision storage, added: Services (done); removed Services (doing).
Dec 11 2017, 3:26 PM · Services (done), ChangeProp, User-mobrovac, Cassandra, RESTBase
Eevans closed T164865: Prototype and test range delete-based current revision storage as Resolved.

@Eevans I think this can be resolved given that we've put our fish in the range-delete basket?

Dec 11 2017, 3:26 PM · Services (done), ChangeProp, User-mobrovac, Cassandra, RESTBase
Eevans closed T164865: Prototype and test range delete-based current revision storage, a subtask of T156199: Low-latency current revision storage, as Resolved.
Dec 11 2017, 3:25 PM · User-mobrovac, Cassandra, Services (designing), Wikimedia-Incident, RfC, RESTBase
Eevans updated the task description for T164865: Prototype and test range delete-based current revision storage.
Dec 11 2017, 3:25 PM · Services (done), ChangeProp, User-mobrovac, Cassandra, RESTBase

Dec 8 2017

Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 8 2017, 5:25 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans added a comment to T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.

Smart Path has been disabled.

Dec 8 2017, 4:42 PM · User-Eevans, Services (doing), Cassandra

Dec 7 2017

Eevans added a comment to T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.

My comments are under the assumption that the difference in behaviour is between the new hosts vs the old hosts, as opposed to the first analysis that was pointing to a single host showing higher latencies.

Dec 7 2017, 5:04 PM · User-Eevans, Services (doing), Cassandra

Dec 6 2017

Eevans renamed T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4} from Investigate aberrant Cassandra columnfamily read latency of restbase1010 to Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.
Dec 6 2017, 9:24 PM · User-Eevans, Services (doing), Cassandra
Eevans updated subscribers of T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.

From a conversation with @Volans on IRC, it might be worthwhile to disabled Smart Path.

Dec 6 2017, 7:46 PM · User-Eevans, Services (doing), Cassandra
Eevans added a comment to T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.

On closer inspection, and now that restbase1012 and restbase1014 have been bootstrapped in eqiad, and the cluster given time to quiesce, we have.

Dec 6 2017, 7:40 PM · User-Eevans, Services (doing), Cassandra
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 6 2017, 6:23 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 6 2017, 5:21 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 6 2017, 5:16 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans changed the status of T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4} from Stalled to Open.
Dec 6 2017, 4:22 PM · User-Eevans, Services (doing), Cassandra
Eevans changed the status of T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}, a subtask of T179416: Program 7 Outcome 2 Objective 1, Q2: Develop a scalable and cost-effective storage solution for backing the REST API, from Stalled to Open.
Dec 6 2017, 4:22 PM · Services (done), Goal, Cassandra, Epic, RESTBase-Cassandra, RESTBase
Eevans added a comment to T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4}.

Update: The firmware has been upgraded, but the aberrant latency remains; Back to the drawing board.

Dec 6 2017, 4:21 PM · User-Eevans, Services (doing), Cassandra
Eevans added a comment to T126619: cassandra slow streaming during (de)commission.

Some additional information: CASSANDRA-4663 adds concurrency at the keyspace level (Cassandra 4.0+)

Dec 6 2017, 3:49 PM · Services (later), RESTBase-Cassandra, Operations

Dec 5 2017

Eevans moved T178177: Investigate aberrant Cassandra columnfamily read latency of restbase101{0,2,4} from Blocked to In-Progress on the User-Eevans board.
Dec 5 2017, 9:56 PM · User-Eevans, Services (doing), Cassandra
Eevans closed T147961: Deploy Brotli compression support for Cassandra as Resolved.

The impetus for this work was optimal storage utilization for full document history. Since we are currently focused on storage of current revisions, (with long-term storage on hold) this doesn't seem relevant any more.

Dec 5 2017, 8:58 PM · Services (later), RESTBase-Cassandra, Cassandra
Eevans closed T150811: Evaluate ScyllaDB as a near-term replacement to Cassandra as Resolved.

Just from an operational perspective, this would be a large undertaking, with questionable benefit (at some level we'd almost certainly be trading one set of problems for another).

Dec 5 2017, 8:54 PM · Services (later), RESTBase, Operations, Cassandra
Eevans closed T150811: Evaluate ScyllaDB as a near-term replacement to Cassandra, a subtask of T125368: Keep an eye on ScyllaDB as an alternative to Cassandra, as Resolved.
Dec 5 2017, 8:54 PM · Services (later), Cassandra, RESTBase
Eevans closed T153703: Option: Consider switching back to leveled compaction (LCS) as Declined.

I feel this is pretty much a non-starter for RESTBase use-cases. The write amplification can be quite problematic, as T180568: Aberrant load on instances involved in recent bootstrap demonstrated. Additionally, our storage strategies generate a significant number of tombstones, and LCS's tombstone GC is legendarily poor.

Dec 5 2017, 8:44 PM · User-mobrovac, Operations, Services (doing), RESTBase, Cassandra, RESTBase-Cassandra
Eevans closed T153703: Option: Consider switching back to leveled compaction (LCS), a subtask of T152724: Current state and next steps for RESTBase storage, as Declined.
Dec 5 2017, 8:44 PM · Services (designing), User-mobrovac, Operations, RESTBase, Cassandra, RESTBase-Cassandra
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 5 2017, 8:28 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans added a project to T130862: Discerning Cassandra instance in Logstash/Kibana: User-Eevans.
Dec 5 2017, 8:25 PM · User-Eevans, Services (later), Cassandra, RESTBase-Cassandra
Eevans closed T126619: cassandra slow streaming during (de)commission as Resolved.

The limitation here is that there is no per-session concurrency for streaming in Cassandra. The throughput observed is the result of what a single thread is capable of decompressing on the receiving end. This has been reported upstream as: https://issues.apache.org/jira/browse/CASSANDRA-14097.

Dec 5 2017, 8:25 PM · Services (later), RESTBase-Cassandra, Operations
Eevans closed T143054: Mutation of <x> bytes is too large..., exceptions as Declined.

This issue is quite old, and it does not appear to be an issue for the new storage strategy; Let's close

Dec 5 2017, 7:48 PM · Services (next), Cassandra
Eevans moved T178905: Evaluate new upstream Cassandra release: 3.11.1 from Backlog to Next on the User-Eevans board.
Dec 5 2017, 3:39 PM · User-Eevans, Services (next), Cassandra
Eevans moved T181964: Clean metrics for restbase erroneus legacy tables from cassandra 3 cluster from Backlog to In-Progress on the User-Eevans board.
Dec 5 2017, 3:39 PM · Services (doing), RESTBase, Cassandra, User-Eevans
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 5 2017, 2:59 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans moved T178839: New upstream jvm-tools from In-Progress to Blocked on the User-Eevans board.
Dec 5 2017, 12:55 AM · Services (doing), User-Eevans, User-fgiunchedi, Operations
Eevans claimed T178839: New upstream jvm-tools.

I've taken a stab at this, but since I'm not 100% certain what the intention was, procedure-wise, I've pushed it to https://github.com/eevans/jvm-tools for the time being.

Dec 5 2017, 12:55 AM · Services (doing), User-Eevans, User-fgiunchedi, Operations
Eevans edited projects for T178839: New upstream jvm-tools, added: Services (doing); removed Services (watching).
Dec 5 2017, 12:41 AM · Services (doing), User-Eevans, User-fgiunchedi, Operations
Eevans moved T178839: New upstream jvm-tools from Backlog to In-Progress on the User-Eevans board.
Dec 5 2017, 12:41 AM · Services (doing), User-Eevans, User-fgiunchedi, Operations

Dec 4 2017

Eevans edited projects for T177501: Prometheus cluster attribute for new RESTBase Cassandra cluster, added: Services (done); removed Services (doing).
Dec 4 2017, 7:25 PM · Services (done), Patch-For-Review, Operations
Eevans closed T177501: Prometheus cluster attribute for new RESTBase Cassandra cluster as Resolved.

Doing this would cause a loss of metric history for the machines with a renamed attribute, and long-term, it is undesirable to perpetuate the _ng suffix; This situation (two clusters) is temporary, and for the time being, livable, so lets pass on doing this.

Dec 4 2017, 7:24 PM · Services (done), Patch-For-Review, Operations
Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 4 2017, 3:02 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)

Dec 1 2017

Eevans updated the task description for T179422: Reshape RESTBase Cassandra clusters.
Dec 1 2017, 10:47 PM · User-Eevans, Patch-For-Review, User-fgiunchedi, RESTBase-Cassandra, RESTBase, Services (doing)
Eevans closed T181689: New RESTBase Cassandra cluster has legacy tables as Resolved.

Erroneous keyspaces have been dropped, and the snapshots (manually) cleared; Closing

Dec 1 2017, 5:12 PM · Services (doing), RESTBase, Cassandra, User-Eevans
Eevans added a comment to T181689: New RESTBase Cassandra cluster has legacy tables.

Update: the process of dropping these keyspaces continues; Using the (conservative) algorithm in mkschema, these DROPs are happening at a rate of ~9/hour. At this time we are ~75% complete.

Dec 1 2017, 3:39 AM · Services (doing), RESTBase, Cassandra, User-Eevans

Nov 30 2017

Eevans moved T179083: Cassandra schema creation seems unreliable from In-Progress to Blocked on the User-Eevans board.
Nov 30 2017, 4:11 PM · User-Eevans, RESTBase-Cassandra, Services (next)
Eevans added a comment to T181689: New RESTBase Cassandra cluster has legacy tables.

+1 on dropping them, but since there are still logs about cluster name disagreements, perhaps it would be good to first do a rolling restart of the Cass3 cluster @Eevans ?

Nov 30 2017, 3:24 PM · Services (doing), RESTBase, Cassandra, User-Eevans