Eevans (Eric Evans)
Senior Software Engineer

Projects (13)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Feb 27 2015, 10:47 PM (176 w, 6 d)
Availability
Available
IRC Nick
urandom
LDAP User
Eevans
MediaWiki User
Unknown

Recent Activity

Wed, Jul 18

Eevans updated the task description for T132815: Develop a RESTBase Cassandra cluster capacity planning document.
Wed, Jul 18, 4:54 PM · User-Eevans, Services (later), Documentation, Cassandra

Tue, Jul 10

Eevans changed the status of T186567: Deprecate cassandra-metrics-collector? from Open to Stalled.

This service is now disabledd/removed on the RESTBase cluster; Thanks a bunch for the assist!

Tue, Jul 10, 7:28 PM · Patch-For-Review, Cassandra, Services (next), User-Eevans

Fri, Jul 6

Eevans added a comment to T197477: RESTBase storage capacity planning.
Test command
fio --randrepeat=1 \
    --ioengine=libaio \
    --direct=1 \
    --gtod_reduce=1 \
    --name=test \
    --filename=/srv/sda4/fio.test \
    --bs=4k \
    --iodepth=4 \
    --size=4G \
    --readwrite=randrw \
    --rwmixread=90
Fri, Jul 6, 7:22 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans added a comment to T198787: Revisit default settings for c-foreach-restart.

So (just to be clear), I use gbp on this repo, and the Debian packaging is in the debian branch, changes to master get merged there before building. I've done that (the merge), and tagged master as upstream/1.0.3. I also updated the changelog to 1.0.3-1.

@MoritzMuehlenhoff, assuming you are still planning to build/upload, I'll leave tagging debian to you, just in case any other changes are needed.

Fri, Jul 6, 4:21 PM · User-Eevans, Patch-For-Review, Services (doing), Cassandra, Operations
Eevans committed rODCTWa05125f70a22: Define a cleaner for git-buildpackage (authored by Eevans).
Define a cleaner for git-buildpackage
Fri, Jul 6, 4:20 PM
Eevans committed rODCTW949b079d33a5: Updated for 1.0.3-1 package release. (authored by Eevans).
Updated for 1.0.3-1 package release.
Fri, Jul 6, 4:20 PM
Eevans committed rODCTWcb691b94b3af: c-foreach-restart: Increase retry and delay defaults (authored by mobrovac).
c-foreach-restart: Increase retry and delay defaults
Fri, Jul 6, 4:20 PM
Eevans added a comment to T198787: Revisit default settings for c-foreach-restart.

So (just to be clear), I use gbp on this repo, and the Debian packaging is in the debian branch, changes to master get merged there before building. I've done that (the merge), and tagged master as upstream/1.0.3. I also updated the changelog to 1.0.3-1.

Fri, Jul 6, 4:04 PM · User-Eevans, Patch-For-Review, Services (doing), Cassandra, Operations
Eevans committed rODCTWbdc417b59012: Updated for 1.0.3-1 package release. (authored by Eevans).
Updated for 1.0.3-1 package release.
Fri, Jul 6, 4:02 PM
Eevans committed rODCTW877017c4225f: Merge master into debian (authored by Eevans).
Merge master into debian
Fri, Jul 6, 4:02 PM
Eevans added a comment to T178690: Better organization for ops grafana dashboards.

[ ... ]

  • restbase (and restbase staging) dashboards I believe can be deleted for the most part as we're now using their prometheus counterpart (cc @Eevans)
Fri, Jul 6, 3:17 PM · Patch-For-Review, User-fgiunchedi, monitoring, Operations

Thu, Jul 5

Eevans moved T198787: Revisit default settings for c-foreach-restart from Backlog to Next on the User-Eevans board.
Thu, Jul 5, 7:14 PM · User-Eevans, Patch-For-Review, Services (doing), Cassandra, Operations
Eevans added a project to T198787: Revisit default settings for c-foreach-restart: User-Eevans.
Thu, Jul 5, 7:14 PM · User-Eevans, Patch-For-Review, Services (doing), Cassandra, Operations
Eevans added a comment to T198787: Revisit default settings for c-foreach-restart.

The change is merged, but we still need to get a package built, have it added to the APT repo, and then upgrade the machines.

Thu, Jul 5, 7:14 PM · User-Eevans, Patch-For-Review, Services (doing), Cassandra, Operations
Eevans added a comment to T198787: Revisit default settings for c-foreach-restart.

With the default settings of c-foreach-restart it's no longer possible to reliably restart Cassandra on the restbase hosts (probably due to increased data set sizes or higher I/O), resulting in tracebacks like the following:

Thu, Jul 5, 7:12 PM · User-Eevans, Patch-For-Review, Services (doing), Cassandra, Operations
Eevans added a comment to T192948: Upgrade prometheus-jmx-exporter on all services using it.

@Pnorman TTBMK, the idea was to ensure that the new Cassandra cluster was setup to use the exporter instead of cassandra-metrics-collector (Graphite).

I don't believe we'll be creating a new Cassandra cluster.

Thu, Jul 5, 7:04 PM · User-Elukey, monitoring, Analytics, Puppet, Services (watching), Cassandra

Tue, Jul 3

Eevans closed T133091: Highest SSTables / read thresholds as Resolved.

This ticket is old, and comes from a period when we were having problems that elevated the importance of this type of reporting. I'm content to call this "unneeded" at this point, and close the issue.

Tue, Jul 3, 10:34 PM · Services (later), Cassandra, Operations, RESTBase-Cassandra
Eevans edited projects for T134461: Evaluate increased memtable_cleanup_threshold values, added: User-Eevans; removed RESTBase-Cassandra.
Tue, Jul 3, 10:32 PM · User-Eevans, Services (attic), Cassandra
Eevans added a project to T106351: RESTBase dashboard annotations for deployments (and more): User-Eevans.
Tue, Jul 3, 10:30 PM · User-Eevans, Services (blocked), Cassandra, RESTBase
Eevans edited projects for T132815: Develop a RESTBase Cassandra cluster capacity planning document, added: User-Eevans; removed RESTBase-Cassandra.
Tue, Jul 3, 10:29 PM · User-Eevans, Services (later), Documentation, Cassandra
Eevans moved T128590: Cassandra uses default ip address for outbound packets while bootstrapping from Backlog to Blocked on the User-Eevans board.
Tue, Jul 3, 10:28 PM · User-Eevans, Services (watching), Cassandra, Operations
Eevans edited projects for T128590: Cassandra uses default ip address for outbound packets while bootstrapping, added: User-Eevans; removed RESTBase-Cassandra.

@fgiunchedi, is this still a thing?

Tue, Jul 3, 10:28 PM · User-Eevans, Services (watching), Cassandra, Operations
Eevans edited projects for T132632: puppetize turning off reserved space for cassandra /srv, added: User-Eevans; removed RESTBase-Cassandra.
Tue, Jul 3, 10:27 PM · User-Eevans, Operations, Services (blocked), Cassandra
Eevans added a comment to T146130: Inconsistent Cassandra disk load shown in metrics and nodetool status.

@elukey, is this still a thing?

Tue, Jul 3, 10:25 PM · Services (watching), Analytics, Cassandra
Eevans added a project to T127472: Investigate reducing impact of single-node Cassandra latencies: User-Eevans.
Tue, Jul 3, 10:23 PM · Services (doing), User-Eevans, Cassandra, RESTBase-Cassandra
Eevans lowered the priority of T127472: Investigate reducing impact of single-node Cassandra latencies from Normal to Low.

@Pchelolo It looks like speculative retries were added to 3.3.0 of the NodeJS driver, can you comment on an expected timeline for moving to it (are we moving to it?)?

Tue, Jul 3, 10:23 PM · Services (doing), User-Eevans, Cassandra, RESTBase-Cassandra
Eevans closed T152724: Current state and next steps for RESTBase storage as Resolved.

I believe this was completed by virtue of the new storage strategy implementation.

Tue, Jul 3, 10:18 PM · Services (designing), User-mobrovac, Operations, RESTBase, Cassandra, RESTBase-Cassandra
Eevans added a project to T158691: Cassandra query tracing: User-Eevans.
Tue, Jul 3, 10:16 PM · User-Eevans, Services (designing), RESTBase, Cassandra
Eevans added a comment to T118128: RFC: Simplify stashing with a shared key-value stash table / bucket.

@mobrovac, @Pchelolo, is this still relevant?

Tue, Jul 3, 10:15 PM · Cassandra, Services (designing), RESTBase
Eevans lowered the priority of T169009: Cassandra Node.JS driver v3.2.2 issues from High to Normal.

@Pchelolo, @mobrovac Is this still relevant?

Tue, Jul 3, 10:13 PM · Services (done), RESTBase, Cassandra
Eevans closed T172997: graphite cassandra metrics disk usage as Resolved.

I think this is solved by the migration to Prometheus (feel free to reopen if that is not the case).

Tue, Jul 3, 10:11 PM · Cassandra, monitoring
Eevans closed T192456: Prometheus metrics missing for some hosts as Resolved.

This was resolved by the upgrade to 1:0.3.0

Tue, Jul 3, 10:09 PM · Services (doing), Cassandra
Eevans added a comment to T192948: Upgrade prometheus-jmx-exporter on all services using it.

We don't see anything for the maps team to do on this - at the very least, we don't think it needs any resources from us.

Tue, Jul 3, 10:08 PM · User-Elukey, monitoring, Analytics, Puppet, Services (watching), Cassandra
Eevans removed a project from T192948: Upgrade prometheus-jmx-exporter on all services using it: RESTBase-Cassandra.
Tue, Jul 3, 10:06 PM · User-Elukey, monitoring, Analytics, Puppet, Services (watching), Cassandra
Eevans updated the task description for T192948: Upgrade prometheus-jmx-exporter on all services using it.
Tue, Jul 3, 10:06 PM · User-Elukey, monitoring, Analytics, Puppet, Services (watching), Cassandra
Eevans added a comment to T192948: Upgrade prometheus-jmx-exporter on all services using it.

RESTBase has been upgraded:

Tue, Jul 3, 10:06 PM · User-Elukey, monitoring, Analytics, Puppet, Services (watching), Cassandra
Eevans updated subscribers of T192948: Upgrade prometheus-jmx-exporter on all services using it.

AQS has been partially upgraded:

Tue, Jul 3, 10:05 PM · User-Elukey, monitoring, Analytics, Puppet, Services (watching), Cassandra
Eevans updated the task description for T179083: Cassandra schema creation seems unreliable.
Tue, Jul 3, 9:18 PM · User-Eevans, RESTBase-Cassandra, Services (next)
Eevans updated the task description for T179083: Cassandra schema creation seems unreliable.
Tue, Jul 3, 9:18 PM · User-Eevans, RESTBase-Cassandra, Services (next)
Eevans lowered the priority of T144431: RESTBase k-r-v as Cassandra anti-pattern from Normal to Lowest.
Tue, Jul 3, 7:50 PM · Services (attic), Operations, Cassandra, RESTBase
Eevans added a comment to T144431: RESTBase k-r-v as Cassandra anti-pattern.

@Eevans @mobrovac I believe this task has served it's purpose and can be closed?

Tue, Jul 3, 7:50 PM · Services (attic), Operations, Cassandra, RESTBase
Eevans closed T122028: RFC: Chunked storage algorithms for archival data vs. large-window brotli compression, a subtask of T120171: RFC: Differentiate storage strategies for archival storage vs. hot current data, as Declined.
Tue, Jul 3, 7:49 PM · Services (later), RfC, RESTBase
Eevans closed T122028: RFC: Chunked storage algorithms for archival data vs. large-window brotli compression as Declined.

@Eevans @mobrovac Do we have any plans to use brotly any more? Should we decline this?

Tue, Jul 3, 7:49 PM · Services (next), RfC, RESTBase
Eevans added a comment to T137419: Investigate aberrant disk read throughput in Cassandra (affects 2.2.x and 3.x).

@Eevans I believe this is done?

Tue, Jul 3, 7:47 PM · User-Eevans, Patch-For-Review, Services (later), Cassandra
Eevans closed T174486: Research: Reduce Cassandra memory usage by avoiding deserialization of expired data as Declined.

@Eevans I believe we can decline this task as we probably will never work on this and the issue is not as pressing as it used to be with the new storage model?

Tue, Jul 3, 7:39 PM · Services (later), Cassandra
Eevans closed T125368: Keep an eye on ScyllaDB as an alternative to Cassandra, a subtask of T152724: Current state and next steps for RESTBase storage, as Resolved.
Tue, Jul 3, 7:24 PM · Services (designing), User-mobrovac, Operations, RESTBase, Cassandra, RESTBase-Cassandra
Eevans closed T125368: Keep an eye on ScyllaDB as an alternative to Cassandra as Resolved.

@Eevans I believe this task can be declined?

Tue, Jul 3, 7:24 PM · Services (later), Cassandra, RESTBase

Fri, Jun 29

Eevans moved T191659: Configure a threshold for earlier notification of /srv/cassandra/instance-data from In-Progress to Blocked on the User-Eevans board.
Fri, Jun 29, 7:25 PM · Patch-For-Review, User-fgiunchedi, Operations, Services (next), RESTBase-Cassandra, User-Eevans, Cassandra
Eevans moved T191659: Configure a threshold for earlier notification of /srv/cassandra/instance-data from Backlog to In-Progress on the User-Eevans board.
Fri, Jun 29, 4:27 PM · Patch-For-Review, User-fgiunchedi, Operations, Services (next), RESTBase-Cassandra, User-Eevans, Cassandra
Eevans awarded T190327: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus a Cookie token.
Fri, Jun 29, 3:52 PM · Services (done), MW-1.32-release-notes (WMF-deploy-2018-04-24 (1.32.0-wmf.1)), Goal, EventBus, MediaWiki-JobQueue, Analytics

Tue, Jun 26

Eevans added a comment to T197062: Upgrade Cassandra on AQS to 2.2.6-wmf5.

Are you going to hack that one host locally, or do you want to pursue fixing the package in a way that works for both Jessie and Stretch?

I think that we could simply keep two 2.2.6 versions in our puppet config, one for Jessie and one for Stretch, and be done with it. So basically re-upload the previous version of 2.2.6 in jessie-wikimedia, and set puppet accordingly. What do you think?

Tue, Jun 26, 5:19 PM · Analytics-Kanban, Patch-For-Review, User-Elukey, Analytics
Eevans added a comment to T197477: RESTBase storage capacity planning.

@Eevans +1 to benchmark different devices and possibly varying the fio parameters too, since now we have a pretty good idea of what the workload looks like.

Do you have specific variations in mind?

Mostly playing with a lower iodepth e.g. 4 as I don't think we see a lot of queueing during normal workloads?

Tue, Jun 26, 4:55 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans added a comment to T197062: Upgrade Cassandra on AQS to 2.2.6-wmf5.

@Eevans for some reason I didn't check what you asked above, I was probably super tired and didn't test this:

elukey@aqs1004:~$ cqlsh -u cassandra aqs1004-a.eqiad.wmnet
Password:
Connected to Analytics Query Service Storage at aqs1004-a.eqiad.wmnet:9042.
[cqlsh 5.0.1 | Cassandra 2.2.6 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>

So simply commenting CQLSH_NO_BUNDLED seems to work like a charm with 2.2.6-wmf5!

Tue, Jun 26, 4:50 PM · Analytics-Kanban, Patch-For-Review, User-Elukey, Analytics
Eevans lowered the priority of T192689: Unchecked storage growth from High to Normal.
Tue, Jun 26, 4:12 PM · Patch-For-Review, RESTBase, User-mobrovac, User-Eevans, Cassandra, Services (doing)

Mon, Jun 25

Eevans added a comment to T197477: RESTBase storage capacity planning.

@Eevans +1 to benchmark different devices and possibly varying the fio parameters too, since now we have a pretty good idea of what the workload looks like.

Mon, Jun 25, 6:25 PM · Services (designing), RESTBase, User-Eevans, Cassandra

Fri, Jun 22

Eevans set the image for User-Eevans to F22483698: profile.
Fri, Jun 22, 3:57 PM

Wed, Jun 20

Eevans added a comment to T197477: RESTBase storage capacity planning.

We have a good understanding of the limits of the Samsung devices (since we routinely run them past those limits), less so for the Intel/HPs, because they are so very under-utilized. Perhaps the easiest way to create a concrete comparison (and infer needed capacity), would be to shutdown Cassandra and benchmark a quiescent storage device. We could do this for a Samsung-equipped host (say restbase2001), and an Intel-equipped one (say restbase2010). Another Intel-equipped host not in use could work as well.

Wed, Jun 20, 9:07 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans updated the task description for T197477: RESTBase storage capacity planning.
Wed, Jun 20, 8:12 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans updated the task description for T197477: RESTBase storage capacity planning.
Wed, Jun 20, 8:08 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans updated the task description for T197477: RESTBase storage capacity planning.
Wed, Jun 20, 8:02 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans updated the task description for T197477: RESTBase storage capacity planning.
Wed, Jun 20, 8:01 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans closed T186751: Restablish RESTBase dev environment with Cassandra 3.11.2 as Resolved.
Wed, Jun 20, 2:52 PM · Services (done), User-Eevans
Eevans closed T186751: Restablish RESTBase dev environment with Cassandra 3.11.2, a subtask of T190689: FY17/18 Q4 Program 7 Services Goal: Language variants support, as Resolved.
Wed, Jun 20, 2:52 PM · Patch-For-Review, Services (doing), Goal, RESTBase-Cassandra, RESTBase-API, RESTBase, Cassandra

Jun 19 2018

Eevans closed T197080: Clean up leftover key spaces as Resolved.
Jun 19 2018, 2:46 AM · User-Eevans, RESTBase, Cassandra, Services (doing)

Jun 18 2018

Eevans updated the task description for T197477: RESTBase storage capacity planning.
Jun 18 2018, 4:16 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans moved T197080: Clean up leftover key spaces from Next to Blocked on the User-Eevans board.
Jun 18 2018, 3:50 PM · User-Eevans, RESTBase, Cassandra, Services (doing)
Eevans added a comment to T197080: Clean up leftover key spaces.

I've verified these myself, by for sake of posterity, I propose applying the following:

Jun 18 2018, 3:50 PM · User-Eevans, RESTBase, Cassandra, Services (doing)
Eevans updated the language for P7268 Masterwork From Distant Lands from autodetect to yaml.
Jun 18 2018, 3:49 PM
Eevans edited P7268 Masterwork From Distant Lands.
Jun 18 2018, 3:49 PM

Jun 15 2018

Eevans updated the task description for T197477: RESTBase storage capacity planning.
Jun 15 2018, 3:40 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans moved T197477: RESTBase storage capacity planning from Backlog to In-Progress on the User-Eevans board.
Jun 15 2018, 3:38 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans added a parent task for T196377: Imbalanced storage distribution over JBOD devices: T197477: RESTBase storage capacity planning.
Jun 15 2018, 3:37 PM · Services (next), Cassandra, User-Eevans
Eevans added a subtask for T197477: RESTBase storage capacity planning: T196377: Imbalanced storage distribution over JBOD devices.
Jun 15 2018, 3:37 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans triaged T197477: RESTBase storage capacity planning as Normal priority.
Jun 15 2018, 3:37 PM · Services (designing), RESTBase, User-Eevans, Cassandra
Eevans created T197477: RESTBase storage capacity planning.
Jun 15 2018, 3:36 PM · Services (designing), RESTBase, User-Eevans, Cassandra

Jun 14 2018

Eevans added a comment to T197062: Upgrade Cassandra on AQS to 2.2.6-wmf5.
  • Fix the patch to source a templated file that defines CQLSH_NO_BUNDLED conditionally (i.e. don't set it on Jessie)
  • ...

Tested this one, leads to:

elukey@aqs1004:~$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

That IIRC it was the original issue :(

Jun 14 2018, 7:14 PM · Analytics-Kanban, Patch-For-Review, User-Elukey, Analytics
Eevans added a comment to T197062: Upgrade Cassandra on AQS to 2.2.6-wmf5.

The new python-cassandra package seems to break cqlsh on Jessie:

elukey@aqs1004:~$ sqls
-bash: sqls: command not found
elukey@aqs1004:~$ cqlsh
Traceback (most recent call last):
  File "/usr/bin/cqlsh.py", line 161, in <module>
    from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
  File "/usr/lib/python2.7/dist-packages/cqlshlib/cql3handling.py", line 17, in <module>
    from .cqlhandling import CqlParsingRuleSet, Hint
  File "/usr/lib/python2.7/dist-packages/cqlshlib/cqlhandling.py", line 21, in <module>
    from cassandra.metadata import cql_keywords_reserved
ImportError: cannot import name cql_keywords_reserved

For the moment I left only aqs1004 with the new 2.2.6-wmf5 cassandra version, meanwhile the other nodes have be wmf3. Rather than fix this issue for jessie I'd concentrate on the Stretch upgrades.

Jun 14 2018, 4:12 PM · Analytics-Kanban, Patch-For-Review, User-Elukey, Analytics

Jun 13 2018

Eevans moved T192689: Unchecked storage growth from Backlog to In-Progress on the User-Eevans board.
Jun 13 2018, 7:33 PM · Patch-For-Review, RESTBase, User-mobrovac, User-Eevans, Cassandra, Services (doing)
Eevans moved T197080: Clean up leftover key spaces from Backlog to Next on the User-Eevans board.
Jun 13 2018, 7:33 PM · User-Eevans, RESTBase, Cassandra, Services (doing)
Eevans triaged T197080: Clean up leftover key spaces as Normal priority.
Jun 13 2018, 7:33 PM · User-Eevans, RESTBase, Cassandra, Services (doing)
Eevans added a comment to T197082: Cassandra schema migrations to add page_language.

These ALTERs LGTM; For posterity, here they are as YAML:

Jun 13 2018, 3:18 PM · Services (done), RESTBase-Cassandra, RESTBase, Cassandra
Eevans updated the language for P7255 Masterwork From Distant Lands from autodetect to yaml.
Jun 13 2018, 3:17 PM
Eevans edited P7255 Masterwork From Distant Lands.
Jun 13 2018, 3:16 PM
Eevans added a comment to T197082: Cassandra schema migrations to add page_language.

Right now here's the difference between the compaction/compression settings between reality in production and what RESTBase will want to set is:

RESTBase:

compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

Production:

compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
compression = {'chunk_length_in_kb': '32', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

The compaction options in RESTBase are controlled by this code, and the pattern for title_revision is not set so it defaults to random_update.

The compression is coming from the table options, however I can't find out why is RESTBase using LZ4 instead of the default Snappy.

Jun 13 2018, 3:12 PM · Services (done), RESTBase-Cassandra, RESTBase, Cassandra

Jun 12 2018

Eevans moved T197019: Publish bulk-import revision culling utility from Backlog to Next on the User-Eevans board.
Jun 12 2018, 5:50 PM · Services (next), RESTBase, User-Eevans, Cassandra
Eevans triaged T197019: Publish bulk-import revision culling utility as Normal priority.
Jun 12 2018, 5:50 PM · Services (next), RESTBase, User-Eevans, Cassandra
Eevans moved T178839: New upstream jvm-tools from In-Progress to Blocked on the User-Eevans board.
Jun 12 2018, 5:07 PM · Services (doing), User-Eevans, User-fgiunchedi, Operations
Eevans added a comment to T196044: cassandra 2.2.6-wmf4 is not compatible with python 2.7.13 (debian stretch).

@Eevans what do we need to do before uploading this to reprepro? I assume some coordination with @elukey is needed. What else?

Jun 12 2018, 5:07 PM · Patch-For-Review, Discovery, Maps, Cassandra, Operations
Eevans moved T197018: Upstream changes to CQLSSTableWriter from Backlog to Next on the User-Eevans board.
Jun 12 2018, 5:00 PM · Services (next), User-Eevans, Cassandra
Eevans triaged T197018: Upstream changes to CQLSSTableWriter as Normal priority.
Jun 12 2018, 5:00 PM · Services (next), User-Eevans, Cassandra
Eevans moved T196377: Imbalanced storage distribution over JBOD devices from Backlog to Next on the User-Eevans board.
Jun 12 2018, 2:21 PM · Services (next), Cassandra, User-Eevans
Eevans closed T178905: Upgrade RESTBase cluster to Cassandra release: 3.11.2 as Resolved.
Jun 12 2018, 2:21 PM · Services (done), User-Eevans, Cassandra
Eevans closed T178905: Upgrade RESTBase cluster to Cassandra release: 3.11.2, a subtask of T177621: Apache Cassandra Tracking, as Resolved.
Jun 12 2018, 2:21 PM · Services (watching), Cassandra

Jun 11 2018

Eevans added a comment to T178905: Upgrade RESTBase cluster to Cassandra release: 3.11.2.

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

Jun 11 2018, 7:43 PM · Services (done), User-Eevans, Cassandra
Eevans closed T196024: Convert timeline keyspaces (indices) to time-windowed compaction as Resolved.
Jun 11 2018, 6:46 PM · Services (done), User-Eevans, RESTBase-Cassandra, Cassandra
Eevans moved T196024: Convert timeline keyspaces (indices) to time-windowed compaction from Backlog to In-Progress on the User-Eevans board.
Jun 11 2018, 3:26 PM · Services (done), User-Eevans, RESTBase-Cassandra, Cassandra
Eevans closed T186360: Cassandra 3.11.0 incompatible with OpenJDK >= 1.8.0_162 as Resolved.

The RESTBase Cassandra cluster has been upgraded to 3.11.2; Resolving

Jun 11 2018, 3:26 PM · Services (blocked), User-Eevans, Cassandra
Eevans edited projects for T179105: Change new storage strategy defaults for Cassandra compression, added: Services; removed Services (doing).
Jun 11 2018, 3:24 PM · Services (next), RESTBase, RESTBase-Cassandra, Cassandra, User-Eevans
Eevans closed T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories, a subtask of T160570: Cassandra 3.x Tracking, as Resolved.
Jun 11 2018, 3:23 PM · Services (doing), Patch-For-Review, Wikimedia-Incident, Cassandra
Eevans closed T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories as Resolved.
Jun 11 2018, 3:23 PM · Patch-For-Review, User-Eevans, Services (next), Cassandra