This service is now disabledd/removed on the RESTBase cluster; Thanks a bunch for the assist!

In T198787#4403893, @Eevans wrote:

So (just to be clear), I use gbp on this repo, and the Debian packaging is in the debian branch, changes to master get merged there before building. I've done that (the merge), and tagged master as upstream/1.0.3. I also updated the changelog to 1.0.3-1.

@MoritzMuehlenhoff, assuming you are still planning to build/upload, I'll leave tagging debian to you, just in case any other changes are needed.

So (just to be clear), I use gbp on this repo, and the Debian packaging is in the debian branch, changes to master get merged there before building. I've done that (the merge), and tagged master as upstream/1.0.3. I also updated the changelog to 1.0.3-1.

In T178690#4403664, @fgiunchedi wrote:

[ ... ]

• restbase (and restbase staging) dashboards I believe can be deleted for the most part as we're now using their prometheus counterpart (cc @Eevans)

The change is merged, but we still need to get a package built, have it added to the APT repo, and then upgrade the machines.

With the default settings of c-foreach-restart it's no longer possible to reliably restart Cassandra on the restbase hosts (probably due to increased data set sizes or higher I/O), resulting in tracebacks like the following:

In T192948#4396564, @Pnorman wrote:

@Pnorman TTBMK, the idea was to ensure that the new Cassandra cluster was setup to use the exporter instead of cassandra-metrics-collector (Graphite).

I don't believe we'll be creating a new Cassandra cluster.

This ticket is old, and comes from a period when we were having problems that elevated the importance of this type of reporting. I'm content to call this "unneeded" at this point, and close the issue.

@fgiunchedi, is this still a thing?

@elukey, is this still a thing?

@Pchelolo It looks like speculative retries were added to 3.3.0 of the NodeJS driver, can you comment on an expected timeline for moving to it (are we moving to it?)?

I believe this was completed by virtue of the new storage strategy implementation.

@mobrovac, @Pchelolo, is this still relevant?

@Pchelolo, @mobrovac Is this still relevant?

I think this is solved by the migration to Prometheus (feel free to reopen if that is not the case).

This was resolved by the upgrade to 1:0.3.0

In T192948#4316415, @Pnorman wrote:

We don't see anything for the maps team to do on this - at the very least, we don't think it needs any resources from us.

RESTBase has been upgraded:

AQS has been partially upgraded:

In T144431#4394285, @Pchelolo wrote:

@Eevans @mobrovac I believe this task has served it's purpose and can be closed?

In T122028#4394277, @Pchelolo wrote:

@Eevans @mobrovac Do we have any plans to use brotly any more? Should we decline this?

In T137419#4394249, @Pchelolo wrote:

@Eevans I believe this is done?

In T174486#4394187, @Pchelolo wrote:

@Eevans I believe we can decline this task as we probably will never work on this and the issue is not as pressing as it used to be with the new storage model?

In T125368#4394192, @Pchelolo wrote:

@Eevans I believe this task can be declined?

In T197062#4316345, @elukey wrote:

In T197062#4316325, @Eevans wrote:

Are you going to hack that one host locally, or do you want to pursue fixing the package in a way that works for both Jessie and Stretch?

I think that we could simply keep two 2.2.6 versions in our puppet config, one for Jessie and one for Stretch, and be done with it. So basically re-upload the previous version of 2.2.6 in jessie-wikimedia, and set puppet accordingly. What do you think?

In T197477#4314389, @fgiunchedi wrote:

In T197477#4312964, @Eevans wrote:

In T197477#4310619, @fgiunchedi wrote:

@Eevans +1 to benchmark different devices and possibly varying the fio parameters too, since now we have a pretty good idea of what the workload looks like.

Do you have specific variations in mind?

Mostly playing with a lower iodepth e.g. 4 as I don't think we see a lot of queueing during normal workloads?

In T197062#4316238, @elukey wrote:

@Eevans for some reason I didn't check what you asked above, I was probably super tired and didn't test this:

elukey@aqs1004:~$ cqlsh -u cassandra aqs1004-a.eqiad.wmnet
Password:
Connected to Analytics Query Service Storage at aqs1004-a.eqiad.wmnet:9042.
[cqlsh 5.0.1 | Cassandra 2.2.6 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>

So simply commenting CQLSH_NO_BUNDLED seems to work like a charm with 2.2.6-wmf5!

In T197477#4310619, @fgiunchedi wrote:

@Eevans +1 to benchmark different devices and possibly varying the fio parameters too, since now we have a pretty good idea of what the workload looks like.

We have a good understanding of the limits of the Samsung devices (since we routinely run them past those limits), less so for the Intel/HPs, because they are so very under-utilized. Perhaps the easiest way to create a concrete comparison (and infer needed capacity), would be to shutdown Cassandra and benchmark a quiescent storage device. We could do this for a Samsung-equipped host (say restbase2001), and an Intel-equipped one (say restbase2010). Another Intel-equipped host not in use could work as well.

I've verified these myself, by for sake of posterity, I propose applying the following:

In T197062#4283220, @elukey wrote:

In T197062#4283216, @Eevans wrote:

• Fix the patch to source a templated file that defines CQLSH_NO_BUNDLED conditionally (i.e. don't set it on Jessie)
• ...

Tested this one, leads to:

elukey@aqs1004:~$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

That IIRC it was the original issue :(

In T197062#4282812, @elukey wrote:

The new python-cassandra package seems to break cqlsh on Jessie:

elukey@aqs1004:~$ sqls
-bash: sqls: command not found
elukey@aqs1004:~$ cqlsh
Traceback (most recent call last):
  File "/usr/bin/cqlsh.py", line 161, in <module>
    from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
  File "/usr/lib/python2.7/dist-packages/cqlshlib/cql3handling.py", line 17, in <module>
    from .cqlhandling import CqlParsingRuleSet, Hint
  File "/usr/lib/python2.7/dist-packages/cqlshlib/cqlhandling.py", line 21, in <module>
    from cassandra.metadata import cql_keywords_reserved
ImportError: cannot import name cql_keywords_reserved

For the moment I left only aqs1004 with the new 2.2.6-wmf5 cassandra version, meanwhile the other nodes have be wmf3. Rather than fix this issue for jessie I'd concentrate on the Stretch upgrades.

These ALTERs LGTM; For posterity, here they are as YAML:

In T197082#4278533, @Pchelolo wrote:

Right now here's the difference between the compaction/compression settings between reality in production and what RESTBase will want to set is:

RESTBase:

compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

Production:

compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
compression = {'chunk_length_in_kb': '32', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

The compaction options in RESTBase are controlled by this code, and the pattern for title_revision is not set so it defaults to random_update.

The compression is coming from the table options, however I can't find out why is RESTBase using LZ4 instead of the default Snappy.

In T196044#4276434, @Gehel wrote:

@Eevans what do we need to do before uploading this to reprepro? I assume some coordination with @elukey is needed. What else?

In T178905#4270966, @MoritzMuehlenhoff wrote:

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

The RESTBase Cassandra cluster has been upgraded to 3.11.2; Resolving