In T363996#9826387, @hnowlan wrote:

certs updated in all DCs, alerts resolved. I sincerely hope we will have the mesh migration resolved so we can avoid having to update echostore's certificates in October, but in case something prevents that and for reference the process was:

• puppet cert revoke sessionstore.discovery.wmnet
• In the puppet repo on your local checkout ./utils/create_ecdsa_cert sessionstore.discovery.wmnet sessionstore.svc.eqiad.wmnet sessionstore.svc.codfw.wmnet
• On the puppetmaster, put the contents of /var/lib/puppet/server/ssl/ca/signed/sessionstore.discovery.wmnet.pem into certs.kask.cert in helmfile.d
• Add the contents of the new private key from ./modules/secret/secrets/ssl/sessionstore.discovery.wmnet.key to hieradata/role/common/deployment_server/kubernetes.yaml
• Validate the files and make sure everything looks okay using openssl ec/openssl x509, then git commit your changes in private
• Follow the Helm rollout process as normal, keeping an eye on the sessionstore graphs and the session loss graphs

In T364583#9827071, @mforns wrote:

Oh! It's great that we can do range deletions from Cassandra!
I think with the deprecation of Druid, we are adopting Cassandra as our main serving layer, and we should be able to delete data from it given a time range.
This would help do idempotent reruns, and implement deletion policies.

I think TTL is convenient, because it's easy. But I'm not sure is the good paradigm for long lived data?
It has some problems, like:

• If we backfill say 1 year of data, then all that data is going to be deleted at the same time.
• If we re-run past data, then the updated data will live longer than more recent data.

We're aware, yeah, current efforts being tracked in T362033: Degraded RAID on aqs1013 (no shortage of tickets on this one 😞).

In T363996#9816587, @elukey wrote:

In T363996#9794048, @hnowlan wrote:

Steps that I see:

• Renewing the existing cergen cert to give us breathing room just in case. We're looking at less than 2 weeks of headroom for a major change to one of our most critical services
• Add toggleable mesh support
• Get a baseline for performance in staging by running siege for a few hours
• Deploy mesh-enabled version to staging, compare performance before and after
• Roll forward if everything looks okay

Only concern around testing in staging is having reasonably representative infrastructure, given that it's both using cassandra-dev, and in codfw from pods in eqiad. That said, we're just looking for a baseline and not identical numbers.

+1! Not sure how to renew the existing cert since IIUC cergen wasn't used, but we can try to check in old tasks to see what it was done.

In T363996#9794048, @hnowlan wrote:

Steps that I see:

• Renewing the existing cergen cert to give us breathing room just in case. We're looking at less than 2 weeks of headroom for a major change to one of our most critical services

[ ... ]

In T364583#9812869, @xcollazo wrote:

In T364583#9810625, @Eevans wrote:

...

CREATE TABLE commons.top_viewed_categories_monthly (
    category_scope      VARCHAR,
    wiki                VARCHAR,
    category            VARCHAR,
    pageview_count      BIGINT,
    rank                INT,
    year                INT,
    month               INT,
    PRIMARY KEY ((category_scope, wiki), year, month, rank)
);

The one obstacle I see in a situation like this is cardinality. Changing it like this means that we're growing the partition from being based on the number of unique rank attributes (for each (category_scope, wiki, year, month)), to 12 * num_years * num_ranks (assuming the number of unique ranks is relatively low, that's not much). The other side of the equation is distribution. Changing it like this means we we're shrinking the number of partitions from the number of unique (category_scope, wiki, year, month) tuples, to the number of unique (category_scope, wiki). If the latter is overly small, then distribution won't be as good.

Do you have some idea of the cardinality of these datasets?

If a change like this will fit your data, I think the schema is all we'd need to fix; All we'd be doing is changing the way Cassandra organizes the data, the schema insofar as your client code is concerned should work as-is.

ranks cardinality is 1000. That is, we calcualte a max of 1000 rows per (category_scope, wiki, year, month). category_scope can only be one of shallow or deep. So cardinality is 2. wiki can be any of our ~900+ wikis.

Calculating over 5 years of data:

Unique (category_scope, wiki, year, month) should be about 2 * 900 * 5 * 12 = 108,000.

Unique (category_scope, wiki) should be about 2 * 900 = 1,800.

So definitely a stark contrast for this particular table.

In T364583#9812940, @xcollazo wrote:

In T364583#9812922, @xcollazo wrote:

Couldn't we add a dummy TIMESTAMP for DELETE purposes?

That is, for all top_* tables, we add:

CREATE TABLE commons.top_viewed_categories_monthly (
    category_scope      VARCHAR,
    wiki                VARCHAR,
    category            VARCHAR,
    pageview_count      BIGINT,
    rank                INT,
    year                INT,
    month               INT,
    internal_dt         TIMESTAMP,                 <<<<<<<<<<<<<<<<
    PRIMARY KEY ((category_scope, wiki, year, month), rank)
);

Then the deletes could be like DELETE FROM top_viewed_categories_monthly WHERE internal_dt < '2020-01-01' since that column is not part of the PRIMARY KEY. Looks like that would cost us a 64-bit integer, but seems like it would solve the issue without modifications to the PRIMARY KEY, and no TTLs needed as well.

Actually, it could be a DATE type, and then it would only cost 32-bits.

In T362033#9798051, @Eevans wrote:

The array has rebuilt, but I could swear I hear it ticking...

In T364583#9810103, @Eevans wrote:

In T364583#9810082, @xcollazo wrote:

This should work for any time-series datasets
...
Other CIM tables (like commons.top_viewed_categories_monthly) could probably be re-thought to make this possible as well.

All top_* tables have rows that belong to a particular year and month. Example:

CREATE TABLE commons.top_viewed_categories_monthly (
    category_scope      VARCHAR,
    wiki                VARCHAR,
    category            VARCHAR,
    pageview_count      BIGINT,
    rank                INT,
    year                INT,
    month               INT,
    PRIMARY KEY ((category_scope, wiki, year, month), rank)
);

So wouldn't the tombstone thingy work similarly? As in:

DELETE FROM top_viewed_categories_monthly WHERE year < 2020

Semantically, that DELETE is ok with our data model.

No, because year & month are a part of the partition key; The category_scope, wiki, year, and month attributes are for all intents and purposes concatenated together. A range delete like this is something that has to be done against the (indexed) partition itself.

In T364583#9810082, @xcollazo wrote:

This should work for any time-series datasets
...
Other CIM tables (like commons.top_viewed_categories_monthly) could probably be re-thought to make this possible as well.

All top_* tables have rows that belong to a particular year and month. Example:

CREATE TABLE commons.top_viewed_categories_monthly (
    category_scope      VARCHAR,
    wiki                VARCHAR,
    category            VARCHAR,
    pageview_count      BIGINT,
    rank                INT,
    year                INT,
    month               INT,
    PRIMARY KEY ((category_scope, wiki, year, month), rank)
);

So wouldn't the tombstone thingy work similarly? As in:

DELETE FROM top_viewed_categories_monthly WHERE year < 2020

Semantically, that DELETE is ok with our data model.

One thing worth considering: Since TTLs are recorded at write-time, on a per-record basis, it makes changing your retention policy later very costly (probably prohibitively so).

@JAllemandou could you elaborate on the performance issues?

In T364921#9807379, @Scott_French wrote:

Many thanks for getting the image builds running and setting up the data_gateway role, @Eevans.

With the exception of a private-puppet patch [0] to wire in the credentials for the data_gateway role, I think all of the patches needed to get us to a working service are ready. I'll start sending those out for review soon.

[0] https://wikitech.wikimedia.org/wiki/Kubernetes/Add_a_new_service#Add_private_data/secrets_(optional)

For posterity sake: I will make a note here that while most of the AQS datasets make use of the Leveled compaction strategy, I've used Size-tiered here. Leveled was chosen (way back when) to maximize read latency (which comes at the expense of significantly higher write throughput). Experience has proven though that the tiering you get from STCS typically matches what you see from Leveled, especially for a total-ordered dataset like this; I suspect that latency will be on par, and that we can extend the life of our SSDs in the process. I'll be monitoring this as the dataset grows though, and if that doesn't prove to be true, we can always change it later.

In T362697#9808708, @Eevans wrote:

In T362697#9807534, @xcollazo wrote:

I have updated the DDL for Cassandra in the description of this task.

The changes are:

• Adopted the recommendations from T362697#9728035 related to demoting some columns from BIGINTs to INTs.
• Adopted the recommendations from T362697#9728035 related to removing the top_data blob and adopting a range query strategy.
• Table names and some column names have been updated as per changes in our Service Layer Design.

Kindly please review, and if no issues, let's deploy to the Cassandra Staging cluster whenever you folks have the time.

Thanks @xcollazo, I'll get started and update you when it's done.

In T362697#9808831, @mforns wrote:

@Eevans it should take around 1 sprint (3 weeks) for us to adapt the service to the latest naming changes, do the integration tests and QA. So, it will be when you come back.

In T362697#9807534, @xcollazo wrote:

I have updated the DDL for Cassandra in the description of this task.

The changes are:

• Adopted the recommendations from T362697#9728035 related to demoting some columns from BIGINTs to INTs.
• Adopted the recommendations from T362697#9728035 related to removing the top_data blob and adopting a range query strategy.
• Table names and some column names have been updated as per changes in our Service Layer Design.

Kindly please review, and if no issues, let's deploy to the Cassandra Staging cluster whenever you folks have the time.

@KOfori as group approver for cassandra-staging-devs...do you? :)

This is now done. The document is here: https://wikitech.wikimedia.org/wiki/Cassandra/Staging (it's still quite bare, so if you have any questions, let me know).

In T364584#9784712, @Eevans wrote:

In T364584#9784601, @xcollazo wrote:

@Eevans if you can point us to any existing documentation about this staging Cassandra cluster I'd appreciate it.

The existing documentation (such as it is) is: https://wikitech.wikimedia.org/wiki/Cassandra/Staging

The process for being added to the cassandra-staging-devs group is the same that is used everywhere else, see: https://wikitech.wikimedia.org/wiki/SRE/Production_access#Access_Request_Process. For those that already have shell access, that will probably just boil down to filing the templated request (one request per user).

A Docker image is now published:

The array has rebuilt, but I could swear I hear it ticking...

The failed device (sdd) was replaced; This time we're using sfdisk to copy the partition table.

That didn't take long:

This no longer happens in deployment-prep (hasn't in a very long time), and was never happening production, so I am going to close this.

In T363996#9785916, @akosiaris wrote:

In T363996#9774208, @JMeybohm wrote:

@akosiaris maybe you recall if there was a deliberate decision not to use service mesh for kask/session store?

Yes it was a deliberate decision. There is some back story but the TL;DR is

• Kask predates the service mesh, needed TLS and so implemented itself
• Sessionstore is pretty critical and could benefit from not having the extra latency the service mesh possibly adds
• A change to sessionstore was somewhat high risk and altering the behavior of a well functioning component didn't seem like it had any kind of significant payoff back then.

If we're not using the mesh, kask should support reloading certs from disk on changes. With that we could just replace the cergen certs with auto-renewing ones like we do in the mesh.

Sure, but I don't know if it actually supports reloading certs from disk, adding @Eevans.

The machine has been reimaged and the instances bootstrapped. 🤞

The reimage is complete, and both instances have been bootstrapped. Closing.

• - access request (or expansion) has sign off of group approver indicated by the approval field in data.yaml

In T364584#9784601, @xcollazo wrote:

@Eevans if you can point us to any existing documentation about this staging Cassandra cluster I'd appreciate it.

In T355730#9781015, @Jgiannelos wrote:

I am trying to get access and face the following error:

> ssh cassandra-dev2001.codfw.wmnet
jgiannelos@cassandra-dev2001:~$ sudo -u cassandra_dev cqlsh cassandra-dev2001-a
sudo: unknown user: cassandra_dev
sudo: error initializing audit plugin sudoers_audit

In T362033#9758428, @Volans wrote:

Maybe a little drastic option, but could we try to reimage one of those 2 server and wait few days?
That will surely wipe clean any manual procedure that was carried on the host since the first disk swap. If it happens again is probably unrelated to anything done on the host and more likely pointing to some hardware issue or a more general software problem.

Ok @Jgiannelos this should now be setup:

The final changeset —r1026194— is up, but will need to pass review by Infrastructure Foundations (reviewed weekly, on Mondays).

Hi @jsn.sherman, You've been added to the deployment group, your shell username is jsn (same as wmf cloud). Let me know if you have any issues!

The rebuild is complete:

In T362033#9758428, @Volans wrote:

Maybe a little drastic option, but could we try to reimage one of those 2 server and wait few days?
That will surely wipe clean any manual procedure that was carried on the host since the first disk swap. If it happens again is probably unrelated to anything done on the host and more likely pointing to some hardware issue or a more general software problem.

@thcipriani It looks like you're the approver for group deployment... do you?

@jsn.sherman Could you please do one of the following? Either:

In T362697#9758025, @WDoranWMF wrote:

@Eevans thanks for following up on.

Did we leave Data_Gateway in an unclear space of ownership or where do it live? Is it on Data Product's side of the road(if so it is not currently anywhere on our list of all things).

If we can sort that out then I think we can roll on.

In T352647#9757647, @elukey wrote:

Restbase done!

Next steps:

• Rollout the new truststore for session store - do we need to schedule maintenance time and depool kask etc..?

In T362697#9754789, @VirginiaPoundstone wrote:

Since it is experimental, I do not think we should go in the HTTP Data Gateway direction at this time.

@WDoranWMF is this interesting to thinking through as a future release?

Ok, sdf has been replaced again, here is a transcript of what was done to add it back to the array:

Ok, so to summarize what has happened so far:

In T362697#9752938, @mforns wrote:

@Eevans yes, of course.
Let's freeze this task for now.
We will meet as a team, since these changes would require some rework.
We'll decide, and we'll get back to you with all the changes to this task and the design doc.
🙏

In T362697#9749806, @mforns wrote:

Yes @xcollazo I also wish I knew about this sooner.
And thanks @Eevans for the careful review!

As for the more technical deets, I have a few things here as well:

First, a lot of the metric values in your proposed schema use bigint, is a 64 bit signed integer warranted here, or would a 32 bit (int) value work?

I'd argue that for all the values that store pageview counts, I'd be safer to keep as bigint considering we do a bunch of rollups and pageview counts can be very big numbers. But agreed that many, like category_metrics_snapshot's media_file_count, could very well be ints. @mforns WDYT?

Agree! Everything that's not pageviews can be INT. I believe media file counts per category are not going over 1M so far, so we'd have quite some slack there. I see some pageview counts that reach 30+B though, so we need a BIGINT there.

In T362697#9749455, @xcollazo wrote:

@Eevans thank you for the extensive comments, and apologies for taking long to respond.

In T362697#9728035, @Eevans wrote:

In T362697#9719451, @xcollazo wrote:

[ ... ]
Second, a number of these tables use an attribute called top_data that is a JSON-encoded blob. Unless there is a very good reason, I would strongly discourage doing this. That serialized value is a part of the data model, a part that has been elided from this schema. As far as I can tell, each example of this could be modeled entirely in Cassandra to produce the same results. Take commons.top_pages_by_category as an example. If you change the table to this:

CREATE TABLE commons.top_pages_by_category (
    category            VARCHAR,
    category_scope      VARCHAR,
    wiki                VARCHAR,
    year                INT,
    month               INT,
    page_title          text,
    page_view_count     int,
    rank                int,
    PRIMARY KEY ((category, category_scope, wiki, year, month), page_title)
);

The same query —using predicates for category, category_scope, wiki, year, and month— will produce the results that you are JSON encoding.

I'll let @mforns expand/refute, but I think the rationale for encoding this inside the top_data was to have point queries and thus avoid range queries like the one you propose. If you believe Cassandra would be happy with range queries that could have up to 1000 rows, perhaps we should reconsider the schema indeed.

Restarted via the drac and everything seems OK now. I skimmed the logs and didn't see anything that seemed unusual prior to the event.

Also unable to login via the serial console.

The first device is done rebuilding:

Ok, the rebuild is complete.

2:23 PM <jclark-ctr> i am swapping sdf again
2:24 PM <jclark-ctr> swapped with one that was just erased

Having some trouble adding sdf2 back into the array: mdadm: Cannot open /dev/sdf2: Device or resource busy :/

In T362841#9741498, @Jclark-ctr wrote:

Looking at lshw.log and inventory on idrac it looks like all the drives are in order except sdf ,sdh are swapped in slots. after sdf rebuilds i can swap sdh

In T362841#9740826, @Jclark-ctr wrote:

@Eevans Replaced drive

In T355730#9740038, @Ladsgroup wrote:

I was asked to provide feedback from mariadb perspective (and how consistent we want to be across different technologies but in the same team).

We don't usually hand over dev accounts to the staging environment. Many development/staging work gets done on mariadb instance outside of production, most notably beta cluster (which has its own issues but I assume setting up a dedicated project for cassandra in cloud VPS and giving access to that wouldn't be too hard). Given that it's outside of prod, the impact of mistakes or compromise is quite limited, It also discourages "testing in production" situation. I know the staging cluster has different data but still it's in prod infra with all the complexities/downsides that it brings with itself.

Here is a transcript of everything done (for posterity sake):

In T362841#9737870, @Jclark-ctr wrote:

@Eevans this one is out of warranty also let me know if i am able to swap drive i can take care of in morning

In T355730#9736032, @Jgiannelos wrote:

Hi, is there any update with dev access for PCS devs?