Restarted via the drac and everything seems OK now. I skimmed the logs and didn't see anything that seemed unusual prior to the event.

Also unable to login via the serial console.

The first device is done rebuilding:

Ok, the rebuild is complete.

2:23 PM <jclark-ctr> i am swapping sdf again
2:24 PM <jclark-ctr> swapped with one that was just erased

Having some trouble adding sdf2 back into the array: mdadm: Cannot open /dev/sdf2: Device or resource busy :/

In T362841#9741498, @Jclark-ctr wrote:

Looking at lshw.log and inventory on idrac it looks like all the drives are in order except sdf ,sdh are swapped in slots. after sdf rebuilds i can swap sdh

In T362841#9740826, @Jclark-ctr wrote:

@Eevans Replaced drive

In T355730#9740038, @Ladsgroup wrote:

I was asked to provide feedback from mariadb perspective (and how consistent we want to be across different technologies but in the same team).

We don't usually hand over dev accounts to the staging environment. Many development/staging work gets done on mariadb instance outside of production, most notably beta cluster (which has its own issues but I assume setting up a dedicated project for cassandra in cloud VPS and giving access to that wouldn't be too hard). Given that it's outside of prod, the impact of mistakes or compromise is quite limited, It also discourages "testing in production" situation. I know the staging cluster has different data but still it's in prod infra with all the complexities/downsides that it brings with itself.

Here is a transcript of everything done (for posterity sake):

In T362841#9737870, @Jclark-ctr wrote:

@Eevans this one is out of warranty also let me know if i am able to swap drive i can take care of in morning

In T355730#9736032, @Jgiannelos wrote:

Hi, is there any update with dev access for PCS devs?

Hey @Jclark-ctr: I hope it's OK to assign this one to you as well.

In T362033#9707550, @Eevans wrote:

In T362033#9700949, @Jclark-ctr wrote:

@Eevans Hey looks like same drive as T354499 is failed again let me know if i can replace it again

Sure, go ahead.

P.S. I think this is the 4th time, are we just really unlucky, or is there some underlying factor at work?

In T362697#9728035, @Eevans wrote:

[ ... ]

Finally, something for you to consider: For image suggestions we created something we're calling Cassandra HTTP Gateway. The idea is that if you are persisting results exactly as you hope to return them, right down to attribute names, then we could give you an HTTP interface (REST) to retrieve them from. This wouldn't be a substitute for your service, but your service could make a simple HTTP request rather than directly query Cassandra. This should make things much simpler for you. The gateway returns JSON-encoded results as an object with a single attribute called rows, that is an array of JSON-encoded row objects ({rows: [{...}, {...}, ]}). For your case, we'd be able to set that up so that you could plug that array right into items: [] in your response objects. Let me know what you think!

In T362697#9719451, @xcollazo wrote:

@Eevans I believe you are the owner of the production Cassandra instance.

In T352647#9718748, @elukey wrote:

In T352647#9718545, @Eevans wrote:

I propose we carry on with the migration to PKI, accepting that Cassandra-based golang services will have to have verification disabled for now. It's not a regression, so I don't think we should let it hold up this work.

+1 makes sense, the plan is to enable host verification etc.. only after the move, so let's proceed! What is the best way forward? Complete AQS, do Restbase and then finally Session Store?

We've made the upgrade to 4.x already, and we did so without a migration. If I've understood the context above, that was the reason for elevating the priority, so I'm going to drop it down now. Please fee free to readjust if that's wrong.

We've encountered a problem enabling verification for gocql-based clients (see: T352647#9715110). We'll need to implement a custom HostDialer for Cassandra-connecting golang services before this work can continue.

In T352647#9718518, @Eevans wrote:

In T352647#9717916, @elukey wrote:

@Eevans sigh :( I found https://github.com/gocql/gocql/issues/1611 that may help, I didn't have time to check the code though.

Oh boy...

First, for those playing along at home, a brief summary of gocql/gocql/issues/1611 is that the list of discovered hosts used to populate gocql's connection pool comes from Cassandra's system.peers table —as IP address. The maintainers have fielded other issues that couldn't be solved as easily as resolving the IP to its FQDN, so they're opting toward a pluggable approach.

---

So gocql/gocql/issues/1611 is still open and hasn't been updated in a couple of years, but it gave rise to gocql/gocql/pull/1629 which was merged some time ago. They've implemented a DIY approach using an interface called HostDialer, basically a hook to override the driver's connection setup with code of your own. It landed in v1.2.0, and we're (for Kask at least) using v1.2.1, so all of the pieces should be there.

In T352647#9717916, @elukey wrote:

@Eevans sigh :( I found https://github.com/gocql/gocql/issues/1611 that may help, I didn't have time to check the code though.

In T352647#9715066, @Eevans wrote:

{"msg":"error: failed to connect to \"[HostInfo hostname=\\\"10.192.48.54\\\" connectAddress=\\\"10.192.48.54\\\" peer=\\\"10.192.48.54\\\" rpc_address=\\\"10.192.48.54\\\" broadcast_address=\\\"\u003cnil\u003e\\\" preferred_ip=\\\"\u003cnil\u003e\\\" connect_addr=\\\"10.192.48.54\\\" connect_addr_source=\\\"connect_address\\\" port=9042 data_centre=\\\"codfw\\\" rack=\\\"A_D\\\" host_id=\\\"5bfa3453-48f8-4c3c-82ea-478c460b6ee5\\\" version=\\\"v4.1.1\\\" state=UP num_tokens=256]\" due to error: x509: cannot validate certificate for 10.192.48.54 because it doesn't contain any IP SANs","appname":"sessionstore","time":"2024-04-15T18:25:35Z","level":"WARNING"}

In T362033#9700949, @Jclark-ctr wrote:

@Eevans Hey looks like same drive as T354499 is failed again let me know if i can replace it again

In T361964#9698601, @WDoranWMF wrote:

@Eevans medium is going to mean that it will likely only make it into a sprint at the end of this quarter. Is that ok or is this a risk?

In T360531#9666350, @BTullis wrote:

[ ... ]

So the question becomes, what shall we call this new chart? What about...

• http-gateway
• cassandra-druid-http-gateway
• aqs-http-gateway
• combined-http-gateway

In T334130#8771980, @BPirkle wrote:

Once the new AQS services are in production, will there be any on-going need?

Possibly. At least, there will be an on-going need for additional testing data. I've seen informal discussion of additional AQS endpoints, which would require additional data to develop against. It is also reasonable to assume that we'll at least occasionally find/create a bug for which we'd like additional tests to prevent future regressions, and for which we need additional test data to run local tests.

Now, that doesn't necessarily mean we need to continue extracting production data in the way we are now. Maybe there's another way to get the data. Or maybe we switch to using mock data representative of production rather than actual extracted data.

The original scope of this ticket was a very specific request to retrieve data, and that request as been met, so I'll close this ticket now.

In T320831#8585544, @mfossati wrote:

In T320831#8583283, @Eevans wrote:

We have a cluster that exists for testing, for some value of "testing". I think in this context the meaning is probably closer to staging than it is to experimenting, for which Cloud VPS is probably a better fit.
So for example, as a sink to receive data from a pipeline that hasn't yet been cleared for production, but maybe not the place to actively develop new applications against. At least, this is my current thinking. I'd like to hear what folks need first.

Fully agree, a staged pathway to production is exactly what we need.
In other words, something we can feel free to feed with data and to eventually wipe it clean.

So as a step 1, can you create a separate ticket with your requirements here?

Here you are: T328778: Cassandra test cluster as a staged pathway to production for image suggestions data pipelines

As a subtask for you @Eevans , T328670: Add section title column to image_suggestions.suggestions table schema would be the final requirement for production.

I think we can safely close this ticket as soon as the subtasks are resolved.

In T352647#9692816, @elukey wrote:

I tried to check the Cassandra AQS' clients and how they trust/validate TLS certificates. IIUC all the clients are on k8s and using the cassandra-http-gateway as chart, that renders a config file like /etc/cassandra-http-gateway/config.yaml containing various info about how to connect to a Cassandra cluster, and among those I found:

tls:
  ca: /etc/ssl/certs/wmf-ca-certificates.crt

This is great and it follows what we currently want on k8s and in production, namely that a daemon/service/etc.. that connects to Cassandra/Kafka/etc.. trusts the bundle composed by Puppet Root CA and PKI Root CA, so we can move towards cfssl freely. I am very puzzled with Cassandra since it uses ca-manger, and self-signed CAs IIRC, so those TLS certificates shouldn't be trusted by something that uses /etc/ssl/certs/wmf-ca-certificates.crt (the TLS connection should fail for TLS cert verification etc..).

I tried to dig a bit more and ended up in the generated-data-platform/aqs/device-analytics repo. Afaics we use github.com/gocql/gocql to manage TLS connections to Cassandra, and the only explanation that I can give is that we don't set either InsecureSkipVerify or EnableHostVerification so we skip TLS cert verification (see this commit).

If the above makes sense it simplifies a lot our work, since we are able to move Cassandra instances to cfssl without modifying any of the k8s clients, we just need to turn on the TLS cert validation once the cluster is fully migrated. Does it make sense?

Am I missing any other big clients that hit AQS?

I'm not sure what to make of the results of disabling read-repair. It did not stop the errors entirely, but we can't say there is no change either. The decommissions are now complete, which makes further experimentation difficult. I think CASSANDRA-19120 is the most promising thing, so I propose that we upgrade to Cassandra 4.1.5 when it becomes available, and leave this issue open until the next decommission is needed.

Done!

In T340105#8957849, @Tgr wrote:

One more hypothesis for PHP IMO is that it requires significantly less long-term maintenance effort than Node when base OS versions (and as a consequence language versions) change, because the library ecosystem tends to be more stable. (Not sure how to quantify and test that, though; it's just a personal impression based on how abandoned Node services fared. Also, no idea if the same is true for Go.)