In T270225#6927504, @Krinkle wrote:

In T270225#6927436, @tstarling wrote:

[ ... ]
After reading T206010, T206015 and the comments above, I don't understand why session deletions are different to session writes.

Me neither. I vaguely recall hearing in a multi-dc strategy meeting that session writes mostly or only happen during POST web requests, which made us not look too closely at the fact that the new Session store does not offer synchronous writes cross-dc.

But, the MediaWiki code has indeed expected for many years to perform SYNC_WRITES, and afaik that is what @aaron preferred we keep. But when the Session storage RFC (T206010) was writtne, it proposed to reduce this to local-dc quorum with eventual consistency, except for deletes. I'm not sure why, and I'm unable to find a discussion about this. I suppose @Eevans might know it this was a trade-off and intentional change.

I agree: The entire purpose of the API gateway was to build apps on, and readers like the iOS and Android apps were the most cited examples. If we can't eat our own dog food, then that's an indication of a pretty serious problem. Until we figure that out, work-arounds are just going to add tech dept to a system we built to try to clean up tech debt.

I'm not sure I properly parse/grok all of this but, I can confirm that Kask does not directly expose Cassandra consistency to callers; The assumption is that all reads, and all writes, will utilize a quorum of nodes in the local datacenter. Writes are still replicated to the remote datacenter(s), but asynchronously, so all the usual caveats apply. Deletes utilize a quorum of nodes from each datacenter, so a successful operation does guarantee that subsequent reads will fail.

At the very least, getting rid of these names would create inconvenience. There are lots of examples of maintenance and admin commands that run against an instance, or resolve IPs in output, and having to map IP addresses -to- instance won't be much fun.

In T258414#6503669, @Eevans wrote:

The following list of keyspaces seem to correspond with these ghost snapshots.

Allegedly obsolete keyspaces

commons_T_parsoid
commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
enwiki_T_parsoid
enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
enwiki_T_references
others_T_parsoid
others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
others_T_references
wikipedia_T_parsoid
wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
wikipedia_T_references

@Pchelolo can you check me on this before I go around rm -r-ing stuff?

The following list of keyspaces seem to correspond with these ghost snapshots.

It seems as though we do in fact have some snapshot data:

In T261512#6480172, @hnowlan wrote:

Which racks should these new hosts go onto? I am guessing one into each of a b and d, which are the eqiad racks atm (all at 4 hosts currently).

In T256863#6365705, @Eevans wrote:

In T256863#6365459, @hnowlan wrote:

I'm looking into this today - I see that restbase2009 is up 9 days, has been configured by puppet and added to the Cassandra cluster but I don't see anything in SAL about who did it. Still investigating

eevans@restbase2009:~$ c-any-nt status -r | grep 2009
UN  restbase2009-a.codfw.wmnet  554.01 GiB  256          6.9%              c0d7a947-d423-49b3-b307-416a783a722f  d
UN  restbase2009-b.codfw.wmnet  523.81 GiB  256          6.6%              3ec9435b-dc45-46d3-a2ea-d0d5153615a9  d
UN  restbase2009-c.codfw.wmnet  470.2 GiB  256          6.5%              2e4e1268-1e17-476f-aa9e-b8c42035d115  d
eevans@restbase2009:~$

Umm, wow.

TL;DR It wasn't me. :)

In T256863#6365459, @hnowlan wrote:

I'm looking into this today - I see that restbase2009 is up 9 days, has been configured by puppet and added to the Cassandra cluster but I don't see anything in SAL about who did it. Still investigating

Setting aside how antithetical this requirement seems for a Wikimedia project...

In T256863#6315661, @wiki_willy wrote:

In T256863#6315360, @Eevans wrote:

In T256863#6313866, @wiki_willy wrote:

Hi @Eevans - it looks like this was originally scheduled to be refreshed this fiscal year during the annual CapEx planning, but then someone decided to push out the refresh until FY21-22. Can this be decommissioned, since we're towards the end of the 5yr server life cycle? Thanks, Willy

Do you mean... indefinitely?

Yeah. Unfortunately, this server is about 5yrs old now, and out of warranty. @Eevans - will you be able to get by without having this system in rotation, until it's time to refresh it? On line 51 for this year's CapEx budget sheet below, I see there's a comment there to postpone the refresh until FY21-22 along with the rest of the batch.

In T256863#6313866, @wiki_willy wrote:

Hi @Eevans - it looks like this was originally scheduled to be refreshed this fiscal year during the annual CapEx planning, but then someone decided to push out the refresh until FY21-22. Can this be decommissioned, since we're towards the end of the 5yr server life cycle? Thanks, Willy

I have a local branch where I ported Kask to Buster (as I recall, there were one or two minor changes to the APIs of dependencies). I'll dig that up.

In T224041#6229281, @jeena wrote:

In T224041#6227592, @akosiaris wrote:

However, up to now we did not have a need to publish the test images to the registry but still run integration tests via helm test (IIRC mathoid runs them). What has changed?

I might have misunderstood that the team wanted to run the functional and integration tests that are in the kask repo written in go.

I just took a closer look at service-checker and I think it would be able to replicate the go integration tests, but only the get and post, not the delete ones, so if that's sufficient, then you are right, we wouldn't need to publish any additional images or make changes to the kask chart (newly added to task checklist).

any information included in JWTs is publicly visible to anyone with access to the JWT. Because the JWT is signed, the rate limiting information cannot be modified/hacked. But it can be seen. Therefore, care should be taken regarding claims that include confidential or non-public information. This is somewhat mitigated by the fact that JWTs primarily function as access tokens. They are therefore exchanged over HTTPS, are not publicly logged, and non-authorized parties having access to them would already constitute a security issue.

In T209106#6190315, @Pchelolo wrote:

Anything left to do here?

I think we can close it.

In T222990#6190344, @Pchelolo wrote:

Anything left to do here?

echoseen in deployment-prep doesn't do HTTPS

In T249756#6145067, @elukey wrote:

@Eevans please be patient, me and Joseph had a long chat about upgrading in place and there are some doubts that we have. Overall, what we'd like to do is something like:

In T249756#6132278, @elukey wrote:

Before starting, there are some notes to keep in mind:

• we have currently 6 nodes running cassandra 2.2
• 3 of them are due to be refreshed due to hw warranty expiration
• we have in mind to expand the cluster to 9/12 nodes to host more data (needs to be verified)
• the goal is to upgrade to Cassandra 3.11 (already running for Restbase)

We have a couple of options, not sure about their flexibility/etc.. yet:

• Upgrade the cluster in place

This is something that wasn't tested by the Services team at the time, since a new Restbase cluster was created. The idea should be to upgrade one node at the time, and call nodetool sstableupgrade (it takes a long time to complete usually).
We could take the current cluster, upgrade it in place, and then add/remove new nodes later on.
The in place upgrade looks appealing, but if we hit bugs half way through we could end up in a weird state (recovering may not be trivial at this point).

• Create a new cluster and stream sstables content to it

This could be doable with sstableloader, but there are some question marks about streaming sstables v2.2 to a 3.11 cluster (should we use sstableloader v3.11 on a 2.2 node? Is it sufficient to use sstableloader 2.2 on a 2.2 node, stream to a 3.11 node and then run nodetool sstableupgrade on it? etc..).

In T251063#6085776, @BPirkle wrote:

There is related discussion at https://www.mediawiki.org/wiki/Topic:Vie8y5khj6w3qs3y

The defining question for me, which also came up in the discussion linked above, is whether we are building only an API Portal, or whether we are building a Developer Portal and we just happen to be starting with the API portion because that's what we're working on right now.

If we're building a Developer Portal, the name developer.wikimedia.org makes sense to me. If we're just building an API Portal and stopping there, then the name api.wikimedia.org makes sense to me. My understanding is that we're building just an API Portal, and I don't see any technical showstoppers for using the name api.wikimedia.org, so that's the name I prefer.

AFAIK, this is complete

In T250050#6071807, @Papaul wrote:

@Eevans /dev/sdc has been replaced. Let me know if you have any questions

I think this popped up when Puppet was re-enabled (which ironically is failing anyway because of the failed unit). It's been put under maintenance.

In T250050#6055117, @Papaul wrote:

@Eevans the IDRAC is not showing any failed drive. Is it possible for you to get me some system logs showing the bad disk so i can upload that when i ask for a disk replacement. The last log i have for this system from the IDRAC is from 2018 .

Also I need to clear the log and upgrade the firmware on this system

BIOS Version 1.5.6
iDRAC Firmware Version 3.21.21.21

new verison

BIOS Version 2.5.4
iDRAC Firmware Version 4.10.10

please let me know when i can do this.

Thanks

In T250050#6055022, @Eevans wrote:

[ ... ]

Once complete we'll need to do the b & c instances as well. Once that is complete, we can either re-image the node entirely, or replace the SSD, rebuild the arrays, and then completely wipe Cassandra state (I'd prefer the former for repeatability sake, but defer to SRE here). I'll update the ticket when we're at that point.

OK, so it seems like we have a failed SSD (/dev/sdc), and as a result, some degraded arrays. Ideally we'd be able to replace the SSD and rebuild the array, but we are using the /dev/sd[x]4 partitions on these machines as a JBOD for Cassandra. Unfortunately, it distributes its own system tables over these devices as well, and isn't recoverable after losing a chunk of them like this.

In T250050#6051028, @elukey wrote:

@Eevans this is the weekend of broken cassandra hosts, adding you as FYI :)

In T235437#6046252, @Mholloway wrote:

Ah, I see. My interest is specifically in service-runner as my understanding is that it will continue to be used by most or all Wikimedia Node.js services. I'm currently working on updating an open-source Node.js service for Wikimedia production and I'm wondering (a) if I should plan to incorporate service-runner as part of that work, and (b) if so, whether I can plan to use service-runner's existing rate-limiting facility or I should plan to look elsewhere for that.

Done.

Ok, these keyspaces have been removed from production, dev, and deployment-prep. Out of an abundance of caution, I will leave this open until Friday, and close after cleaning up the snapshots.

In T248018#6000114, @Pchelolo wrote:

LGTM. Beta cluster has one as well

In T248018#5999449, @Pchelolo wrote:

This has been deployed, so we can drop those key spaces!

In T239856#5966749, @Pchelolo wrote:

Is there anything left to do/review here?

In T243544#5947739, @hnowlan wrote:

I've overhauled things and moved stuff to a more Debian-compliant layout here: https://github.com/nosmo/cpp-driver/tree/debian/debian
Still not sure if this is up to snuff though, needs some testing.

In T137419#5935635, @Aklapper wrote:

In T137419#4395956, @Eevans wrote:

This is actually something we should be following up on with upstream more aggressively.

@Eevans: Hi, do you plan to do this? Asking as you you have been task assignee for a while now.

Yeah, it's ready to be closed, but AFAIK, we're supposed to wait for the PM (@CCicalese_WMF) to close it after moving it to Done on the workboard.

I believe this is pending https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/574034 (T224712).

I don't think this is sessionstore, (at least, it's not the timeout issue with Cassandra that we saw before).

In T244508#5866291, @Dzahn wrote:

@Eevans You should have +2 on the mw-config repo now. Probably after logging out and back in.