In T222907#5202665, @BPirkle wrote:

[ ... ]

From the task description:

Put another way, what is at question is NOT whether arbitrary TTLs are possible, but whether the client can override the default with something shorter.

The ability to configure session TTLs is baked in to our current SessionManager/SessionBackend/BagOStuff implementation, and nothing I do in RESTBagOStuff will eliminate that configuration setting. So it will be possible for someone to *think* they are configuring an arbitrary TTL on the client (wiki) side. But we can, of course, ignore that setting. And in fact, we currently are ignoring it - the current RESTBagOStuff implementation never even sends it to Kask.

So if it suffices to provide a consistent TTL across all sessions on all wikis (and it appears it does) then I think that we are, technically speaking, done.

However, it feels sketchy to me that it is possible to configure a TTL that gets silently ignored. Someone might set $wgObjectCaacheSessionExpiry on a wiki that uses RESTBagOStuff and reasonably expect it to do something. But it would not.

In T222990#5200565, @kchapman wrote:

@Eevans is this a task for you or were you looking for input from @EvanProdromou ?

Not production; This host can be taken down at any time, without coordination.

In T215533#5185925, @BPirkle wrote:

[ ... ]
However, if there's no compelling need to do Basic Auth, I'd vote to support the option that it sounds like both RESTBagOStuff and Kask already support.

Additional notes and considerations:
[ ... ]

• The test runner should not hard code any knowledge about the MediaWiki action API, and should be designed to be usable for testing other APIs, such as RESTbase.

In T219831#5173655, @Eevans wrote:

In T219831#5173498, @sbassett wrote:

[ ... ]
Policy Compliance
I am assuming data will live in Cassandra and within relevant logs in accordance with Wikimedia's existing data retention guidelines.

No PII should ever by logged, but session data is persisted in Cassandra. Sessions expire well before the max retention period, but that data is not immediately removed from storage, it's GC'd by compaction (as dictated by the compaction algorithm in use) after a moratorium (defaults to 10 days). I would be really surprised if that process were to drag out past 90 days (or anything even close to 90 days), but it might be worth putting a pin in this, and conduct an audit at some point to verify just how long they do hang around.

This is awesome @sbassett; Thanks for looking it over! I left some comments inline:

FYI, I think CentralAuth needs a TTL different to that of regular sessions (IIRC, this was cited as a reason for exposing the TTL). Regardless of whether or not this is true, we should probably consider using a different instance of the service for CentralAuth (it's trivial to do). If we do, we can configure it with a different default TTL as well.

This task is to check if this is needed now, or ever.

In T222908#5170802, @EvanProdromou wrote:

I'll check through our session-handling code to see if we ever use this type of call (I think it's add() in BagOStuff) for sessions, and if it seems like it's necessary at that point.

Off my dome, I could see using this type of write when you're worried about propagation delays between data centres. One scenario:

1. Client connects to server in DC1 and gets a session ID AAAA
2. Application server in DC1 unconditionally writes an empty session object with key AAAA to DC1 cluster
3. User authenticates to application server in DC1
4. Application server in DC1 unconditionally writes a session object with user ID with key AAAA to DC1 cluster
5. Client connects to application server in DC2, with session ID AAAA
6. Application server in DC2 reads session with key AAAA from DC2 cluster, gets no value
7. Empty session object with key AAAA propagates from DC1 cluster to DC2 cluster
8. Session object with user ID and key AAAA propagates from DC1 cluster to DC2 cluster
9. Application server in DC2 unconditionally writes empty session object for key AAAA to cluster in DC2
10. Empty session object with key AAAA propagates from DC2 cluster to DC1 cluster

In T215533#5170933, @EvanProdromou wrote:

One thing I want to make sure of is that KaskBagOStuff is compatible with the TLS setup defined in T209109, as well as supporting username/password HTTP Basic authentication.

According to the site, the benefits of OpenAPI are...

In T221292#5145019, @EvanProdromou wrote:

@Eevans maybe the mystery of the longer requests is just that they're writes instead of reads?

In T221292#5144700, @EvanProdromou wrote:

Also, is this task done when we're within the boundaries of T211721 or is the task just to have a tool to do measurements?

In T211721#5145107, @EvanProdromou wrote:

On a related note, do we want or need an SLA on consistency?

In T211721#5145739, @aaron wrote:

[ ... ]

... My understanding is session writes would be visible in all DCs once an HTTP response is sent (at least the old security team wanted that).

In T211721#5144702, @mobrovac wrote:

In T211721#5144690, @Eevans wrote:

In T211721#5144537, @EvanProdromou wrote:

@Eevans have we been considering cross-DC writes in the performance testing? Are we comparing apples to apples here?

Cross-DC only occurs for DELETE operations and should be identical to POST, plus whatever inter-DC latency is. I have not included them in performance testing thus far because the focus (thus far) has been on Kask performance (and any fixes/optimizations needed there), and including DELETE would only provide a measurement of Cassandra over the WAN.

Why would there be cross-DC requests to Kask at all? MW should send requests only to Kask in the local DC, and then it's up to Cassandra to propagate the change to the other DCs. Or am I missing something?

In T211721#5144537, @EvanProdromou wrote:

@Eevans have we been considering cross-DC writes in the performance testing? Are we comparing apples to apples here?

In T221292#5144480, @EvanProdromou wrote:

@Eevans should we break out a separate task for identifying the cause of the extra 40-50ms delay?

In T220246#5125453, @Joe wrote:

In this scenario, our service "foo" is already accessing table "foo.meta" and we're currently adding a column to the table. The process of altering the schema and upgrading the application need to be able to be performed asynchronously. because neither schema changes nor application deployments are atomic. Let's assume we do what MediaWiki does, that is make the code deployable with or without the new schema, and protecting the use of the new features behind a feature flag. In this scenario:

1. Code (and the proposed alter) are written, deployed to production
2. This is just a schema change, so the cql command for the schema change can be issued whenever we want - even as a final step of deployment - directly by the deployer.
3. The feature flag gets flipped in a subsequent deployment when it's assured the schema change has happened everywhere.

In T220401#5134204, @akosiaris wrote:

@Eevans @Clarakosi chart has been merged and is published. The only thing missing before we can move on to the deployment is the swagger/openapi spec so that service-checker[1] can run and monitor this service.

[1] https://github.com/wikimedia/operations-software-service-checker

Considering this has since been scoped as being just the Cassandra cluster, I believe we can call it complete.

In T219831#5135390, @JBennett wrote:

@Eevans @Clarakosi This one is a bit out of our wheelhouse and something we can provide only a cursory review of. I'd like to propose the following: The Security team can perform a basic security review of this but I would recommend a secondary review by our 3rd party partners at BishopFox. That said, the 2nd review would incur some cost but I'm not sure how much. Do you have any budget available for something like that??

Production environment {{done}}

Dev environment {{done}}

I cannot speak to the keyspaces being dropped, but the statements LGTM!

Done.

In T221528#5127730, @mobrovac wrote:

Here are the creation statements for dev and production:

[ ... ]

Please review.

Details on test methodology, and some initial results are forthcoming, but to elaborate on comments made elsewhere:

Applied to the dev cluster; Done

This has been applied to the production cluster

In T221031#5113112, @Pchelolo wrote:

LGTM!

Let's first execute in deployment-prep and follow into production after we merge/deploy https://github.com/wikimedia/restbase/pull/1117 in beta to verify all good and correct?

Deployment-prep:

Production/staging DDL here

In T215533#5107326, @BPirkle wrote:

I have, on my local development wiki, a working KaskBagOStuff (naming suggestions welcome) that communicates with the development instance at deployment-sessionstore01.deployment-prep.eqiad.wmflabs (via some tricky port forwarding) and successfully allows basic session management for happy login and logout.

Out of curiosity, what is it that prevents you from using RESTBagOStuff?

RESTBagOStuff is hard-coded to use PUT, while the API uses POST. Also, RESTBagOStuff is hard-coded to use PHP serialization for the value data, which I'm guessing nobody is in favor of. (Well, Joe wanted "json storing base64-encoded serialized php"... <grin>)

I rearranged things a bit so that RESTBagOStuff and KaskBagOStuff share almost all their code, except overrides for the essential differences, so there's not really going to be a lot of new code for this, mostly just some shuffling.