In T227776#5398584, @Joe wrote:

I like the idea of having the ParserCache being a more generalized caching mechanism for MediaWiki. I have serious doubts about other things hinted here, specifically exposing a caching endpoint to other services. I'd argue that such a caching service should be separated from MediaWiki, have a simple API, and probably be structured around the page/revision identifier. We also probably don't want such a system to be written in PHP, as we would aim for the highest possible throughput.
.
We do not want an application doing some business logic to also be the cache storage for everything else. It was wrong with restbase, it would be wrong here. Each application should manage its own caching logic. This logic should not be delegated to another application and should not rely on the automagic properties of some centralized management system that then becomes the brain of the whole architecture. The only exception I see to this could be some purging logic.
.
So if we want such a system to be generalized and usable outside of MediaWiki it should be a thin service in front of a storage system[1] and it should:

• Have primitives that reproduce whatever API we use with e.g. BagOfStuff
• Be able to work across datacenters in write/write mode

In T227776#5345491, @daniel wrote:

In T227776#5339505, @Krinkle wrote:

What is the use case for external access?

To clarify: I was referring to access outside MW core, but inside the local (in our case, WMF) network. The intent is not to make this a public service that can be accessed directly by external clients.
Concrete use cases (some currently in core), for extracting data from page content, and caching it for later access: Wikibase constraint validation, graphoid, kartographer, mathoid, template data, page summary...

How would an external consumer deal with value validation (e.g. matches known rev id), fragmentation parameters

Ideally, the cache service itself would know about these things and handle them correctly. E.g. before returning a cache entry, it would check that it's not stale, and when purging the entry for a given page, it would purge the entire "bucket" of cached variants.

and how would it deal with absence of the value? -I see ParserCache as fundamentally a getWithSet-like interface (with very high persistence and poolcounter etc, but nonetheless fundamentally lazy-populated).

Currently, ParserCache isn't getWithSet. If there is no entry cached or the cached entry is stale, you get nothing back. Generating and then caching is the caller's responsibility.
For the new component described here, I'd propose to keep it that way. Generally, a component that accesses the cache (inside mw core or as a standalone service) would be using the cache for a kind of derived resource it knows how to generate.
The idea is: there would be one place to go to for getting rendered content, and one to go to for getting extracted infobox data, and one to go to for graphoid output, etc - and each of these places knows how to generate the derived resources, and uses the unified cache internally. This makes more sense to me than a generic end point for fetchi9ng any kind of resource, with some kind of internal routing to generate each resource.
When then should different components that derive different kind of things from pages share the caching infrastructure, instead of writing their own? Because the purging mechanism is the same, and the access keys are the same, and the scale is similar. Having to re-invent this wheel leads to duplication and annoyance, or the abuse of less-than-ideal mechanisms that exist, like page props.

In T227776#5328770, @Eevans wrote:

In T227776#5326474, @daniel wrote:

In T227776#5326455, @Anomie wrote:

Kask,[1] accessed via RESTBagOStuff?

Probably not Kask, but perhaps something similar, or a derivative or successor of Kask.

This sounds an awful lot like file storage (where I'm defining "file" to mean some semi-large (for definition of large) chunk of opaque data), which Kask (and Cassandra) aren't well suited for.

Though I'm not entirely sure that we want Cassandra as a backend for this.

Same.

An additional 2¢

During discussions w/ @Catrope, we determined that the impact of timestamp misses were sufficiently minor as to not justify us spending the time to write, test, and debug a migration of data from Redis. What we will do instead: Deploy with a MultiWriteBagOStuff that wraps the new and old store (read-from-new, fallback-to-old, write-to-both). There will be some 90 days or more between this deployment and the decommission of Redis, during which time most active users will have seen-times seeded into the new store.

Summarizing an IRC discussion: @Catrope will pick this up mid-November(ish), and we'll target deployment for sometime after the November freeze (27th–29th), and before the December freeze (December 23rd-January 3rd).

In T222851#5624749, @Eevans wrote:

I believe this task to be done.

I believe this task to be done.

@Catrope given the difficulty in doing a phased migration (vis-a-vis global notifications), do you have any objections to transitioning to the new store in a single go (i.e. updating it to be the default)? We accidentally did this yesterday, and TTBMK, there were no issues. We'd of course keep the multi-write configuration in place for the time being, so if rolling back were necessary, up-to-date seen times would continue to exist in Redis.

In T92471#5615626, @WDoranWMF wrote:

@Eevans Should we move this to the Icebox?

To summarize a conversation with @Catrope on IRC:

The config change was rolled out in SWAT today. It was rolled back a short time later, out of an abundance of caution, because it seemed to apply more broadly than just testwiki (our expectation). During the time it was deployed, we were seeing ~1k/s requests. Sampling the records in Cassandra, the vast majority were of the form global:echo:seen:{alert,message}:time:%d. A number of them however were for enwiki, ruwiki, idwiki, outreachwiki, wikidatawiki, and others.

This is now complete.

In T230848#5556127, @eprodromou wrote:

@mobrovac We already have a hierarchy for getting the content of the file. This is the metadata endpoint. I'll update the user story to reflect that.

In T230848#5556127, @eprodromou wrote:

@mobrovac We already have a hierarchy for getting the content of the file. This is the metadata endpoint. I'll update the user story to reflect that.

This is now done: See https://logstash.wikimedia.org/app/kibana#/dashboard/AW3go_uCx3rdj6D8q-he & https://grafana.wikimedia.org/d/IfJykaTZk/echostore

I believe this is complete.

In T234376#5582893, @Joe wrote:

Heh yes sorry, I forgot to tell you yesterday - you need to use helmfile destroy in newer versions of helmfile.

Hat tip to @CDanis who pointed me at https://github.com/helm/helm/issues/3208#issuecomment-348154521; A helm delete production --purge did the trick.

From a conversation w/ @Joe on IRC, it seems the nodeAffinity section (copypasta from the sessionstore deployment) was likely causing the problem. I issued a helmfile delete, and updated the config (removing that section), but am now getting:

I'm unable to deploy to codfw; I'm seeing the following:

In T235299#5577226, @Eevans wrote:

In T235299#5569440, @mobrovac wrote:

Instead of having full templates in roles profiles, we could have an add_user.pp manifest in modules/cassandra that compiles it for the profile, so a structure with the relevant info could be passed, something like:

[
  {
    title => 'title1',
    user => 'user1',
    pass => 'pass1',
    keyspaces => ['ks1', 'ks2']
  },
  {
    title => 'title2,
    user => 'user2',
    pass => 'pass2',
    keyspaces => ['ks3', 'ks4']
  },
]

Compiling the CQL from these statements and writing it to disk on the target nodes would be rather straightforward. We could have a special 'all' argument for keyspaces for cases that remain unchanged, like AQS or Maps.

This might work. It would need to be more robust than this though. For example: (for historical reasons) RESTBase includes GRANTs for CREATE, ALTER, and DROP, but that would not be the case typically. Come to think of it, we could probably remove those for RESTBase now and standardize on an assumption of SELECT and MODIFY, but there may come a day when those assumptions don't hold anymore either.

In T235299#5569440, @mobrovac wrote:

Instead of having full templates in roles profiles, we could have an add_user.pp manifest in modules/cassandra that compiles it for the profile, so a structure with the relevant info could be passed, something like:

[
  {
    title => 'title1',
    user => 'user1',
    pass => 'pass1',
    keyspaces => ['ks1', 'ks2']
  },
  {
    title => 'title2,
    user => 'user2',
    pass => 'pass2',
    keyspaces => ['ks3', 'ks4']
  },
]

Compiling the CQL from these statements and writing it to disk on the target nodes would be rather straightforward. We could have a special 'all' argument for keyspaces for cases that remain unchanged, like AQS or Maps.

In T200803#5564291, @Eevans wrote:

If I hear no objections, I will upgrade a canary node in each datacenter of the RESTBase cluster on Monday, and plan to upgrade the remaining nodes on Tuesday if everything checks out.

In T234464#5575683, @Joe wrote:

I've done all the puppet/dns prep work. You can now proceed to prepare this new kask deployment in operations/deployment-charts.
[ ... ]

/cc @mobrovac

/cc @elukey, @Joe

T235299: Cassandra cluster management support for multi-tenancy has been created for follow-up, this issue is otherwise complete.

The session storage cluster has been upgraded, and things look Good. If I hear no objections, I will upgrade a canary node in each datacenter of the RESTBase cluster on Monday, and plan to upgrade the remaining nodes on Tuesday if everything checks out.

The following has been created on the RESTBase cluster:

Applicable staging/test environments have been updated to 3.11.4 and no issues are apparent. If this continues to look OK, and there are no objections, I will upgrade the production sessionstore cluster tomorrow (technically, it is not yet in production).j

In T234928#5559579, @mobrovac wrote:

Ha! I think I found the bug. The problem is that when a user first opens the page and starts editing with VE, then VE calls /page/html/{title}?stash=true, but RESTBase expects the revision ID to be present as well, which causes it to store the page under the key {title}:undefined:{tid} instead of {title}:{revid}:{tid} in the stash bucket. However, if the user tries to VE-edit a page that they edited before (i.e. it was already loaded, because they have already edited it either via VE or the wt editor), then VE calls /page/html/{title}/{revision}?stash=true.
I have deployed the fix as well as a fallback to look for {title}:undefined:{tid} in case the original stash could not be found, and so far so good - no transforms are failing! I will keep the task open and monitor for a little while longer to ensure this is truly the case (and that no other edge cases materialise).

Additionally, a dedicated application user should be created, with corresponding access rights to the allocated table.

In T234928#5555665, @mobrovac wrote:

@Pchelolo @Eevans could you take a look? Perhaps I've overlooked something, but to me this is starting to smell like Cassandra is losing data somehow.

In T209110#5554361, @akosiaris wrote:

We've been calling this out as a blocker to moving session storage to production, so I guess what I'm trying to determine is: Are we still blocked?

For a few more days, it looks like yes.

I think this means the block has been removed.

I think there were (implicitly) two issues related to this open task: a) a superfluous log message (aka log spam), and b) unstructured log messages. The latter is now solved, the former is not. Possible options for addressing (a):

Deployed to staging, the log output now looks like: