TTBMK, everything here is done.

Done.

Done.

In T242461#5798385, @Eevans wrote:

In T242461#5797481, @akosiaris wrote:

In T242461#5793618, @Pchelolo wrote:

Since (long-term) we aim to replace all of this, is abandoning it entirely an option?

Is it possible to take it out for now until we either prioritize it again or drop it entirely?

You mean undeploy? Sure we can undeploy it. The only caveat being that redeploying it will take some time as we will need to create the necessary resources again (LVS entries, DNS, kubernetes namespaces etc).

We're running CI for RESTBase in both RESTBase and RESTRouter modes, so it will be in mostly deployable state if we want to put it back online, however maintaining an unused production deployment seems like a waste.

Indeed.

A lot has changed since we began this migration, including https://www.mediawiki.org/wiki/Core_Platform_Team/Decisions_Architecture_Research_Documentation/Services_Architecture_Recommendations_(2019), which is expected be a lengthly process, but will ultimately result in REST{Router,Base}-less world. I guess the question we should be asking is: Is this still something we should do in the meantime (and schedule and resource to complete), or should we cut bait, undeploy from k8s, and leave things as they are?
@WDoranWMF ?

In T242461#5797481, @akosiaris wrote:

In T242461#5793618, @Pchelolo wrote:

Since (long-term) we aim to replace all of this, is abandoning it entirely an option?

Is it possible to take it out for now until we either prioritize it again or drop it entirely?

You mean undeploy? Sure we can undeploy it. The only caveat being that redeploying it will take some time as we will need to create the necessary resources again (LVS entries, DNS, kubernetes namespaces etc).

We're running CI for RESTBase in both RESTBase and RESTRouter modes, so it will be in mostly deployable state if we want to put it back online, however maintaining an unused production deployment seems like a waste.

Indeed.

It's not clear to me what the status of this is. Do we need to deploy the latest code here? Since (long-term) we aim to replace all of this, is abandoning it entirely an option?

The tables have been dropped in all 3 environments. The only thing remaining is to clear the snapshots (and actually reclaim the space). Out of an abundance of caution, I'll sit on this for a couple days and close the ticket once complete.

OK, here is what I propose applying; Review appreciated!

In T228294#5782442, @eprodromou wrote:

This seems in our wheelhouse.

• Once we have a packaged driver, what would we do with it?

In T238580#5710739, @Eevans wrote:

In T238580#5709953, @RobH wrote:

Also note I assumed details for the racking/hostnames and would appreciate confirmation of those details in task description, thanks!

This cluster uses a replication count of 3 (per-DC), and for eqiad we have machines evenly distributed over a, b, and d. This replica-to-row affinity makes it very nice to reason about where data will be moving from/to on topology changes and it would be a shame if we lost that now. Will there be a problem keeping these to the same 3 rows currently in-use?

This is now done. Sorry for the long delay.

In T122825#5752420, @Joe wrote:

I think most of the issues described here have been in the meantime solved by the implementation of the code stewardship review process and a list of developers/maintainers. @Pchelolo @Eevans @Clarakosi any opinions?

In T218609#5666608, @Krenair wrote:

@Eevans: It has been 6 months, please respond.

In T222851#5626849, @Eevans wrote:

Summarizing an IRC discussion: @Catrope will pick this up mid-November(ish), and we'll target deployment for sometime after the November freeze (27th–29th), and before the December freeze (December 23rd-January 3rd).

In T236113#5707312, @eprodromou wrote:

We discussed this in our kickoff meeting today.
There was a lot of resistance to the idea of having an endpoint for OpenAPI 3.0 definitions of the (other) endpoints. I like the idea of using OpenAPI since there are a lot of other tools that would benefit, such as client code generators.

This was deployed during SWAT. See: https://logstash.wikimedia.org/goto/a61eb70c51d26b11835e4bb4caadda0b

Since an upgrade to Cassandra 3.11.4 was pending anyway (T200803: Upgrade Cassandra 3.11.2 clusters to 3.11.4 (bugfix release)), we prioritized that work a) in the event the issue had been fixed upstream, and b) so that if we had to dig deeper to troubleshoot, that we would be doing so against a current release. Unfortunately, this does not seem to have fixed (see: T238591), and we will indeed need to dig deeper.

This looks suspiciously similar to T231027, right down to the read timeout exceptions during read-repair that precede the event:

In T227776#5398584, @Joe wrote:

I like the idea of having the ParserCache being a more generalized caching mechanism for MediaWiki. I have serious doubts about other things hinted here, specifically exposing a caching endpoint to other services. I'd argue that such a caching service should be separated from MediaWiki, have a simple API, and probably be structured around the page/revision identifier. We also probably don't want such a system to be written in PHP, as we would aim for the highest possible throughput.
.
We do not want an application doing some business logic to also be the cache storage for everything else. It was wrong with restbase, it would be wrong here. Each application should manage its own caching logic. This logic should not be delegated to another application and should not rely on the automagic properties of some centralized management system that then becomes the brain of the whole architecture. The only exception I see to this could be some purging logic.
.
So if we want such a system to be generalized and usable outside of MediaWiki it should be a thin service in front of a storage system[1] and it should:

• Have primitives that reproduce whatever API we use with e.g. BagOfStuff
• Be able to work across datacenters in write/write mode

In T227776#5345491, @daniel wrote:

In T227776#5339505, @Krinkle wrote:

What is the use case for external access?

To clarify: I was referring to access outside MW core, but inside the local (in our case, WMF) network. The intent is not to make this a public service that can be accessed directly by external clients.
Concrete use cases (some currently in core), for extracting data from page content, and caching it for later access: Wikibase constraint validation, graphoid, kartographer, mathoid, template data, page summary...

How would an external consumer deal with value validation (e.g. matches known rev id), fragmentation parameters

Ideally, the cache service itself would know about these things and handle them correctly. E.g. before returning a cache entry, it would check that it's not stale, and when purging the entry for a given page, it would purge the entire "bucket" of cached variants.

and how would it deal with absence of the value? -I see ParserCache as fundamentally a getWithSet-like interface (with very high persistence and poolcounter etc, but nonetheless fundamentally lazy-populated).

Currently, ParserCache isn't getWithSet. If there is no entry cached or the cached entry is stale, you get nothing back. Generating and then caching is the caller's responsibility.
For the new component described here, I'd propose to keep it that way. Generally, a component that accesses the cache (inside mw core or as a standalone service) would be using the cache for a kind of derived resource it knows how to generate.
The idea is: there would be one place to go to for getting rendered content, and one to go to for getting extracted infobox data, and one to go to for graphoid output, etc - and each of these places knows how to generate the derived resources, and uses the unified cache internally. This makes more sense to me than a generic end point for fetchi9ng any kind of resource, with some kind of internal routing to generate each resource.
When then should different components that derive different kind of things from pages share the caching infrastructure, instead of writing their own? Because the purging mechanism is the same, and the access keys are the same, and the scale is similar. Having to re-invent this wheel leads to duplication and annoyance, or the abuse of less-than-ideal mechanisms that exist, like page props.

In T227776#5328770, @Eevans wrote:

In T227776#5326474, @daniel wrote:

In T227776#5326455, @Anomie wrote:

Kask,[1] accessed via RESTBagOStuff?

Probably not Kask, but perhaps something similar, or a derivative or successor of Kask.

This sounds an awful lot like file storage (where I'm defining "file" to mean some semi-large (for definition of large) chunk of opaque data), which Kask (and Cassandra) aren't well suited for.

Though I'm not entirely sure that we want Cassandra as a backend for this.

Same.

An additional 2¢

During discussions w/ @Catrope, we determined that the impact of timestamp misses were sufficiently minor as to not justify us spending the time to write, test, and debug a migration of data from Redis. What we will do instead: Deploy with a MultiWriteBagOStuff that wraps the new and old store (read-from-new, fallback-to-old, write-to-both). There will be some 90 days or more between this deployment and the decommission of Redis, during which time most active users will have seen-times seeded into the new store.

Summarizing an IRC discussion: @Catrope will pick this up mid-November(ish), and we'll target deployment for sometime after the November freeze (27th–29th), and before the December freeze (December 23rd-January 3rd).

In T222851#5624749, @Eevans wrote:

I believe this task to be done.

I believe this task to be done.

@Catrope given the difficulty in doing a phased migration (vis-a-vis global notifications), do you have any objections to transitioning to the new store in a single go (i.e. updating it to be the default)? We accidentally did this yesterday, and TTBMK, there were no issues. We'd of course keep the multi-write configuration in place for the time being, so if rolling back were necessary, up-to-date seen times would continue to exist in Redis.

In T92471#5615626, @WDoranWMF wrote:

@Eevans Should we move this to the Icebox?

To summarize a conversation with @Catrope on IRC:

The config change was rolled out in SWAT today. It was rolled back a short time later, out of an abundance of caution, because it seemed to apply more broadly than just testwiki (our expectation). During the time it was deployed, we were seeing ~1k/s requests. Sampling the records in Cassandra, the vast majority were of the form global:echo:seen:{alert,message}:time:%d. A number of them however were for enwiki, ruwiki, idwiki, outreachwiki, wikidatawiki, and others.