In T291120#7464525, @awight wrote:

The theory is that event sourcing is specifically designed to be replayed, so this greatly simplifies the requirement to build quasi-two-phase-commit architecture.

That would require us to REPLACE the database as the primary source of truth. That basically means reachitecting and rewriting a good chunk of MW. Even if we knew that this is the direction we want to take for the future, we are far far away from it. I think it makes more sense to focus on getting a reasonably reliable stream of events from mediawiki, in a standardized way.

The format of "problem statement first, solutions later" is very hard to follow, since I feel immediate urge jump to solutions, I'll try to stick to the problem.

After our most recent set of improvements under T215001 and thanks to @Milimetric and @JAllemandou analysis, we're now down to 0.0007% of revision-create events missing. We do not have exact numbers on the rest of event types since we didn't do such good analysis, but it's safe to assume that the numbers are similar. IMHO, this already looks quite good. I believe it's still possible to drop this number even more without drastically reinventing the architecture. We could look more into eventgate ( interactions between node.js thread poll and librdkafka own thread pool has always been suspicious), or we can experiment with replacing eventgate with librdkafka-based PHP kafka driver (there's a new one on the market that we've never evaluated).

I wonder, for which use-cases would 0.0001% be not enough? 0.00001%? Also, revisions are 'self-healing' for use-cases that rely on the lates view of the data, so when the page is edited next the view of the latest state becomes consistent again. I believe that if there are use-cases where 100% reliability is required we can find solutions based on some form of eventual consistency or reconciliation.

Non-guaranteed delivery of things like page deletes or revision suppressions are more important since these are not going to be naturally eventually consistent like revisions. But these operations are much much more rare then page edits, so perhaps we shouldn't put them together in one bag with revisions - solutions that are unacceptable for revisions might be totally ok for much more rare page deletes or renames.

The problem statement seems to suggest there will be a golden bullet that would allow us to make all events 100% reliable. That's mathematically impossible without sacrificing reliability. I think the solution would be unique for each type of event or use-case with a sliding scale between availability and consistency. The problem now imho is that we don't have any other option of event delivery and we don't have any easy option for reconciliation, so our sliding scale is missing a knob.

Thanks Petr! For consitency's sake (;p), let's keep discussions about consistency in T120242: Eventually Consistent MediaWiki State Change Events.

The problem statement seems to suggest there will be a golden bullet that would allow us to make all events 100% reliable.

I hope not! But, the problem state does suggest that we should try to make (important/relevant) MW state change events as (or as close to) reliable as MariaDB replication. That seems possible to me. Perhaps reconciliation will be the final solution, but I think there are other options we should consider too.

In T291120#7467254, @Ottomata wrote:

But, the problem state does suggest that we should try to make (important/relevant) MW state change events as (or as close to) reliable as MariaDB replication.

I know this is jumping to "solutioning" but the 'cheap' thing that is as reliable as MariaDB replication are MariaDB binlogs. Can the types of events desired be reconstructed from binlogs?

I know this is jumping to "solutioning" but the 'cheap' thing that is as reliable as MariaDB replication are MariaDB binlogs. Can the types of events desired be reconstructed from binlogs?

@bd808, see T120242: Eventually Consistent MediaWiki State Change Events

A lot of thoughtful comments have been made, and like others I find it difficult to separate the technical options/trade-offs from the problem statement itself. There seems to be a consensus that a reliable events architecture is desirable, but not if we are willing to pay a price in reliability for it. Some points we could include to make the problem statement more "decidable":

• a commitment to create a core event infrastructure that is a source of truth for which we are willing to pay a price in reliability (and engineering resources). That price is determined by many factors yet to be determined/discussed, and should be "configurable" by SRE. I.e. there will be some sort of coupling between the transactions to the MW DB and the event system; if one transaction fails the other gets a chance to react to it. This is difficult to discuss without getting technical, but without coupling there is no events with guarantees, and we can't have coupling without a price in reliability.
• explicitly exclude as goal to make events as a source of truth for MW itself. This seems too risky/ambitious, it could still be done more incrementally in the future once events are a source of truth.

The changes required to make events a core feature with guarantees, and the technical discussions/decisions preceding it, should also offer the chance to decrease the overall complexity of the WMF systems. We currently pay a hard-to-quantify but non-trivial price to build new systems/services that don't need to be a part of MW, in terms of human effort but also in terms of the options that are even considered feasible/reasonable to implement. Have we done a back-of-the-envelope evaluation of projects listed in the description in terms of estimated work given the status quo vs estimated work given a core event infrastructure with some correctness guarantees/streaming pipelines/etc? In my opinion, this aggregate cost needs to be weighted against the price in reliability it would cost us to build a core event infrastructure.

Hello, apologies for a late question (I have marked the "feedback" deadline for a wrong Friday and only now realized it was last week's Friday):
I might simply know too little about the current mediawiki's "event system" but reading through the problem statement, and thinking of its title ("source of truth") I do not fully understand why does it focus only on "services that need realtime MediaWiki data outside of MediaWiki"? If the event system was reliable enough to be considered a "source of truth" I'd imagine that (at least some parts of) Mediawiki (and its extensions) could actually move away from using the current SQL tables (revision, text, etc) to get the data they need. Is the different "performance" requirement level something that leads to ruling this application out and focus on "services outside of Mediawiki"? Or am I misinterpreting what "Mediawiki" refers to here? (e.g. would a hypothetical event-based service that updates Wikipedia infoboxes when Wikidata data changes -- currently implemented as Wikidata pushing updates to Wikipedias via mediawiki job queues -- be an instance of "service outside of Mediawiki")

I apologize if it occurs to be a loaded/non-constructive question. It is not my intention.

My understanding is that some of us got swept up into discussing the choice of a single source of truth (SSOT) but the bug author meant a source of truth, in a relative sense. In other words, the RFC is concerned with making the event streams trustworthy, but not trying to rearchitect MediaWiki to model all data flow as events. I propose that the RFC be renamed to clear this up, maybe to "Reliable event streams"?

As long as streams are reliable, we get most of the benefits of CQRS/ES even without our entire application using events internally. Waving hands in the general direction of what we might be missing without going for the "full" rewrite is that, tight RDBMS coupling will keep our application mostly monolithic, the API surface will continue to be the primary write interface, it will continu to be awkward for mw-core to ingest from externally-produced event sources.

oh dear, your favorite facepalm meme could go here. Thanks @awight for pointing out my stupidity. I am apparently so deep into the cool kids talks about the single source of truth all over the internet, that I've semiconsciously added it to the title of this document, while it indeed "only" talks about a source of truth.
Apologies for the noise.

After some discussion with Tim, it seems to me like the most realistic way to do this is Change Data Capture (CDC): we look at the databases replication log and generate events from that. However, direct CDC means binding directly to the raw database schema. That should be avoided. We'd want a layer that turns database row changes into more abstract events for general consumption. The code for the CDC event emitter should live in the same repo as the code that writes to the database - that is, it would live in MW core. It would be executed as a permanently running maintenance script. The log position could be maintained in kafka. This would provide parity with database replication in terms of latency and consistency.

I understand that we should not be "solutioning" too much while discussing the problem statement. I'm outlining the solution mainly to betetr understand the requirements - would this solution solve the problem outlined? Does it match the requirements?

My problem is that we don't know why we want a solution in this direction. The underlying problem is mentioned as:

For example, we know that currently, ~1% of revision create events are missing from the event streams

Let's assume the issue is in eventgate. Then we will be building a lot of code that would couple with mediawiki's database schema (increasing the cost of its much-needed improvements) for it to lose the 1% again and gain nothing. I would really like to see an investigation on this 1%. Where it vanishes into the void, how it can be improved, etc. If we know it's something in the architecture of the system, then yes, let's change the architecture.

Re CDC idea, pros and cons, see T120242: Eventually Consistent MediaWiki State Change Events, let's keep that discussion there. @daniel see the Hybrid CDC + Transactional Outbox idea for how we could have control over the event model, but still use the binlog directly (without a complicated raw row change translation layer).

For example, we know that currently, ~1% of revision create events are missing from the event streams

Recent fixes (some envoy timeout race conditions) may have brought this number way way down, which is great. We are about to have the full month of October imported into Hadoop, so we can do further analysis to really get this number.

However,

If we know it's something in the architecture of the system, then yes, let's change the architecture.

We do know this. No matter how reliable we make the deferred update async POSTing of a core state change event, we know that there will be discrepancies that over time will really matter. A solution like CDC might make the stream so good that any discrepancies (between what is in the binlog and what is in the stream) would totally disappear...but also maybe they won't? How confident are we currently that a MariaDB replica's database, or even just the revision table is, is 100% consistent with the master? Should a reliable event stream do better? I'd argue that it should be AS good, or better.

In either case, others have often argued that even if we get our event production consistency really really good, we will always need a way to reconcile discrepancies in the stream with the 'real' source of truth...the MediaWiki MariaDB. Perhaps all we will need is better EventGate SLOs (which maybe we can make now?) and a reconciliation mechanism built into MediaWiki. I was skeptical that this could work, but I'm starting to be convinced it might be possible. Hm, I should update the description of T120242 to include that as a possible solution.

@WMDE-leszek, if we can rely on event streams to source MW data for non-MW services...there's no reason we couldn't do it for MW too (at least in cases where MW is ok with eventual consistency). This problem statement def focuses on external services though, since MW already has a 'source of truth' that it alone has access to.

The replies above suggest that this ticket is indeed about events as the source of truth rather than a source of truth. @Ottomata can you clarify?

The replies above suggest that this ticket is indeed about events as the source of truth rather than a source of truth. @Ottomata can you clarify?

Hm, I'm not sure, but I think the confusion is if events are the (only?) source of truth then we'd have to event source MediaWiki in for order MediaWiki itself to be consistent. While that would be a nice solution if we were starting from scratch, doing that is intractable and not what this problem statement is concerned with. Instead, this is about being able to use MediaWiki state change events as a reliable source of truth for MediaWiki state changes. This would make it possible to externalize MediaWiki state in realtime without a runtime coupling to MediaWiki (and its MariaDB db).

In T291120#7492406, @Ottomata wrote:

In either case, others have often argued that even if we get our event production consistency really really good, we will always need a way to reconcile discrepancies in the stream with the 'real' source of truth...the MediaWiki MariaDB. Perhaps all we will need is better EventGate SLOs (which maybe we can make now?) and a reconciliation mechanism built into MediaWiki.

This is a really exciting suggestion—if a reconciliation mechanism is built-in and can be made flexible enough, it could also solve for a broad range of use cases: * edit conflict resolution * real-time collaborative editing * offline editing * and federation, just to name a handful...

In T291120#7493138, @Ottomata wrote:

Instead, this is about being able to use MediaWiki state change events as a reliable source of truth for MediaWiki state changes. This would make it possible to externalize MediaWiki state in realtime without a runtime coupling to MediaWiki (and its MariaDB db).

If this is the case, I suggest removing the "source of truth" wording from the proposal entirely. As is evenident from the discussion on this ticket, it's bound to create confusion. Personally, I have never seen "source of truth" used to refer to anything other than the single source of truth. Hoe about just saying that we need MediaWiki to act as a reliable even source, and focus the discussion on what "reliable" should mean in terms of consistency and latency.

I suggest removing the "source of truth" wording from the proposal entirely
I have never seen "source of truth" used to refer to anything other than the single source of truth

Hm, perhaps you are right.

However, this ticket is about making it possible to use MediaWiki produced events as the only source of truth a non-MediaWiki service needs to get MediaWiki data. I'm not suggesting that MediaWiki also only use events as SSOT (single source of truth) because of the intractability of that. It is much easier to design Event Sourced services from the start, rather than try to convert one to Event Sourced later. MediaWiki is huge and complicated and too important; I wouldn't suggest any kind of concerted effort to change its data architecture outright. If we get events as a source of truth, then parts of MediaWiki could incrementally be moved to an Event Sourced architecture. In an (my?) ideal world, this could be a theoretical goal, so in that sense I'd love to theoretically make MediaWiki events THE single source of truth for everything, including MediaWiki. It's just that I don't think doing so is practical, so I excluded that from consideration in this problem statement.

I suppose, since this is just a very abstract problem statement, we could frame the problem to include MediaWiki too? Although that would probably confuse more people and cause more friction.

How about just saying that we need MediaWiki to act as a reliable event source

Reliable and comprehensive. Meaning that any time someone wants to set up a new service with transformed MediaWiki data, there is a way to get all the state they need from events. If the specific state changes they need aren't yet in the streams, there should be something that allows them to code up MediaWiki to begin emitting that state as events, for themselves and others to use.

Just following up because I may have been wrong to introduce the idea of "a" source of truth above. I did this to try to show where the confusion in this task is coming from, but it's important to note that "single source of truth" has a very strict definition, if we had a SSOT it would be the MediaWiki primary database and there are no other sources of truth. For example, if you want to make an update it must be to this database, not in an event stream, etc.

Also following the definition of SSOT, the event source records are not another source of truth. They are in fact breaking ideal SSOT because they are copying data literally rather than as a reference. If we sent events that simply said "revision: 123" and you had to query the database to get the revision content, this would be closer to pure SSOT. Instead, we inline the content of the revision, which becomes problematic if a revision is deleted. In this case the event is out-of-sync with the actual source of truth. This is exactly the reason that SSOT exists, and shows how breaking it causes issues for us.

+1 that it seems most productive to focus on just "reliable and complete" event sourcing for this RFC, and give it a concrete target threshold of percentage messages dropped.

which becomes problematic if a revision is deleted

If it is deleted without a corresponding revision-delete state change event, that is problematic. Otherwise, fine, no?

They are in fact breaking ideal SSOT because they are copying data literally rather than as a reference

IIUC it is totally fine to copy data literally, as long as you are copying it from the SSOT, and the SSOT is reliable and complete.

I'd like to make it possible for everything except MediaWiki to consider events as their source of truth. Technically this isn't SINGLE source of truth, I guess, since the fundamental MW truth is MariaDB. I'd like to abstract that away from everything else, so from the perspective of a service that doesn't have access to MW MariaDB, the events are the only 'source of truth' they need.

In T291120#7503969, @Ottomata wrote:

which becomes problematic if a revision is deleted

If it is deleted without a corresponding revision-delete state change event, that is problematic. Otherwise, fine, no?

I only call this problematic because it's important that our downstream consumers respect the social convention that revision-delete is projected to delete previously received data. This makes event sourcing feel slightly uncomfortable, similar to if we were broadcasting protected data such as PII but asking our consumers to only retain for at most the legally-allowed 30-day period, for example. In both cases consumers have an obligation to periodically transform the entire history of each stream in order to comply with rules, which feels at odds with the ideal of a write-only, permanent event store.

But I agree that it's totally fine to do this, and to copy denormalized data literally, sending as events. What events have already done brilliantly is to give CQRS access to MediaWiki records, to external peers. From the perspective of an event consumer, it doesn't matter where the boundaries are between our event producer and the authoritative DB rows. Whether this is a tightly-coupled replication log follower or a set of MediaWiki hooks, the event stream will look the same, the only differences are in the expected reliability.

I think you said it well, that events are the only truth many consumers will need. Let's abandon this "truth" terminology if possible, though—if data is followed through our systems e.g. as a data-flow diagram, then each data-transforming process has a source data store and a sink data store. The source of any one thread of data flow is not necessarily true, it could already be out-of-date relative to the furthest upstream producer. Truth is a very strong claim to make about a piece of data (I would even hesitate to claim that the primary db contains truth), and also doesn't have a convenient pairing (i.e., the sink of a data flow is not "untrue") so not a good fit here. One could say "authoritative", but that would be tiresome to repeat across each data flow: "events are less authoritative than the primary database, but more authoritative than a locally-cached projection of these events". Maybe we can avoid this sort of language entirely, instead pointing to a DFD showing that e.g. the EventStreams ES endpoint mediawiki.revision-create mirrors the internal Kafka topic of the same name, which is produced by the EventBus extension hooking into onRevisionRecordInserted. Describing as a series of data flows lets the reader see the relationship between each node, choose which is the most appropriate for their use case, and also hints at potential losses through each linkage.

Of course, the point of this task is that we should project event streams from all important MediaWiki activities, and do it reliably. I'm completely in favor, apologies for all the words thrown at such a small issue of language. I would love to see what additional event streams are being considered (doc is not readable to the public yet). I'm sure that OKAPI will provide plenty of business motivation to guarantee a high level of reliability, deciding on the exact threshold is more of a cost/benefit thing than a RFC question. So maybe we should be talking about what's in the newly proposed event streams, governance over data retention, and for fun we could look at the potential for *incoming* event streams?

I only call this problematic because it's important that our downstream consumers respect the social convention that revision-delete is projected to delete previously received data

This is true of the replication binlog now, right? (If MariaDB didn't respect deletes). But also kafka compacted topics may help.

(doc is not readable to the public yet).

fixed.

what's in the newly proposed event streams, governance over data retention, and for fun we could look at the potential for *incoming* event streams?

Heckay ya. There aren't any new proposed event streams at this time, only talk about putting things like wikitext and/or html in a stream. Its going to be hard defining what is 'complete', so we changed our hopes to just making it easy enough for MW state change events to made more complete over time, as more use cases emerge and are implemented.

Let's abandon this "truth" terminology if possible

I don't mind.

One could say "authoritative",

Canonical?

source of any one thread of data flow is not necessarily true

But each data source should have a one official producer that emits the original state change event. The stream of these events is the series of true/canonical state changes for the relevant entity. I think the 'truth' term is a philosophical one, not a practical one. If you are able to traverse and apply the full history of all events, to a particular point in time, then you should have a full snapshot slice of what the full current state of all entities looked like at that time. In practice/realtime, the externalized state is not going to be consistent everywhere at once, only eventually so.

I think we agree @awight, and I am not attached to the term 'truth' in any way. I mostly want MediaWiki to produce state change event streams that other services can fully rely on to get what they need. "Events as a reliable and comprehensive source of MediaWiki state changes?"

In T291120#7506676, @Ottomata wrote:

I think we agree @awight, and I am not attached to the term 'truth' in any way. I mostly want MediaWiki to produce state change event streams that other services can fully rely on to get what they need. "Events as a reliable and comprehensive source of MediaWiki state changes?"

I feel that I am missing some subtlety in the distinction that you are drawing between 'truth' and 'can fully rely on'. What concretely is traded away in the sense of CAP theorem's framing of the binary choice of decreasing availability or decreasing consistency in the face of message loss in your fully reliable designation?

What concretely is traded away in the sense of CAP theorem's framing of the binary choice of decreasing availability or decreasing consistency in the face of message loss in your fully reliable designation?

In CAP terms, we lose Consistency, but only in that CAP defines it as "Every read receives the most recent write or an error.". State changes via event streams only provide eventual consistency (as do MariaDB replicas).

I've "boldly" edited the title to reflect how I understand the consensus above. We aren't trying to shift the source of truth, the point is to make events more useful for external system and service integrations.

Hmmm, bold! :) This task is owned by the Architecture team and the Technical Decision Forum chairs. They use this to create 'artifacts', so the description here needs to be synced to their artifact documents. We probably shouldn't change it without some coordination.

I'd like to discuss a little more before changing the title. I think 'a source of truth' is accurate for the intent: event streams should capture all MediaWiki state changes, so that it may be used as a 'source of truth' for MediaWiki. If that confuses people, I'm not opposed to changing the title, but I want to make sure that the intent is still captured somehow.

Also, this task is explicitly about MediaWiki event streams; so we should keep 'MediaWiki' in the title.

Of course, no problem at all to "revert" the title change :-), it was just a thought in order to avoid further confusion about the intent. Just to illustrate the problem with "source of truth", here are the Wikipedia search results.

Aye, k I'll revert for now. I'm mostly getting my 'source of truth' terminology from https://www.confluent.io/blog/messaging-single-source-truth/

Just to add my 2 cents, I too find the "A source of truth" terminology confusing. I am so accustomed to equate Source Of Truth to SSOT that it would take me some mental effort to accommodate for that. And while I now am able to do so, thanks to this discussion, newcomers will not have that historical benefit. I 'd argue that it's not worth it to confuse newcomers and ask them to do with that nuance.

Aye, k I'll revert for now. I'm mostly getting my 'source of truth' terminology from https://www.confluent.io/blog/messaging-single-source-truth/

Please forgive me for saying so, but that page has "single" in the URL and the title and "One" (capitalized) in the 3rd subheading. A cursory reading makes me suppose that it supports SSOT and not ASOT via messages, so it's confusing me somewhat more.

supports SSOT and not ASOT via messages, so it's confusing me somewhat more.

I think the philosophical intention is to be able to use streams as a SSOT. Its just that we will likely never rearchitect MediaWiki itself to use this streaming source of truth for all its state changes, even if we might wish that we could.

In the 'Getting Events into the Log' section of that article, it specifically talks about how database only based legacy systems can use other techniques (like CDC) to build the event streams other services can use as source of truth, and then incrementally also use that stream to rearchitect the legacy system into an event sourced one. Sure that would be great! I just don't want to propose that we should work on that.

So, in principal, I'm suggesting we should have a SSOT, but in practice we'll likely never get there...so ASOT is great too. :)

But, heard. Am willing to change title, lets discuss with TechForum folks.

In T291120#7534341, @Ottomata wrote:

But, heard. Am willing to change title, lets discuss with TechForum folks.

❤

In the 'Getting Events into the Log' section of that article, it specifically talks about how database only based legacy systems can use other techniques (like CDC) to build the event streams other services can use as source of truth, and then incrementally also use that stream to rearchitect the legacy system into an event sourced one. Sure that would be great! I just don't want to propose that we should work on that.

Thanks for clarifying that.

FYI, if anyone is interested, there is a free talk from Confluent on Dec 16 2021: Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka

Perhaps a better titled would be "Event Carried State Transfer of MediaWiki State"?

Ok, going with 'MediaWiki Event Carried State Transfer' as title.

@Jenlenfantwright @LNguyen This task changed state twice (in February and in March) despite a lack of substantial updates since November last year. Is there a record of any discussions or decisions that is available elsewhere?

Discussions that caused changes to the task here are all in the comments.

Some notes were taken during the meetings about the Decision Record, and are included at the bottom of that document.

The current status is that we have submitted the Decision Record to the Tech Forum, and will be meeting to discuss with Tech Forum board(?) on March 14 for their sign off. I probably should have posted a comment here last week noting that, sorry about that.

@LSobanski if you or anyone wants to discuss more, feel free to make a meeting or to just ask here. :)

Oh, BTW in case you weren't aware, the Decision Record we are submitting now is explicitly about the 'Comprehensiveness' problem, not the 'Consistency' problem. So, we are trying to solve for getting more MW state into streams, but not solving for improving the consistency of state streams emitted from MW. The consistency problem is much less pressing now that Petr has fixed some proxy timeout settings, and it's solutions (described in T120242) are also much more controversial, so we are punting on those for now.