Change Details

Investigating what data sources to use for T201656, I considered options for getting data more frequently than what `wmf.mediawiki_history` would enable. One idea was to use the MW database, another was to use `event.mediawiki_revision_create`. I wanted to check whether `mediawiki_revision_create` stores all revisions or not. Comparing the random month of October 2018 against the history table for English Wikipedia, I found 70,719 revisions were missing (or 1.5%). Investigating some of them in the MW database, I did not find a clear pattern, but on the other hand I have not been digging very deeply. I'm unsure about what the expected reliability of an EventBus table should be, so I thought I'd open this task to discuss it.

Investigating what data sources to use for T201656, I considered options for getting data more frequently than what `wmf.mediawiki_history` would enable. One idea was to use the MW database, another was to use `event.mediawiki_revision_create`. I wanted to check whether `mediawiki_revision_create` stores all revisions or not. Comparing the random month of October 2018 against the history table for English Wikipedia, I found 70,719 revisions were missing (or 1.5%). Investigating some of them in the MW database, I did not find a clear pattern, but on the other hand I have not been digging very deeply. I'm unsure about what the expected reliability of an EventBus table should be, so I thought I'd open this task to discuss it. =Update= After improvements to envoy proxies, we found that we are missing **0.0014%** of events. The remaining missing events don't seem possible to fix in the current architecture, we would need some other way to send them more reliably or reconcile them later.

Investigating what data sources to use for T201656, I considered options for getting data more frequently than what `wmf.mediawiki_history` would enable. One idea was to use the MW database, another was to use `event.mediawiki_revision_create`. I wanted to check whether `mediawiki_revision_create` stores all revisions or not. Comparing the random month of October 2018 against the history table for English Wikipedia, I found 70,719 revisions were missing (or 1.5%). Investigating some of them in the MW database, I did not find a clear pattern, but on the other hand I have not been digging very deeply. I'm unsure about what the expected reliability of an EventBus table should be, so I thought I'd open this task to discuss it. =Update= After improvements to envoy proxies, we found that we are missing **0.0014%** of events. The remaining missing events don't seem possible to fix in the current architecture, we would need some other way to send them more reliably or reconcile them later.