|Open||None||T258511 Data Lake incremental Data Updates|
|Open||None||T240387 MW REST API Historical Data Endpoint Needs|
|Resolved||Milimetric||T241184 Design Document that proposes an alternative architecture for historic data endpoints|
My apologies cause i though this task had been updated a while back, here is the document: https://docs.google.com/document/d/1IacO1l5Y0uGrVnOZQ-3JskZaRqtiwqvpAffmAlJZquQ/edit
This task is not fully finished, we have halted progress due to resourcing on covifd issues, @Milimetric to provide an update
We basically split this effort into two phases. The design for the first, designing a better data pipeline which would feed highly available historic data endpoints, is done. The second phase is a bit too abstract so I'm going to break it out into a separate task. For now, I'll take this, polish up anything that needs polishing on the document, and call it done. We can then implement whenever Core Platform is ready to work together on it.
Thank you this is important work~ Kudos to the team.
Can we make https://docs.google.com/document/d/1IacO1l5Y0uGrVnOZQ-3JskZaRqtiwqvpAffmAlJZquQ/edit accessible to general reader? I am interested in reading it.
I don't see why not, making it read-only now. The first section on the lambda architecture is considered reviewed and done, we're still playing around with the second section.