Oh! It's great that we can do range deletions from Cassandra!
I think with the deprecation of Druid, we are adopting Cassandra as our main serving layer, and we should be able to delete data from it given a time range.
This would help do idempotent reruns, and implement deletion policies.

Hi @Scott_French, We've had to do some refactoring of the whole pipeline, and it has delayed us.
Yesterday we deployed the new pipeline and today we'll start loading data to Cassandra.
We should be there soon, I imagine in a couple days.

@Eevans it should take around 1 sprint (3 weeks) for us to adapt the service to the latest naming changes, do the integration tests and QA. So, it will be when you come back.

@Eevans yes, of course.
Let's freeze this task for now.
We will meet as a team, since these changes would require some rework.
We'll decide, and we'll get back to you with all the changes to this task and the design doc.
🙏

Yes @xcollazo I also wish I knew about this sooner.
And thanks @Eevans for the careful review!

Oh, and here's the notes from the dumps design session. https://docs.google.com/document/d/1he7AziO1CYAFPeyRGgs5Ez-4G_jRDvAl0l8zsBEtc9A

Regarding the size of the dumps, tl;dr:

I think the checksum might need to contain the state of both the default values coming from the code AND the override values coming from the Airflow Variable.

I think the solution I propose above does not work, I think this assumption is wrong:

If there's a difference between a code property and its variable override, but the checksum still checks (variable property has not changed), it means the code change should prevail,

If we did this, any override would change the DAG value just for one parsing cycle. The second parsing of the DAG would reset the property value to the default.

• When do you anticipate having a minimal binary that successfully builds? (to unblock build pipeline setup)

Does this binary need to implement some of the endpoints, or just the frame of the service?

@BTullis and @brouberol sorry for the delay.
This is blocking the MW history snapshot automation work, but it's not urgent, so don't worry.
It doesn't block the commons metrics work so far. So also no worries there.
Thanks for pinging :-)

Here's the link to the doc: https://docs.google.com/document/d/1sWPJzO9J6nwfhzJAwrbWZQBcvwiT8UhBBfn2kFW3pNQ/edit
And the link to the miro board: https://miro.com/app/board/uXjVNi0kGyo=/#tpicker-content

Hi @kai.nissen and @gabriel-wmde!