This sounds like it would work... but I do want to point out a potential maintenance issue:

The three of us (Dan, Xabriel, Jennifer) from Data Products who onboarded onto Dumps 1.0 were doing so with the assumption that we were going to get *more* resources to finish Dumps 2.0 and that it would be done by now. So we never learned in-depth what maintaining the dumps scripts would entail. Since then, the Dumps 2.0 work was deprioritized almost completely for the last few months.

The problem is, what little we learned about Dumps 1.0 is changing a bit with the wrapper scripts and containerization described above. So that's a potential risk for long-term maintenance of Dumps 1.0. We are trying to advocate for re-prioritization but I also have a sabbatical coming up. Just wanted to detail the precarious nature of this situation so that decision makers know what's going on. cc @VirginiaPoundstone and @WDoranWMF.

Thanks for the flag @Milimetric.

@Joe thanks for thinking this through. I have three follow-up questions:

1. what is the timeline you are thinking about for this work?
2. Are there any hard deadlines?
3. If the new dumps is not ready to replace the current dumps by end of fiscal year, what gets blocked and/or toil is endured?

While we have made progress, we still can not accurately predict how long it will take us to get to the new dumps (there are a couple big puzzles we still need to solve: scale & drift ), but we do plan on prioritizing the work towards end of Q3 23/24.

In T352650#9388845, @VirginiaPoundstone wrote:

@Joe thanks for thinking this through. I have three follow-up questions:

Answers inline :)

1. what is the timeline you are thinking about for this work?

We'd hope to start working on this late next quarter, or at the start of Q4 at the latest. We've gone this way when we were asked if we could live without dumps 2.0 before the end of the fiscal year, so we got creative. To be clear, what I am proposing here is a terrible hack we should never ever do.

And yes, we'll still need support from the team responsible for maintaining dumps.

2. Are there any hard deadlines?

We need to have switched old dumps off completely, or to this new setup, by end of June 2024.

3. If the new dumps is not ready to replace the current dumps by end of fiscal year, what gets blocked and/or toil is endured?

We're building even more technical debt on top of an infrastructure, dumps, that is both fundamental to our community and also a good pyle of tech debt itself. A platform that while fundamental doesn't have proper software-side support (based on what @Milimetric was saying).

While we have made progress, we still can not accurately predict how long it will take us to get to the new dumps (there are a couple big puzzles we still need to solve: scale & drift ), but we do plan on prioritizing the work towards end of Q3 23/24.

I don't think it's realistic to expect dumps 2.0 to have fully replaced the old dumps by the end of the fiscal year. And if we don't tie up these remaining loose ends, all the work we've done for the annual plan to migrate mediawiki on k8s would reap half the benefits.

So I think this work still needs to be done at this point.