@RobH Thanks, I think I've determined I have appropriate access to deploy, and will deal with anything extra in additional requests, if it turns out I was wrong...

@Ladsgroup identified an important side benefit of doing this work: this will give us the opportunity to evaluate the different ways we generate data sets for each wiki, and decide which are the correct ways going forwards.

FWIW, I see that I have access to tin and can presumably deploy there. However, I don't have ssh access to the canary server scb1002 or any of the other workers. This might be the correct access level for my team, in which case I think we're finished here. Let me know!

@Halfak @Ladsgroup
I'm curious about how we plan to use the codfw boxes--will they simply be warm standby, or will they be included in the worker pool?

Verified that I've been added to the research/ores group and seem to have Gerrit rights. Thanks for the blindingly fast work!

Looks like we have a nowiki model, now. Thanks for the nudge!

We should run some data through and inspect the output.

In T168442#3364694, @awight wrote:

Confirmed that I already have access.

Confirmed that I already have access.

Thanks for the note! I've made both agreen and jseddon shared list admins, in case they need the flexibility.

I can volunteer to flip the wiki switches, once @Halfak 's concern above is resolved.

Hi, let's mark this resolved. Making the diagrams was a great learning experience for me, and I feel like the data flow is pretty well covered.

Diluting the pending store with non-cc transactions is turning out to be more expensive than we thought. Each non-cc pending record takes 3 seconds to skip (unknown why this is so slow), and since the orphan rectifier is time-limited, we end up only processing about 10 orphans every 15 minutes during high iDEAL volume campaigns.

I realize now that my solution was too naive. Specifically, staggering the queue consumer and thank-you jobs using the cheesy consume_and_thank parent job paradigm prevents reentry of the entire chain, when it should only lock sub-jobs. In other words, we prevent the next queue consumer from running if the previous thank-you job is still going, when we should let them overlap instead.

We've added the hostname to the default template now.

@Jgreen
Wicked, thanks for the soft landing!

See https://phabricator.wikimedia.org/T156184

See https://phabricator.wikimedia.org/T162716