Background
After some necessary scope/architecture changes, we now have a plan for how this script will look that is slightly different than the initial versions checked into production
Due to the anticipated filesize, we have a need for initializing the script before the file is in place on the host, so we will now need to poll with a timeout until the file is ready and available on the host
Additionally, there is still a batching requirement, but now the grouping strategy is better understood - we can do group lookups by email address, but if we want to issue batch preference updates, we'll need to be grouping by the preference values themselves. That is, the script should first sort users into buckets by their donor status ID, then perform batch DB operations on them
User story
As a WMF employee (or python wrapper) running this script, I'd like to be confident I can pass in an input file with potentially millions of rows and the script will perform
Requirements
- The script can operate safely on ~250k contacts normally, ~1m when donation volume is higher. See {T418164#11643063}
- The script should be able to handle large CSVs e.g. 1 million rows in a way that doesn't cause problems for SRE
- The script should be able to handle a file being copied into the container post-initialization, and should timeout if the file is not ready within a preset amount of time
BDD
- For QA engineer to fill out
Test Steps
Check requirements in T416949: Implement the first pass of a maintenance script that supports preference updates for consenting donors have been satisfied.
Design
https://wikimedia.slack.com/archives/C07C9137QGH/p1778618400416329
Acceptance criteria
The script works and can be tested manually with a file that is in place immediately locally (large or small), as well as with the wrapper script to ensure the polling functionality is working accordingly
Communication criteria - does this need an announcement or discussion?
One day this whole project will require communication, but this specific decision is probably okay to keep localized to Slack 🤞
Rollback plan
If the script has been run in prod, we will need to do DB maintenance of some sort to clean up the affected rows. A full DB rollback seems unlikely (🤞). I think there's a script for deleting preference rows that hopefully includes the globalpreference table (🤞🤞)