I don't recall why we decided to make the silverpop database instances on lutetium vs fundraisingdb (db1025) independent. But bear in mind if we make lutetium a realtime replica of fundraisingdb, the lutetium copy must become read-only. Is that the desired outcome?

I think so... As long as I can still run queries off it, that's all I need.

The history was strange, I forget the specifics too so I'll just recap what I know: We make a few hours of very expensive queries that almost certainly cause table or at least row locking, although we've tried to mitigate these side-effects by messing with the transaction isolation level. It would be nicest to run the Silverpop script on a replica of the civicrm database. At one point there was a theory that we would query silverpop tables directly from Civi, I think that's when we moved the master from lutetium's db server to live on frdb-write. Pretty sure we abandoned that approach, so maybe we should talk about moving the SP db master back to lutetium?

Can someone point me to the scripts that are in use? I want get a look at how they're using the databases.

@Jgreen, here are the scripts:

https://git.wikimedia.org/blob/wikimedia%2Ffundraising%2Ftools.git/master/silverpop_export%2Fupdate_table.sql

We only copy data from live tables into temp tables, then dump from the temp tables into the persistent export table.

Each query is prefixed with

SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;-- Silverpop Export Script, %s

by the update.py script that runs them

Ok thanks everyone. I don't see an issue with putting lutetium back into replication for the silverpop database. I don't have any brilliant way to do it without a bit of a short master database "pause" during which payments would be disrupted, so we'll need to schedule a maintenance window.

@Jgreen
Works for me! I guess, sooner is better.

this is done

@Jgreen just wanted to say thanks! It was so satisfying to run a query in
silverpop_export. This is going so helpful to me.