Page MenuHomePhabricator

Silverpop replication in frdb1003
Open, MediumPublic

Description

@DStrine please let me know if I'm creating this in the correct place.

We discussed a possible fix to getting silverpop data into frdb1003 during our tech / analytics sync this afternoon. Since the silverpop tables are, I believe, created using a python script enhancing the wmf_donor table, it might be possible to implement the same script on frdb1003. Placing note here to investigate.

To Do (add things here as we figure them out)

  • configure FR deployment for 'tools' project to fran1001
  • add python3-pymysql to fran1001
  • examine how silverpop updater works now, and what components/data are used to refresh the tables used for analytics
  • depending on how it works now, consider moving or duplicating some tables to the civi/drupal replication cluster
  • depending on how it works now, set up scheduled job on fran1001 to refresh tables on frdb1003

Event Timeline

EYener created this task.Mar 12 2020, 11:42 PM
Jgreen moved this task from Triage to Watching on the fundraising-tech-ops board.Mar 13 2020, 1:53 PM
DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.Mar 16 2020, 7:51 PM
Jgreen updated the task description. (Show Details)Mar 18 2020, 1:43 PM
Jgreen removed Jgreen as the assignee of this task.EditedMar 18 2020, 8:38 PM
Jgreen triaged this task as Medium priority.

Next step is to dig into /srv/tools/silverpop_export to figure out what to run locally on fran1001. It looks like it gets database settings from /etc/fundraising/silverpop_export.yaml which i, handily/shockingly, puppetized so we should be able to easily vary the config for analytics.

Jgreen updated the task description. (Show Details)Apr 9 2020, 3:47 PM
Jgreen moved this task from Up Next to Watching on the fundraising-tech-ops board.

Removing fundraising-tech-ops tag because we're not talking about simply copying data around at this point. We should be looking at making silverpop exporter compatible with database replication, so we can move its database into the replication cluster.