Page MenuHomePhabricator

Migrate DjangoBannerStats processing and database to fundraising analytics cluster
Closed, ResolvedPublic

Description

The log processor currently runs on civi1001 and the database (pgehres) on frdb1002 replicating to the rest of the fundraisingdb cluster. We should be able to move these to fran1001/frdb1003 and reduce demands on donation processing systems. This has been discussed on T255456.

Event Timeline

Jgreen created this task.Jul 21 2020, 7:51 PM
  • configured nfs mount fran1001 -> frban1001 for access to impression logs
  • disabled 'pgehres' database replication to frdb1003
  • installed DjangoBannerStats and process-control on fran1001
  • configured DjangoBannerStats on fran1001 to update frdb1003/pgehres database
  • enabled fran1001 process-control job banner_impressions_loader

Next step is to migrate users to frdb1003/pgehres database.

Jgreen triaged this task as Medium priority.Jul 21 2020, 8:00 PM
Jgreen moved this task from Triage to In Progress on the fundraising-tech-ops board.
EYener added a subscriber: EYener.Jul 23 2020, 6:58 PM

Change 620725 had a related patch set uploaded (by Jgreen; owner: Jgreen):
[operations/puppet@production] nsca_frack.cfg.erb - remove check_ipsec from civicrm servers, add to fran1001

https://gerrit.wikimedia.org/r/620725

Jgreen updated the task description. (Show Details)Aug 17 2020, 4:16 PM

pfw policy update 16d09991bb5

Change 620725 merged by Jgreen:
[operations/puppet@production] nsca_frack.cfg.erb - remove check_ipsec from civicrm servers, add to fran1001

https://gerrit.wikimedia.org/r/620725

  • process-control banner log update job removed from civi1001
  • DjangoBannerStats code removed from civi1001
  • frdeploy configured not to deploy DjangoBannerStats to civi1001
  • nfs and ipsec configuration removed from civi*
  • iptables policy updated and refreshed on all hosts
  • database dropped from frdb1002 and replicas
  • database permissions removed from frdb1002 and replicas
Jgreen closed this task as Resolved.Aug 17 2020, 6:14 PM
Jgreen moved this task from In Progress to Done on the fundraising-tech-ops board.