Page MenuHomePhabricator

Epic: fundraising database approaching data partition capacity
Open, MediumPublic

Description

The fundraising production database has been growing at very roughly 50% per year, and is currently about 1.6. The data partition on our database machines ranges from 3.6T to 5.2T. frdev1001 is approaching capacity fairly quickly since it hosts a live replica of the civi/drupal/fredge/pgehres/faulkner/smashpig database (~1.6TB) plus several dev databases (~970GB). We need to start planning for a refactor of some kind.

Current database footprint:

2020-06-292020-11-17
smashpig2G6G
geonames6G6G
fredge39G51G
drupal56G70G
pgehres293G(migrated to frdb1003)
faulkner354G(migrated to frdb1003)
civicrm820G840G

Data partition (/srv) capacity varies by server according to when we bought them:

host/srvpurchaserefreshdcnotes
frdb10013.6T2016-05-13FY20-21eqiadstandard civi/drupal db
frdb10023.6T2017-02-24FY21-22eqiadstandard civi/drupal db (master)
frdb20015.2T2020-02-14FY24-25codfwstandard civi/drupal db
frdb20025.2T2019-12-03FY24-25codfwstandard civi/drupal db
frdev10013.6T2017-02-24FY21-22eqiaddev database (eqiad, also includes civi etc)
frdb10035.2T2019-11-01FY24-25eqiadanalytics (also includes civi etc)
frdb10045.2T*2020-09-21FY25-26eqiadstandard civi/drupal db

Rough growth numbers for the civicrm database and projections on when database footprint will exceed working capacity are here https://docs.google.com/spreadsheets/d/1FqQtn4XnXBnAIH8_FjFEp0mDty5RD85gJvWBh2xXyyY/edit?usp=sharing

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedEileenmcnaughton
ResolvedEileenmcnaughton
ResolvedEileenmcnaughton
ResolvedEileenmcnaughton
ResolvedEileenmcnaughton
ResolvedEileenmcnaughton
ResolvedEileenmcnaughton
OpenEileenmcnaughton
OpenEileenmcnaughton
OpenEileenmcnaughton
ResolvedEileenmcnaughton
ResolvedEjegg
OpenEileenmcnaughton
ResolvedDwisehaupt
OpenNone
OpenNone
ResolvedNone
ResolvedJgreen
ResolvedJgreen
ResolvedJgreen
OpenJgreen
ResolvedDwisehaupt
ResolvedJgreen
ResolvedPcoombe
OpenNone

Event Timeline

Jgreen created this task.Dec 18 2019, 8:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 18 2019, 8:48 PM
DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.Dec 19 2019, 6:54 PM
Jgreen triaged this task as Medium priority.Jan 8 2020, 6:23 PM
Jgreen removed Jgreen as the assignee of this task.Jan 16 2020, 9:33 PM
DStrine renamed this task from frdev1001 approaching data partition capacity to Epic: civi crm db approaching data partition capacity .Jan 29 2020, 12:50 AM
DStrine added a project: Epic.
Jgreen closed subtask Restricted Task as Resolved.Feb 19 2020, 10:42 PM
Jgreen moved this task from Triage to Watching on the fundraising-tech-ops board.Feb 19 2020, 10:46 PM
Jgreen changed the status of subtask T249046: Look at deleting indexes from Duplicate to Resolved.Jun 15 2020, 3:07 PM
Jgreen updated the task description. (Show Details)Jun 22 2020, 8:04 PM
Jgreen updated the task description. (Show Details)Jun 22 2020, 8:12 PM
Jgreen updated the task description. (Show Details)Jun 22 2020, 8:18 PM
Jgreen renamed this task from Epic: civi crm db approaching data partition capacity to Epic: civicrm database approaching data partition capacity .Jun 23 2020, 1:59 PM
Jgreen renamed this task from Epic: civicrm database approaching data partition capacity to Epic: fundraising database approaching data partition capacity .
Jgreen updated the task description. (Show Details)
Jgreen updated the task description. (Show Details)Jun 23 2020, 2:03 PM
Jgreen updated the task description. (Show Details)Jun 23 2020, 2:07 PM
Jgreen updated the task description. (Show Details)Jun 23 2020, 2:11 PM
Jgreen updated the task description. (Show Details)Jun 29 2020, 3:17 PM
Jgreen updated the task description. (Show Details)Jun 29 2020, 3:19 PM
Jgreen updated the task description. (Show Details)Jun 29 2020, 4:00 PM
Jgreen updated the task description. (Show Details)
Jgreen updated the task description. (Show Details)Jun 29 2020, 4:16 PM
Jgreen updated the task description. (Show Details)
Jgreen updated the task description. (Show Details)Jun 29 2020, 6:05 PM
EYener added a subscriber: EYener.Jul 23 2020, 6:59 PM
Jgreen updated the task description. (Show Details)Nov 17 2020, 5:34 PM
Jgreen updated the task description. (Show Details)Nov 17 2020, 5:45 PM
Jgreen updated the task description. (Show Details)Nov 17 2020, 5:49 PM