Page MenuHomePhabricator
Feed Advanced Search

Today

Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

Ok, another run:

  • dev_civicrm fresh restore from prod 2020-08-07 (after table truncating and drops on this task)
  • across-the-board keep <= 550 days
  • run time was 2:20
  • footprint reduction of 190GB (from 723GB to 533GB)
Tue, Aug 11, 4:15 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops

Yesterday

Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

@Jgreen that is about 200GB less than is was at the start I think (+ 50GB off the drupal DB - definitely something)

Mon, Aug 10, 4:50 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops

Fri, Aug 7

Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

Cool - & that was 85 GB?

Fri, Aug 7, 3:49 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Fri, Aug 7, 4:41 AM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

@Jgreen I think we should drop the triggers & tables too from the top of this phab today too - it's pretty cautious not dropping them without an outage IMHO since the tables are completely unused without the triggers & both should be super quick to drop - so it would be good to use the outage to do it

Fri, Aug 7, 4:26 AM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Fri, Aug 7, 4:14 AM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Fri, Aug 7, 4:12 AM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Fri, Aug 7, 2:52 AM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops

Thu, Aug 6

Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

@Jgreen was I wrong in thinking the civi upgrade is today?

Thu, Aug 6, 9:32 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

@Jgreen those tables that we plan to drop all together were included in the 200GB? Will we drop those today on live during the outage?

Thu, Aug 6, 9:27 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

Testing on frdev1001 dev_civicrm database, purging to 750 days for all log tables reduced the database footprint from 862GB to 652GB. Found four tables missing the log_id auto_increment column, the script passed over those by design.

Thu, Aug 6, 1:16 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops

Thu, Jul 30

Jgreen created T259259: Get fr-tech folks set up with icinga access and familiarized on 'ack'-ing alerts .
Thu, Jul 30, 3:23 PM · Fundraising-Backlog

Fri, Jul 24

Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

The more I think about this the more I'd like to identify the tables that tell us about the contact & leave them out. I'm comfortable being more aggressive on all the other tables but I think the most likely thing for us to want to know after 6 months is address history & history of opt out etc.

I think contribution & activity data is large & I think they have less longer term value

Fri, Jul 24, 2:46 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops

Thu, Jul 23

Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

Testing on frdev1003/dev_civicrm, I wrote a script that creates log_*_postpurge tables for each log_ table, and populates them with the past 750 days of data, figuring 550 days discussed above plus ~200 days since dev_civicrm was reloaded. This takes about 3:40 to run and reduces the log table data footprint from 551GB to 151GB.

Thu, Jul 23, 9:15 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops

Wed, Jul 22

Jgreen updated the task description for T258600: reconfigure 'statler' to use fundraising analytics database (frdb1003) instead of frdev1001.
Wed, Jul 22, 3:26 PM · fundraising-tech-ops
Jgreen updated subscribers of T258601: move frdev1001:/srv/br/analytics out to it's own directory.
Wed, Jul 22, 3:25 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen removed a project from T258601: move frdev1001:/srv/br/analytics out to it's own directory: Epic.
Wed, Jul 22, 3:25 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T258601: move frdev1001:/srv/br/analytics out to it's own directory.
Wed, Jul 22, 3:25 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T258600: reconfigure 'statler' to use fundraising analytics database (frdb1003) instead of frdev1001.
Wed, Jul 22, 3:14 PM · fundraising-tech-ops

Tue, Jul 21

Jgreen updated the task description for T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster .
Tue, Jul 21, 8:04 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen removed a subtask for T242270: rack/setup/install civi2001.frack.codfw.wmnet: T246823: figure out Buster package for python3-mysql.connector for use with fruec.
Tue, Jul 21, 8:03 PM · Patch-For-Review, fundraising-tech-ops
Jgreen removed parent tasks for T246823: figure out Buster package for python3-mysql.connector for use with fruec: T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster , T242270: rack/setup/install civi2001.frack.codfw.wmnet.
Tue, Jul 21, 8:03 PM · Patch-For-Review, Fundraising-Backlog, fundraising-tech-ops
Jgreen removed a subtask for T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster : T246823: figure out Buster package for python3-mysql.connector for use with fruec.
Tue, Jul 21, 8:03 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster from Up Next to In Progress on the fundraising-tech-ops board.
Tue, Jul 21, 8:00 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen triaged T258526: Migrate DjangoBannerStats processing and database to fundraising analytics cluster as Medium priority.
Tue, Jul 21, 8:00 PM · fundraising-tech-ops
Jgreen added a comment to T258526: Migrate DjangoBannerStats processing and database to fundraising analytics cluster .

Next step is to migrate users to frdb1003/pgehres database.

Tue, Jul 21, 7:59 PM · fundraising-tech-ops
Jgreen added a comment to T258526: Migrate DjangoBannerStats processing and database to fundraising analytics cluster .
  • configured nfs mount fran1001 -> frban1001 for access to impression logs
  • disabled 'pgehres' database replication to frdb1003
  • installed DjangoBannerStats and process-control on fran1001
  • configured DjangoBannerStats on fran1001 to update frdb1003/pgehres database
  • enabled fran1001 process-control job banner_impressions_loader
Tue, Jul 21, 7:59 PM · fundraising-tech-ops
Jgreen closed T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster, a subtask of T241083: Epic: fundraising database approaching data partition capacity , as Resolved.
Tue, Jul 21, 7:55 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster as Resolved.

Investigation is complete.

Tue, Jul 21, 7:55 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster.
Tue, Jul 21, 7:55 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen created T258526: Migrate DjangoBannerStats processing and database to fundraising analytics cluster .
Tue, Jul 21, 7:51 PM · fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

I think some tables have more value for old data than others - notably contact & email data

Tue, Jul 21, 7:21 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Tue, Jul 21, 5:09 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

My understanding is that we have a lot of flexibility here. I'm pretty sure we could choose either 1 exact year or 1 year from a date and be fine. @Ejegg @Eileenmcnaughton or @XenoRyet could confirm though.

Tue, Jul 21, 5:06 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T258013: Fix stuck recurring contributions, add alarm.

Icinga alarm added to check_fundraising_job where other recurring alerts are triggered. I have found a way to have it notify once per day.

The check is running and correctly pulling the contribution numbers. Alerting will be turned on after @Jgreen review.

Jul 20 23:07:11 frdb1002 nagios_nsca[28941]: frdb1002 check_recurring_contrib_processing 2 CRITICAL recurring_contrib_processing=783 [critical >=500]
Tue, Jul 21, 2:35 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

My understanding was that we'd discussed a year for the log tables but the only question was whether it was a 1 year ago or 1 year as specified at a specific time of the year. I might be wrong though.

Tue, Jul 21, 2:30 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

What's the definition of done on this task? I see a merged patch, a related task, and a lot of comments. I'm curious what the next steps are.

Tue, Jul 21, 2:18 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops

Mon, Jul 20

Jgreen moved T248700: Investigate the rebrand url changes from FR-Ops to Triage on the Fundraising-Backlog board.
Mon, Jul 20, 7:30 PM · Fundraising-Backlog
Jgreen moved T202419: Page fr-tech about spikes in frtechmail from Watching to Backlog on the fundraising-tech-ops board.
Mon, Jul 20, 7:27 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T257919: Plan for FY20-21 Q1 fundraising database maintenance window from Triage to Up Next on the fundraising-tech-ops board.
Mon, Jul 20, 5:17 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster from Triage to Up Next on the fundraising-tech-ops board.
Mon, Jul 20, 5:17 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T258013: Fix stuck recurring contributions, add alarm from Triage to In Progress on the fundraising-tech-ops board.
Mon, Jul 20, 5:17 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen added subtasks for T255063: Package PrivateBin for FR-Tech hosts: Unknown Object (Task), Unknown Object (Task), Unknown Object (Task).
Mon, Jul 20, 1:04 PM · fundraising-tech-ops, Fundraising-Backlog

Tue, Jul 14

Jgreen updated the task description for T242270: rack/setup/install civi2001.frack.codfw.wmnet.
Tue, Jul 14, 9:54 PM · Patch-For-Review, fundraising-tech-ops
Jgreen added a parent task for T246823: figure out Buster package for python3-mysql.connector for use with fruec: T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster .
Tue, Jul 14, 9:54 PM · Patch-For-Review, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a subtask for T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster : T246823: figure out Buster package for python3-mysql.connector for use with fruec.
Tue, Jul 14, 9:54 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T257985: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster .
Tue, Jul 14, 9:53 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T252715: Enable Civi to more accurately reflect Silverpop unsubscribes from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Jul 14, 4:09 PM · fr-donorservices, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen moved T248701: investigate setting up a staging version of payments wiki from FR-Ops to Triage on the Fundraising-Backlog board.

Moving from FR-Ops back into Triage because afaict this is not blocked on fr-tech-ops, we'll need design decisions about how we want queues and payment-providers to work to proceed.

Tue, Jul 14, 4:08 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T223658: Enable Content Security Policy on donate-wiki from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Jul 14, 4:05 PM · Fundraising-Backlog
Jgreen moved T223636: CentralNotice: Create virtual server for user testing of new features from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Jul 14, 3:59 PM · MediaWiki-extensions-CentralNotice, Fundraising-Backlog
Jgreen moved T199471: Figure out where json_encode will screw us in php 7.1+ from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Jul 14, 3:57 PM · Fundraising-Backlog
Jgreen moved T154509: create and/or automate tools for auditing composer-installed php packages from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Jul 14, 3:56 PM · Fundraising-Backlog
Jgreen created T257919: Plan for FY20-21 Q1 fundraising database maintenance window.
Tue, Jul 14, 12:49 PM · Fundraising-Backlog, fundraising-tech-ops

Mon, Jul 13

Jgreen updated the task description for T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster.
Mon, Jul 13, 3:10 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster.

@Jgreen I'd like to set up more alerting on frdb1003 for before we make this change so that my data cube testing and runs don't cause the server to lag behind master. Any ideas you have on this - or ideas on how I can leverage existing alerting - would be great!

Mon, Jul 13, 12:42 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster.
Mon, Jul 13, 12:39 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops

Jul 10 2020

Jgreen closed T257680: make kafkatee banner impression logs accessible from fundraising analytics server, a subtask of T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster, as Resolved.
Jul 10 2020, 7:31 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T257680: make kafkatee banner impression logs accessible from fundraising analytics server as Resolved.
Jul 10 2020, 7:31 PM · fundraising-tech-ops
Jgreen updated the task description for T257680: make kafkatee banner impression logs accessible from fundraising analytics server.
Jul 10 2020, 5:58 PM · fundraising-tech-ops
Jgreen updated the task description for T257680: make kafkatee banner impression logs accessible from fundraising analytics server.
Jul 10 2020, 4:45 PM · fundraising-tech-ops
Jgreen triaged T257680: make kafkatee banner impression logs accessible from fundraising analytics server as Medium priority.
Jul 10 2020, 3:49 PM · fundraising-tech-ops
Jgreen created T257680: make kafkatee banner impression logs accessible from fundraising analytics server.
Jul 10 2020, 3:47 PM · fundraising-tech-ops
Jgreen moved T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster from Up Next to In Progress on the fundraising-tech-ops board.
Jul 10 2020, 3:31 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops

Jul 9 2020

Jgreen added a comment to T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster.

From today's analytics coordination meeting: The pghehres database is queried directly from frdev during campaigns to monitor banner activity. It's also accessed for various reporting functions using Peter Coombe's statler software, https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/analytics/+/585235/
It sounds feasible to move the database and database generation script to frdb1003/fran1001. We will need to add users frdev1001 mariadb users to frdb1003 first.

Jul 9 2020, 7:30 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Jul 9 2020, 6:40 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T257232: map out a civicrm.log_% table data expiration/delete strategy from Up Next to In Progress on the fundraising-tech-ops board.
Jul 9 2020, 5:33 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T195259: Testing infrastructure for EventLogging ingress of banner impression and landing page data from Triage to Done on the fundraising-tech-ops board.
Jul 9 2020, 3:54 PM · fundraising-tech-ops
Jgreen added a comment to T195259: Testing infrastructure for EventLogging ingress of banner impression and landing page data.

We ended up testing on the otherwise unused codfw banner logger server.

Jul 9 2020, 3:49 PM · fundraising-tech-ops
Jgreen updated the task description for T195259: Testing infrastructure for EventLogging ingress of banner impression and landing page data.
Jul 9 2020, 3:48 PM · fundraising-tech-ops
Jgreen closed T195259: Testing infrastructure for EventLogging ingress of banner impression and landing page data as Resolved.
Jul 9 2020, 3:48 PM · fundraising-tech-ops
Jgreen removed a project from T199471: Figure out where json_encode will screw us in php 7.1+: fundraising-tech-ops.
Jul 9 2020, 3:47 PM · Fundraising-Backlog
Jgreen closed T204832: CPS contractor contact information as Resolved.
Jul 9 2020, 3:46 PM · Fundraising-Backlog
Jgreen closed T207511: Sort out fr-tech work phone situation, a subtask of T91508: [Epic] overhaul fundraising cluster monitoring, as Declined.
Jul 9 2020, 3:45 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T207511: Sort out fr-tech work phone situation, a subtask of T202419: Page fr-tech about spikes in frtechmail, as Declined.
Jul 9 2020, 3:45 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T207511: Sort out fr-tech work phone situation as Declined.
Jul 9 2020, 3:45 PM · observability, Fundraising-Backlog
Jgreen moved T202314: icingawm is missing from #wikimedia-fundraising channel from Triage to Done on the fundraising-tech-ops board.
Jul 9 2020, 3:44 PM · fundraising-tech-ops
Jgreen closed T202314: icingawm is missing from #wikimedia-fundraising channel as Resolved.
Jul 9 2020, 3:44 PM · fundraising-tech-ops
Jgreen removed a project from T223658: Enable Content Security Policy on donate-wiki: fundraising-tech-ops.

Removed tag because there's nothing actionable for fundraising-tech-ops, donate-wiki is not administered by our team.

Jul 9 2020, 3:39 PM · Fundraising-Backlog
Jgreen removed a project from T223636: CentralNotice: Create virtual server for user testing of new features: fundraising-tech-ops.

Removed tag because there isn't anything actionable from a fundraising-tech-ops perspective.

Jul 9 2020, 3:36 PM · MediaWiki-extensions-CentralNotice, Fundraising-Backlog
Jgreen removed a project from T248700: Investigate the rebrand url changes: fundraising-tech-ops.

Removing fundraising-tech-ops tag because the fr-tech-ops actionable part was broken out to a separate task T247846 and completed.

Jul 9 2020, 3:32 PM · Fundraising-Backlog
Jgreen removed a project from T188561: SSL cert for links.email.wikimedia.org: Fundraising-Backlog.
Jul 9 2020, 3:19 PM · Operations, Traffic, fundraising-tech-ops
Jgreen added a comment to T188561: SSL cert for links.email.wikimedia.org.

Checking back on this it looks like https://links.e.uso.org has slipped back to a B rating because they haven't ceased TLS 1.0/1.1 support. I'm not sure if that's still a valid hostname to check, or if this task is still relevant however.

Jul 9 2020, 3:19 PM · Operations, Traffic, fundraising-tech-ops
Jgreen closed Restricted Task, a subtask of T248700: Investigate the rebrand url changes, as Resolved.
Jul 9 2020, 3:07 PM · Fundraising-Backlog
Jgreen moved T110227: Adjust modsecurity rules to work for us from Backlog to Done on the fundraising-tech-ops board.
Jul 9 2020, 3:06 PM · fundraising-tech-ops
Jgreen added a comment to T110227: Adjust modsecurity rules to work for us.

Closing task because we did about as much ruleset tuning as was feasible given the application. Spun off T122322 to look at the possibility of modifying payments-wiki behavior to make it easier to work with a WAF.

Jul 9 2020, 3:05 PM · fundraising-tech-ops
Jgreen closed T110227: Adjust modsecurity rules to work for us as Resolved.
Jul 9 2020, 3:03 PM · fundraising-tech-ops
Jgreen moved T110227: Adjust modsecurity rules to work for us from Stalled to Backlog on the fundraising-tech-ops board.
Jul 9 2020, 3:01 PM · fundraising-tech-ops
Jgreen removed a project from T254298: Upgrade payments to MediaWiki 1.35: Fundraising-Backlog.

Removed Fundraising-Backlog tag. The upgrade (this task) is a fr-tech-ops project. Broke out the Fundraising Tech parts to separate tasks for clearer task dependency tracking.

Jul 9 2020, 2:19 PM · FR-Q2-FY2020-21-cleanup-list, fundraising-tech-ops
Jgreen removed a project from T256153: Configure payments2xxx for MediaWiki 1.35 upgrade testing: Fundraising-Backlog.
Jul 9 2020, 2:00 PM · fundraising-tech-ops

Jul 8 2020

Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Jul 8 2020, 8:23 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T257482: Access to Civi Terminal for Evelyn Martin for Reconciliation queries from Triage to Up Next on the fundraising-tech-ops board.
Jul 8 2020, 5:42 PM · fundraising-tech-ops
Jgreen renamed T257482: Access to Civi Terminal for Evelyn Martin for Reconciliation queries from Access to Civi Terminal for Reconciliation queries to Access to Civi Terminal for Evelyn Martin for Reconciliation queries.
Jul 8 2020, 5:41 PM · fundraising-tech-ops
Jgreen added a project to T257482: Access to Civi Terminal for Evelyn Martin for Reconciliation queries: fundraising-tech-ops.
Jul 8 2020, 5:39 PM · fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Jul 8 2020, 5:14 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

@Jgreen I can take a stab at those answers
for log_civicrm_entity_tag, yep, that's the idea. We used to automatically apply a couple of tags to contacts and contributions during queue consumption, but we no longer do. So 99%+ of the existing log table is that automatic useless stuff. We can drop all the old stuff and just log new changes, which will presumable be human-initiated.

log_civicrm_subscription_history should definitely be dropped along with associated triggers - the subscription_history table itself is basically a log of what groups people have been assigned to, so a log table on those values is redundant.

Jul 8 2020, 5:13 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Jul 8 2020, 2:17 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Jul 8 2020, 2:16 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T257232: map out a civicrm.log_% table data expiration/delete strategy.

@Eileenmcnaughton re. the list of tables to drop, can you clarify these two?

Jul 8 2020, 2:10 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Jul 8 2020, 2:06 PM · Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops