Page MenuHomePhabricator

Refactor fundraising database schema
Open, Needs TriagePublic

Description

The current database schema has three major problems:

  • German column names - This leads to confusion for non-German speaking developers, diminishing developer experience, making onboarding more involved and leading to subtle errors: adresstyp vs address_type, note the different amount of "d"
  • Large Object (LOB) Columns (data fields) - This makes it hard to export and query existing data.
  • Persistence layer breaks the boundaries of bounded contexts. Example: Both Donations and Memberships have a relation to Address Changes, but Address Change is its own bounded context and the Donation and Membership Entities (Business Objects) don't have references to Address Change. The reference is only there as a convenience, to trigger the creation of an AddressChange record when we save Donations/Memberships. Here, the framework/storage layer bleeds into the BC.

During this refactoring, these problems should be rectified, with the following constraints:

  • The FundraisingFrontend, Fundraising Operation Center and Export (SpendenDumper) must be operational at all times.
  • To enforce the separation of the bounded context, we want to get rid of FundraisingStore "library" dependency in the fundraising-donation, fundraising-memberships and fundraising-subscriptions code repositories. Each bounded context should manage its own database schema.
  • Entities in different bounded contexts are not allowed to reference each other (creating a dependency on entities outside the bounded context). If there is a reference on the storage level, for example to aggregate data for export, the reference should be inserted/changed at the storage level and should not show up in the entity definition.
  • We want to keep our domain objects unaffected by database technology - no ORM annotations on them. Use a separate XML file instead

Acceptance Criteria:

  • All column names are in English
  • If using ENUM columns (e.g. address_type), the values are in English.
  • The data stored in the LOB is split out into tables
  • All database access is behind one or more interfaces, using the Repository Pattern
  • We're still able to do database migrations across all bounded contexts.

Note:

  • To keep the Fundraising Operation Center and export working without huge refactorings, we could store the data redundantly, using the StranglerApplication pattern:
    1. Copy the DoctrineDonationRepository and DoctrineMembershipApplicationRepository to the FundraisingFrontend, change the Doctrine prefix to Legacy
    2. Change the DoctrineDonationRepository in fundraising-donation and fundraising-membership to map the Domain entities to database tables.
    3. In FundraisingFrontend, create DonationRepositoryWrapper and MembershipApplicationRepositoryWrapper classes that implement the repository interfaces and have the Legacy and newly created repositories as dependencies. Whenever an interface method is called, both repositories must receive the method call.

Possible tables/entities (to be discussed)

  • donation
    • With a better state, separating payment state, soft delete, export, etc. See for example T137704: Add `isApproved` field for comments
    • with the concept of a "followup donation" that's linked to a previous donation and shares its entities
  • donor (mostly differentiated by different address types, so far it's "private" and "company", but could become "email" in the future, see T220367: Split e-mail address out of postal address section)
  • donation events/change log: Commented, canceled, paid, moderated, exported. Those can be triggered from frontend and FOC
  • payment (we'll have to see how we model the different payment types & their metadata)
  • comment
  • tracking (impcount, keyword and campaign from banner)
  • buckets (bucket and name campaign from banner)
  • membership_applications
  • membership_applicants
  • subscriptions (we can probably drop the address part because we use the subscription feature only for collecting emails for various purposes)
  • address_change

Impression counting should be its own domain/entities in the backend.

Maybe not every Domain object needs a table, minor objects (like DonorName) could be implemented as Mapping Types.

Resources:

Event Timeline

Restricted Application added a project: WMDE-FUN-Team. · View Herald TranscriptSep 6 2018, 3:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gabriel-wmde updated the task description. (Show Details)Sep 6 2018, 3:25 PM
gabriel-wmde renamed this task from Refactor database schema to Refactor fundraising database schema.Sep 6 2018, 3:30 PM
gabriel-wmde updated the task description. (Show Details)Dec 19 2018, 1:58 PM
gabriel-wmde updated the task description. (Show Details)Jan 11 2019, 1:50 PM
gabriel-wmde updated the task description. (Show Details)Jan 11 2019, 3:57 PM
gabriel-wmde updated the task description. (Show Details)Jan 11 2019, 6:13 PM
gabriel-wmde updated the task description. (Show Details)Apr 15 2019, 9:07 PM
gabriel-wmde updated the task description. (Show Details)Apr 26 2019, 5:10 PM
gabriel-wmde updated the task description. (Show Details)May 29 2019, 4:18 PM
gabriel-wmde updated the task description. (Show Details)Wed, Dec 4, 4:13 PM