Page MenuHomePhabricator

Create a basic product API to MySQL/MariaDB ETL
Closed, DuplicatePublic

Description

Per related discussions at T301428: Security API Storage Needs and T305114: Set up MariaDB for iPoid. Basic assumptions for this effort:

  1. Dealing with a single API endpoint providing a real-time (5-minute, ~30 Mb) data feed.
  2. Data will be appended to either a single, flat table or small group of normalized tables
  3. Don't need to worry about data duplication or existing data invalidation for now (may become a later concern though)
  4. Keep dynamic config in app.conf layer

Event Timeline

sbassett triaged this task as Medium priority.Apr 11 2022, 4:06 PM
sbassett moved this task from Incoming to In Progress on the Security-Team board.
sbassett added a project: SecTeam-Processed.

We should now be able to more properly model this process flow given the completion of the test vendor API endpoint within T309213.