Page MenuHomePhabricator

Create export endpoint for for Snapshots service
Closed, ResolvedPublic13 Estimated Story Points

Description

Create exports endpoint for Snapshots service that will query appropriate kafka topic (or partition), create tar gz file with articles in NDJSON and upload it to s3 bucket so our users can download it though the API.

Acceptance criteria
Exports handler is able to create tar.gz and json (metadata) files and upload it to s3 bucket.

To-Do

  • create snapshots.proto file that will describe export endpoint, should take project identifier and namespace under general/proto repository
  • generate grpc server and setup dependency ingestion, refer to services/hourlys for example
  • create export endpoint that will create export based on project identifier and namespace for example
  • add unit tests
  • add integration tests

Notes

  • scheduler will send a command for a specific project
  • this endpoint will be called every day
  • please refer to documentation/api-specs-v2 to identify paths for json and tar.gz files
  • also example of how we did similar things in v1

Event Timeline

Protsack.stephan renamed this task from Create Export Endpoint For Exports Service to Create export endpoint for for snapshots service.Oct 12 2022, 9:37 AM
Protsack.stephan renamed this task from Create export endpoint for for snapshots service to Create export endpoint for for Snapshots service.
Protsack.stephan raised the priority of this task from Low to Needs Triage.Oct 12 2022, 9:40 AM
Daria_Kevana changed the task status from Open to In Progress.Dec 16 2022, 1:06 PM
Daria_Kevana changed the task status from In Progress to Open.Jan 26 2023, 1:00 PM