Page MenuHomePhabricator

Plan and execute ingestion to include image in snapshot
Closed, ResolvedPublic8 Estimated Story Points

Description

We need to do a new ingestion to introduce main image into our snapshots API, so that we can offer it in all of our product offerings.

Acceptance criteria
Image object is in the snapshots.

ToDo

  • automate the ingestion process
    • add the ability to bulk-ingestion DAG to upscale articlebulk service
    • add the ability to bulk-ingestion DAG to upscale bulk-ingestion service
    • add the ability to bulk-ingestion DAG to monitor messages lag for article-names topic
  • check how it's going to influence snapshots (might have duplications in the topic for the time before compaction runs)
  • double check structured-data-articlebulk check in there any changes needs to be introduced
  • notify SRE about incoming traffic from us
  • deploy changes to prod
    • structured data article bulk service deployment
    • scheduler deployment
    • infrastructure deployment
  • run the bulk-ingestion DAG in prod

Event Timeline

Protsack.stephan renamed this task from [STUB] Plan ingestion to include image in snapshot to Plan ingestion to include image in snapshot.Jun 21 2023, 12:36 PM
Protsack.stephan triaged this task as High priority.
Protsack.stephan updated the task description. (Show Details)
Felixejofre set the point value for this task to 8.Jun 21 2023, 1:43 PM
Protsack.stephan renamed this task from Plan ingestion to include image in snapshot to Plan and execute ingestion to include image in snapshot.Jul 19 2023, 1:37 PM
JArguello-WMF changed the task status from Open to In Progress.Jul 27 2023, 4:50 AM

Note: In general it took around 36h to ingest namespace 0 for all projects.