Page MenuHomePhabricator
Create Task
Maniphest T364990

Structured Contents Snapshots Code
Open, In Progress, Needs TriagePublic3 Estimated Story Points
Actions

Description

As wme engineer i need a service that will read from project topics and create snapshots with all content.

Acceptance Criteria

  • Code developed and pipeline working

TODO

  • Change Snapshots GRPC API to allow new parameter named type with values article, as default, and structured
  • Change the snapshots flow to listen to topics accordingly to the type
  • Change the snapshots flow to marshal and unmarshall messages according to the message.key.type field. article as default, structured will do the new schema
  • Get feedback from Ricardo & Prabhat on code choices (specifically passing Type through ExportRequest proto object
  • Add unit tests and QA shakedown

Related Objects
Search...

Event Timeline

REsquito-WMF created this task.May 15 2024, 12:16 PM
REsquito-WMF edited parent tasks, added: T362319: {Machine Readability} Structured Contents MVP Release; removed: T362409: Q4 O1.KR2: Machine Readability Service delivered by end of quarter, per Structured Contents ERD .May 15 2024, 12:48 PM
REsquito-WMF moved this task from Incoming to To Be Estimated/To Be Discussed on the Wikimedia Enterprise board.Tue, Jun 4, 2:02 PM
REsquito-WMF set the point value for this task to 3.Tue, Jun 4, 2:09 PM
REsquito-WMF moved this task from To Be Estimated/To Be Discussed to Estimated /Discussed on the Wikimedia Enterprise board.Tue, Jun 4, 2:34 PM
JArguello-WMF moved this task from Estimated /Discussed to Sprint 62 on the Wikimedia Enterprise board.Thu, Jun 13, 1:09 PM
JArguello-WMF edited projects, added Wikimedia Enterprise (Sprint 62); removed Wikimedia Enterprise.
ROdonnell-WMF claimed this task.Thu, Jun 13, 7:02 PM
ROdonnell-WMF moved this task from Next Up to In Progress on the Wikimedia Enterprise (Sprint 62) board.
ROdonnell-WMF updated the task description. (Show Details)Fri, Jun 14, 5:21 PM
ROdonnell-WMF added a comment.Fri, Jun 14, 5:35 PM

I'm unsure if the snapshot code should be checking the Config's structured_projects.json slice to see if we should export or skip this Kafka message. Maybe it is done in a different part of the pipeline?

I've added this logic for the MR.

ROdonnell-WMF changed the task status from Open to In Progress.Fri, Jun 14, 9:02 PM
ROdonnell-WMF moved this task from In Progress to MR on the Wikimedia Enterprise (Sprint 62) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL