Page MenuHomePhabricator
Create Task
Maniphest T311734

Design technical architecture for section topics pipeline
Closed, ResolvedPublic
Actions

Description

This ticket is to represent the work to design the technical architecture for the section topics data pipeline.

See the miro board at https://miro.com/app/board/uXjVOpiLkTQ=/

Highlights

Related Objects
Search...

Event Timeline

CBogen created this task.Jun 30 2022, 3:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2022, 3:21 PM
mfossati added a parent task: T311745: [EPIC] Section topics data pipeline.Jun 30 2022, 4:25 PM
mfossati renamed this task from Design techncial architecture for section topics pipeline to Design technical architecture for section topics pipeline.Jun 30 2022, 4:51 PM
CBogen assigned this task to mfossati.Jul 18 2022, 5:08 PM
CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.
CBogen moved this task from Incoming to Doing on the Structured-Data-Backlog (Current Work) board.
mfossati changed the status of subtask T312900: [M] Design database model for section topics pipeline from Open to In Progress.Jul 28 2022, 11:09 AM
mfossati closed subtask T312900: [M] Design database model for section topics pipeline as Resolved.Aug 10 2022, 9:45 AM
mfossati updated the task description. (Show Details)Aug 10 2022, 2:16 PM
CBogen added a comment.Aug 10 2022, 2:22 PM

@mfossati and @lbowmaker, what's left to do before we can consider this resolved?

mfossati updated the task description. (Show Details)Aug 10 2022, 2:26 PM

I think we need to decide whether we:

  • compute diffs or not
  • write the output dataset to Parquet or Hive
mfossati closed this task as Resolved.Aug 11 2022, 3:39 PM

We agreed on the following decisions, based on the Image-Suggestions experience:

  • don't compute diffs. This has been one of the most time-consuming and error-prone tasks
  • write the output Spark dataframe to Parquet, which is straightforward and effective, compared to tweaks required by Hive

This is reflected in the Miro board.
Closing this ticket.

CBogen mentioned this in T315021: [M] Document technical architecture for section topics onwiki.Aug 11 2022, 3:55 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL