Page MenuHomePhabricator

[Spike] potential technical implementations of JADE
Closed, ResolvedPublic

Description

Based on T153152: Design JADE data storage schema, review potential technical implementations of JADE

  • Within MediaWiki's MySQL tables
  • Wikilabels PSQL database
  • Independent relational database
  • Something else?

Concerns:

  • Query-able
  • Joinable with mediawiki tables
  • Curation of text fields

Event Timeline

Halfak triaged this task as High priority.Jun 1 2017, 2:49 PM
Halfak moved this task from Unsorted to New development on the Machine-Learning-Team board.

Talked to @Ottomata about using EventBus to free us from settling on a "final" database store. If we can tie into EventBus, we can write to multiple data stores. E.g. we write to a Postgres DB as our main true storage, we also write to a redis LRU for fast lookups along with ORES scores, and we also write to a table in labsDB so that users can easily query the records.

I'm imagining two types of events:

  • judgement -- A user creates or updates a judgement that is stored
  • suppression -- A user with suppression rights flags a judgement to be hidden from view (e.g. it contains copyrighted content or is doxes another user) or restores it as a viewable

Oh! One more thought... We'll need to have the judgement storage operate at high performance so that judgements can be returned with ORES scores and not slow them down too much. So I'm thinking we should maintain a parallel cache storage for judgements (probably redis).

Sometimes, it’s too bad we don’t do the avro thing! Then you could use
this: http://docs.datamountaineer.com/en/latest/redis.html
(and also a PostGRES one)

awight renamed this task from [Spike] potential technical implementations of ORES meta to [Spike] potential technical implementations of Meta-ORES.Jul 27 2017, 6:10 PM
Halfak renamed this task from [Spike] potential technical implementations of Meta-ORES to [Spike] potential technical implementations of JADE.Sep 20 2017, 3:48 PM
Halfak updated the task description. (Show Details)
awight claimed this task.
awight edited projects, added Jade; removed Documentation, ORES.