[Spike] potential technical implementations of Meta-ORES
Open, HighPublic

Description

Based on T153152: Design Meta ORES data storage schema, review potential technical implementations of meta ORES.

  • Within MediaWiki's MySQL tables
  • Wikilabels PSQL database
  • Independent relational database
  • Something else?

Concerns:

  • Query-able
  • Joinable with mediawiki tables
  • Curation of text fields
Halfak triaged this task as High priority.
Halfak added a subscriber: Ottomata.EditedThu, Jul 27, 4:53 PM

Talked to @Ottomata about using EventBus to free us from settling on a "final" database store. If we can tie into EventBus, we can write to multiple data stores. E.g. we write to a Postgres DB as our main true storage, we also write to a redis LRU for fast lookups along with ORES scores, and we also write to a table in labsDB so that users can easily query the records.

I'm imagining two types of events:

  • judgement -- A user creates or updates a judgement that is stored
  • suppression -- A user with suppression rights flags a judgement to be hidden from view (e.g. it contains copyrighted content or is doxes another user) or restores it as a viewable

Oh! One more thought... We'll need to have the judgement storage operate at high performance so that judgements can be returned with ORES scores and not slow them down too much. So I'm thinking we should maintain a parallel cache storage for judgements (probably redis).

Sometimes, it’s too bad we don’t do the avro thing! Then you could use
this: http://docs.datamountaineer.com/en/latest/redis.html
(and also a PostGRES one)

awight renamed this task from [Spike] potential technical implementations of ORES meta to [Spike] potential technical implementations of Meta-ORES.Thu, Jul 27, 6:10 PM