Page MenuHomePhabricator

Clearly document ExternalStore as append-only interface (with rationale)
Closed, ResolvedPublic

Description


The ES clusters which are not currently being written to have read-only mode enabled in MariaDB, so you can't modify them in any way. They also don't have replication running. So ES is managed by the DBAs as an append-only cluster, a strategy which is enabled by the restricted API presented by MediaWiki. Presenting ES as an append-only store provides some flexibility for implementation options which would not be there if we allowed deletion and update.

I don't think it's worth changing the system just for this script. I suggest just writing new ES blobs and orphaning the old ES rows. Considering the ridiculous inefficiency of text storage in ES, this seems like a minor crime. The old recompression system dropped orphaned rows, so if we ever run that again, storage space would be freed at that time.


Thanks for the explanation, I wasn't aware of this fact. Perhaps it should be documented somewhere (code and/or mw.org)?


A few rough ideas:

  • Ensure class-level doc block for the service class (ExternalStoreAccess) and main interface (ExternalStoreMedium) mention this aspect.
  • Ensure these classes and their core implementations are in the ExtenalStorage doc group.
  • Ensure a Markdown file exists for the ExtenalStorage component, linked to and from its doc group, in which a bit more historic context and rationale is written about its strategy and design choices. (see JobQueue and FileBackend for examples).

Event Timeline

Change 747223 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] externalstore: Improve overall documentation

https://gerrit.wikimedia.org/r/747223

Change 747223 merged by jenkins-bot:

[mediawiki/core@master] externalstore: Improve overall documentation

https://gerrit.wikimedia.org/r/747223