Page MenuHomePhabricator

Compress data at external storage
Open, NormalPublic

Description

External storage can be compressed by about a factor of 20 by running the maintenance scripts trackBlobs.php and recompressTracked.php. There has been recently a space shortage on ES hosts (T105843), and that is going to be fixed short term in hardware, but investigation on software/operations should ensue.

Two important issues, however, must be fixed first:

  • Record Flow references to ExternalStore in text T106363
  • The script has been untested for years, and it should be checked before its execution (T106388, plus additional general testing)

References:

See T119056: External Storage on codfw (es2005-2010) is consuming 100-90GB of disk space per server and per month and it has 370GB available.

Related Objects

Event Timeline

jcrespo created this task.Jul 21 2015, 4:25 PM
jcrespo updated the task description. (Show Details)
jcrespo raised the priority of this task from to Needs Triage.
jcrespo added a subscriber: jcrespo.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 21 2015, 4:25 PM
jcrespo updated the task description. (Show Details)Jul 21 2015, 4:40 PM

Can the current External Store be backed up before being decommissioned?

@Mattflaschen once new servers are available, data will be transferred without losing records or availability. Data will not be backuped, in the traditional sense, but it will be available on different servers on multiple datacenters, and those different servers can be managed separatelly.

@jcrespo: Flow doesn't currently record it's entries in text: they're stored separately (extension1, flow_revision.rev_content) to be easily accessible cross-wiki. We'd rather not store or duplicate them in tables that are specific to a wiki.
So instead of trying to duplicate Flow's references to DB://cluster24/... & DB://cluster25/... into text, I suggest we set up a new ExternalStore cluster specific to Flow. We're building a script that will loop all existing Flow entries in cluster24 & cluster25 and recreate them elsewhere. Then Flow is no longer blocking anything on cluster24 & cluster25 and trackBlobs.php and recompressTracked.php can be run.
Is that a sane idea?

Eventually, Flow's data might also need to be re-compressed the way recompressTracked.php currently does it, and if we don't move our data to text, we can't use those scripts for Flow's ExternalStore entries.
I think it's better to then recreate recompressTracked.php's functionality specific to Flow, rather than trying to re-use text table & recompressTracked.php: the most complex parts in that script are not so much the compression (which seems mostly isolated in ConcatenatedGzipHistoryBlob easily reusable) but iterating over and updating text, and grouping per page.page_id, all of which are pretty much irrelevant to Flow and we would already have an alternative for with the previous script that would let us copy Flow entries to a new ES cluster.

Is that a sane idea?

@matthiasmullie I do not have enough architecture and flow knowledge to agree or disagree with your suggestion (I only created this task because it was suggested by Tim). I can only own the original issue (lack of space) and have a support role on the compression. Please, other subscribers, comment on this (here or probably better on the more specific T106388).

The only thing I can comment is the operational point of view. A new cluster requires new resources (which requires budget- and we are limited by that). Please agree first if that is sane from the application point of view (or other options are a better idea) and then submit a more detailed request of the consequences in connections, shards, storage changes, latency requirements, etc. so I can better advise on physical requirements for high availability, performance and reliability. You can find me at IRC as jynus. Please note that with higher capacity servers it could be possible to consolidate several services on the same hardware.

My only request is to please, reduce the scope of short-term architectural changes to the minimum possible, as we are on a timer (less than 3 months to failure), and further improvements could be done after the immediate space problem is solved. In particular, new hardware deployment will not be blocked by this ticket.

Krenair added a subscriber: Krenair.Aug 9 2015, 3:33 PM

I think trackBlobs would currently fail the initial integrity check (at least on dewiki and probably other wikis which are old enough) due to the presence of HistoryBlobStub objects in the text table. More details on T108495.

mysql:wikiadmin@db2038.codfw.wmnet [dewiki]> select count(*) from text where old_flags LIKE '%object%' AND old_flags NOT LIKE '%external%' AND LOWER(CONVERT(LEFT(old_text,22) USING latin1)) = 'o:15:"historyblobstub"';
+----------+
| count(*) |
+----------+
|     3449 |
+----------+
1 row in set (1 min 46.46 sec)
mysql:wikiadmin@db2034.codfw.wmnet [enwiki]> select count(*) from text where old_flags LIKE '%object%' AND old_flags NOT LIKE '%external%' AND LOWER(CONVERT(LEFT(old_text,22) USING latin1)) = 'o:15:"historyblobstub"';
+----------+
| count(*) |
+----------+
|     2602 |
+----------+
1 row in set (9 min 56.51 sec)
MoritzMuehlenhoff triaged this task as Normal priority.Aug 21 2015, 8:29 AM

The new servers are about to arrive. There are 2 options for the immediate migration, before compression:

  • Stop writing to es2 (blobs_cluster24) and es3 (blobs_cluster25), put those clusters in read-only (old cluster configuration) mode and start writing to blobs_cluster26 and blobs_cluster27, on 2 new shards. Eventually migrate es[123] to a new ro set of servers. This may be easier and faster to do, but for a long time, the new servers would be under-utilized.
  • Clone es2 and es3 "as is", on the new servers, with larger space and capacity. Maybe eventually create new clusters, but on the same physical host, before compression. This requires a data copy, but it is not a huge issue, and usage will be good from the beginning.

I would prefer the second option, assuming thing can can be reorganized (old clusters moved around) afterwards with no issue (after all, compression should be that, a reorganization of the data). Anyone has any comments?

It sounds like either will work for Flow.

If there's anything we need to do (e.g. during the time window when switching to the clone), let us know.

We will have to go through a process for Flow later, before the compression.

I am soon going to resolve T105843#1614596. Please be prepared soon to recontinue this task!

Please keep me in the loop, so that I can help, and provide a rollback plan in case something goes wrong.

Noted, thanks. What's the status now?

Data migration done. I would recommend starting testing the compression on a slave on codfw to avoid production impact.

jcrespo raised the priority of this task from Normal to High.Sep 27 2015, 3:22 PM

We should start by creating a new cluster (enwiki table has surpased the 1 TB size). Inserting to a separate table will be faster.

I can own this, but I need support from someone with good mediawiki database code knowledge to +1 my configuration patches.

Augmenting the priority because codfw will fail in 1-2 months unless we do this.

greg added a comment.Oct 1 2015, 3:16 PM

This was mentioned in yesterday's SoS: @jcrespo: I'd like to make sure you get the help you need, but I'm not sure who to task with that. Do you have anyone in mind?

@greg No, I do not know who should be the best fit. This is a "mediawiki-core" change. Who maintains the mysql ORM?

If the answer to that is "nobody" then, someone with some mediawiki/PHP experience. *I will be doing the actual maintenance*, but I lack lots of experience with mediawiki codebase. This will be a stab at its internal guts, and need a second pair of eyes before I bring the whole editing workflow down.

Reedy added a subscriber: Reedy.Oct 1 2015, 4:36 PM

@greg No, I do not know who should be the best fit. This is a "mediawiki-core" change. Who maintains the mysql ORM?

If the answer to that is "nobody" then, someone with some mediawiki/PHP experience. *I will be doing the actual maintenance*, but I lack lots of experience with mediawiki codebase. This will be a stab at its internal guts, and need a second pair of eyes before I bring the whole editing workflow down.

There isn't really anyone specifically. Probably a handful of people that could help.

A big question would be whether you want someone in an European based TZ?

Europe is ok, but not a hard blocker.

Tickets to read before starting this: T22757.

fgiunchedi lowered the priority of this task from High to Normal.Dec 1 2016, 8:18 PM
fgiunchedi added a subscriber: fgiunchedi.

Lowering to 'normal', looks like we aren't under high pressure now for ES compression

Krinkle moved this task from Backlog to Unused on the MediaWiki-Database board.Jul 28 2018, 10:09 PM
greg removed a subscriber: greg.Feb 12 2019, 11:21 PM