Can the current External Store be backed up before being decommissioned?

@Mattflaschen once new servers are available, data will be transferred without losing records or availability. Data will not be backuped, in the traditional sense, but it will be available on different servers on multiple datacenters, and those different servers can be managed separatelly.

@jcrespo: Flow doesn't currently record it's entries in text: they're stored separately (extension1, flow_revision.rev_content) to be easily accessible cross-wiki. We'd rather not store or duplicate them in tables that are specific to a wiki.
So instead of trying to duplicate Flow's references to DB://cluster24/... & DB://cluster25/... into text, I suggest we set up a new ExternalStore cluster specific to Flow. We're building a script that will loop all existing Flow entries in cluster24 & cluster25 and recreate them elsewhere. Then Flow is no longer blocking anything on cluster24 & cluster25 and trackBlobs.php and recompressTracked.php can be run.
Is that a sane idea?

Eventually, Flow's data might also need to be re-compressed the way recompressTracked.php currently does it, and if we don't move our data to text, we can't use those scripts for Flow's ExternalStore entries.
I think it's better to then recreate recompressTracked.php's functionality specific to Flow, rather than trying to re-use text table & recompressTracked.php: the most complex parts in that script are not so much the compression (which seems mostly isolated in ConcatenatedGzipHistoryBlob easily reusable) but iterating over and updating text, and grouping per page.page_id, all of which are pretty much irrelevant to Flow and we would already have an alternative for with the previous script that would let us copy Flow entries to a new ES cluster.

Is that a sane idea?

@matthiasmullie I do not have enough architecture and flow knowledge to agree or disagree with your suggestion (I only created this task because it was suggested by Tim). I can only own the original issue (lack of space) and have a support role on the compression. Please, other subscribers, comment on this (here or probably better on the more specific T106388).

The only thing I can comment is the operational point of view. A new cluster requires new resources (which requires budget- and we are limited by that). Please agree first if that is sane from the application point of view (or other options are a better idea) and then submit a more detailed request of the consequences in connections, shards, storage changes, latency requirements, etc. so I can better advise on physical requirements for high availability, performance and reliability. You can find me at IRC as jynus. Please note that with higher capacity servers it could be possible to consolidate several services on the same hardware.

My only request is to please, reduce the scope of short-term architectural changes to the minimum possible, as we are on a timer (less than 3 months to failure), and further improvements could be done after the immediate space problem is solved. In particular, new hardware deployment will not be blocked by this ticket.

I think trackBlobs would currently fail the initial integrity check (at least on dewiki and probably other wikis which are old enough) due to the presence of HistoryBlobStub objects in the text table. More details on T108495.

The new servers are about to arrive. There are 2 options for the immediate migration, before compression:

• Stop writing to es2 (blobs_cluster24) and es3 (blobs_cluster25), put those clusters in read-only (old cluster configuration) mode and start writing to blobs_cluster26 and blobs_cluster27, on 2 new shards. Eventually migrate es[123] to a new ro set of servers. This may be easier and faster to do, but for a long time, the new servers would be under-utilized.
• Clone es2 and es3 "as is", on the new servers, with larger space and capacity. Maybe eventually create new clusters, but on the same physical host, before compression. This requires a data copy, but it is not a huge issue, and usage will be good from the beginning.

I would prefer the second option, assuming thing can can be reorganized (old clusters moved around) afterwards with no issue (after all, compression should be that, a reorganization of the data). Anyone has any comments?

It sounds like either will work for Flow.

If there's anything we need to do (e.g. during the time window when switching to the clone), let us know.

We will have to go through a process for Flow later, before the compression.

I am soon going to resolve T105843#1614596. Please be prepared soon to recontinue this task!

Please keep me in the loop, so that I can help, and provide a rollback plan in case something goes wrong.

Noted, thanks. What's the status now?

Data migration done. I would recommend starting testing the compression on a slave on codfw to avoid production impact.

This was mentioned in yesterday's SoS: @jcrespo: I'd like to make sure you get the help you need, but I'm not sure who to task with that. Do you have anyone in mind?

@greg No, I do not know who should be the best fit. This is a "mediawiki-core" change. Who maintains the mysql ORM?

If the answer to that is "nobody" then, someone with some mediawiki/PHP experience. *I will be doing the actual maintenance*, but I lack lots of experience with mediawiki codebase. This will be a stab at its internal guts, and need a second pair of eyes before I bring the whole editing workflow down.

In T106386#1693559, @jcrespo wrote:

@greg No, I do not know who should be the best fit. This is a "mediawiki-core" change. Who maintains the mysql ORM?

If the answer to that is "nobody" then, someone with some mediawiki/PHP experience. *I will be doing the actual maintenance*, but I lack lots of experience with mediawiki codebase. This will be a stab at its internal guts, and need a second pair of eyes before I bring the whole editing workflow down.

There isn't really anyone specifically. Probably a handful of people that could help.

A big question would be whether you want someone in an European based TZ?

Europe is ok, but not a hard blocker.

Tickets to read before starting this: T22757.