Migrate Flow content to new separate logical External Store in production
Open, HighPublic

Description

ExternalStore is nearly full and ops will either buy more storage and/or compress the data.
According to Tim, the script to compress ES data omits entries missing from text table.

I believe we are not currently storing references in text. Let's make sure we do that soon enough. <- outdated, might go with another approach

We plan to solve this by setting up a new External Store (one that will only be used by Flow) then migrating Flow to use that (details at T107610: Setup separate logical External Store for Flow in production). That will then free up the non-Flow one to use the normal compression procedure.

Use:

foreachwikiindblist flow.dblist extensions/Flow/maintenance/FlowExternalStoreMoveCluster.php

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes
greg added a subscriber: greg.Jul 21 2015, 5:06 AM
mark added a subscriber: mark.Jul 21 2015, 2:13 PM

AbuseFilter does:

		$old_id = $dbw->nextSequenceValue( 'text_old_id_seq' );
		$dbw->insert( 'text',
			array(
				'old_id'    => $old_id,
				'old_text'  => $text,
				'old_flags' => implode( ',', $flags ),
			), __METHOD__
		);
		return $dbw->insertId();

see AbuseFilter::storeVarDump()

Change 226544 had a related patch set uploaded (by Matthias Mullie):
Record Flow references to ExternalStore in

https://gerrit.wikimedia.org/r/226544

matthiasmullie renamed this task from Record Flow references to ExternalStore in `text` to Don't block trackBlobs.php and recompressTracked.php.Jul 28 2015, 12:25 PM
matthiasmullie updated the task description. (Show Details)

Suggestion: move Flow's ExternalStore entries on cluster24 & cluster25 elsewhere.
We'll need ops to weigh in here and set up that cluster. This was suggested here: https://phabricator.wikimedia.org/T106386#1487961
If agreed, we can use https://gerrit.wikimedia.org/r/#/c/226544/ to complete the move.

After that, we'll probably need to make a 2nd script to enable the same kind of recompression recompressTracked.php currently provides, but for Flow entries. This is less urgent.

Mattflaschen-WMF lowered the priority of this task from Unbreak Now! to High.Aug 5 2015, 11:12 PM
Mattflaschen-WMF renamed this task from Don't block trackBlobs.php and recompressTracked.php to Migrate Flow content to new separate logical External Store.Aug 6 2015, 11:42 PM
Mattflaschen-WMF updated the task description. (Show Details)
Mattflaschen-WMF added a subscriber: tstarling.

T105843#1654271 indicates the new hardware phase is almost done, so adding this back to Current.

The script will only be run in production after careful testing elsewhere.

The initial steps will be T119566: Add dry-run mode to Flow External Store migration script and T119567: Run Flow External Store migration in dry-run mode on Beta.

Change 226544 merged by jenkins-bot:
Move Flow ExternalStore entries to separate cluster

https://gerrit.wikimedia.org/r/226544

Catrope closed this task as Resolved.Dec 10 2015, 4:33 AM
Catrope added a subscriber: Catrope.
Mattflaschen-WMF reopened this task as Open.Dec 16 2015, 9:53 PM

It hasn't actually been migrated yet.

Thanks to @Volans, new codfw external storage servers were setup. Would the old servers be helpful in any way for this task? (they would have similar specs and contents to real servers- but they are not in production right now. Otherwise, they may be erased, etc: T129452

Yes, they could potentially be used for the dedicated Flow External Store.

jcrespo added a comment.EditedMar 10 2016, 5:17 PM

@matthiasmullie No, those can be around for testing for a bit longer, but they are old and out of warranty (which means they cannot be used for production, only for testing). I was asking if they could be used as test hosts for the script (before being fully decommissioned).

You really want to use the new servers for the final service (10x times faster) and we have 20 TB free on those (and 0 on these).

@matthiasmullie No, those can be around for testing for a bit longer, but they are old and out of warranty (which means they cannot be used for production, only for testing). I was asking if they could be used as test hosts for the script (before being fully decommissioned).

You really want to use the new servers for the final service (10x times faster) and we have 20 TB free on those (and 0 on these).

Okay. I forget, are we doing a test run in production, or just Beta? If we're doing one on prod, we could use it.

(BTW, you @-ed Matthias, when I think you meant to reply to me; we both worked on it though).

Sorry about that, I just preset tab.

are we doing a test run in production

Well, considering we now have a reasonable way to test it closer to production, I was just providing more options in case they were needed (if only to see how much it would take on non-trivial datasets like beta). But it was just a suggestion/offering (which we didn't have before).

Mattflaschen-WMF renamed this task from Migrate Flow content to new separate logical External Store to Migrate Flow content to new separate logical External Store in production.Apr 12 2016, 9:12 PM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 5:40 PM
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptOct 4 2016, 10:54 PM

This ticket still needs to happen. However, I am thinking if we should refactor the external storage servers in a different way other than regular compression (parent ticket). Making the external storage transparent for the application (a pure key-value store) and use a similar strategy, but implemented by opensource mainstream software such as RocksDB or other. I will ask around.