Page MenuHomePhabricator

Run Flow External Store migration in dry-run mode on Beta
Closed, ResolvedPublic

Description

We need to get permission from RelEng when details are finalized.

Related Objects

Event Timeline

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

@jcrespo said he would like to:

  1. Take a backup.
  1. Add more (non-Flow) pages to Beta Cluster (these will go in cluster1). He mentioned wanting to make the script runtime a little more realistic. However, I don't think non-Flow content will have much impact on the runtime, so if we really care about having a longer runtime, I think we need to switch $wgFlowExternalStore back and create more Flow content.
  1. Review the FlowExternalStoreMoveCluster.php script again.

and that he would post here when this is done (of course, ping me anytime if you want to discuss something)

We also want to check with @greg. I pinged him on IRC, but I think he was away.

I'm working half-time right now (parental leave), so IRC is really laggy on my end :)

What should I know about the script running? Locking db/read-only?

I'm working half-time right now (parental leave), so IRC is really laggy on my end :)

What should I know about the script running? Locking db/read-only?

No, it shouldn't do either of those. There might be a slight impact due to a large number of inserts to the new External Store, but I doubt noticeable.

Mentioned in SAL [2016-05-24T10:32:00Z] <jynus> creating backup of beta database just in case T119567

jcrespo moved this task from Triage to In progress on the DBA board.

The backup on /mnt/mysql_backup.sql.bz2 has finally finished (there were issues with labswiki). You can continue with the script.

Really sorry, I accidentally started running it without the dry run option. So it did it for real for some of the rows.

It completed aawiki, cawiki, commonswiki, deploymentwiki (this does not have Flow), dewiki, en_rtlwiki, and started enwiki.

1foreachwiki extensions/Flow/maintenance/FlowExternalStoreMoveCluster.php --from=cluster1 --to=flow_cluster1|tee /home/mattflaschen/external_store_dry_run_all_wikis_2016-06-08.txt
2-----------------------------------------------------------------
3aawiki
4-----------------------------------------------------------------
5aawiki: Completed
6-----------------------------------------------------------------
7arwiki
8-----------------------------------------------------------------
9arwiki: Completed
10-----------------------------------------------------------------
11cawiki
12-----------------------------------------------------------------
13cawiki: Processing 18 rows
14cawiki: Completed
15-----------------------------------------------------------------
16commonswiki
17-----------------------------------------------------------------
18commonswiki: Completed
19-----------------------------------------------------------------
20deploymentwiki
21-----------------------------------------------------------------
22deploymentwiki:
23deploymentwiki: Fatal error: Class 'Flow\Container' not found in /mnt/srv/mediawiki/php-master/extensions/Flow/maintenance/FlowExternalStoreMoveCluster.php on line 269
24-----------------------------------------------------------------
25dewiki
26-----------------------------------------------------------------
27dewiki: Completed
28-----------------------------------------------------------------
29en_rtlwiki
30-----------------------------------------------------------------
31en_rtlwiki: Completed
32-----------------------------------------------------------------
33enwiki
34-----------------------------------------------------------------
35enwiki: Processing 300 rows
36enwiki: Processing 300 rows
37enwiki: Processing 300 rows
38enwiki: Processing 300 rows
39enwiki: Processing 300 rows
40enwiki: Processing 300 rows
41enwiki: Processing 300 rows
42enwiki: Processing 300 rows

Will re-run correctly now.

Dry run completed: P3226

Skimming it, it looks good.

@jcrespo Let me know when I should proceed to finishing the real run on Beta.

Go for it. You may want to have some kind of check/flowdb list usage maybe, thinking in production? I think that only needs foreachwikiin flowdb.dblist, assuming that exists? Maybe that doesn't exist in beta and it was the intention all the time.

Go for it. You may want to have some kind of check/flowdb list usage maybe, thinking in production? I think that only needs foreachwikiin flowdb.dblist, assuming that exists? Maybe that doesn't exist in beta and it was the intention all the time.

This is a good reminder, thanks. It doesn't exist in Beta, but does in production.

Looks fine in vagrant both for dry-run and real run.

vagrant@mediawiki-vagrant:~$ foreachwiki extensions/Flow/maintenance/FlowExternalStoreMoveCluster.php --from=cluster1 --to=flow_cluster1 --dry-run
-----------------------------------------------------------------
centralauthtestwiki
-----------------------------------------------------------------
centralauthtestwiki:  Starting dry run
centralauthtestwiki:  
centralauthtestwiki:  Dry run completed
-----------------------------------------------------------------
frwiki
-----------------------------------------------------------------
frwiki:  Starting dry run
frwiki:  
frwiki:  Dry run completed
-----------------------------------------------------------------
frwiktionarywiki
-----------------------------------------------------------------
frwiktionarywiki:  Starting dry run
frwiktionarywiki:  
frwiktionarywiki:  Dry run completed
-----------------------------------------------------------------
hewiki
-----------------------------------------------------------------
hewiki:  Starting dry run
hewiki:  
hewiki:  Dry run completed
-----------------------------------------------------------------
loginwiki
-----------------------------------------------------------------
loginwiki:  Starting dry run
loginwiki:  
loginwiki:  Dry run completed
-----------------------------------------------------------------
wiki
-----------------------------------------------------------------
wiki:  Starting dry run
wiki:  
wiki:  Dry run completed
-----------------------------------------------------------------
zhwikivoyagewiki
-----------------------------------------------------------------
zhwikivoyagewiki:  Starting dry run
zhwikivoyagewiki:  
zhwikivoyagewiki:  Dry run completed

It's not doing anything in Vagrant because there is only one External Store configured, and the name is 'external', not 'cluster1'.

I could show you how to set up multiple External Store on Vagrant if you want, but it's probably enough to review the Beta logs.

Thx! I've set up External Store on vagrant.