Run Flow External Store migration in dry-run mode on Beta
Closed, ResolvedPublic

Description

We need to get permission from RelEng when details are finalized.

Mattflaschen-WMF updated the task description. (Show Details)
Mattflaschen-WMF raised the priority of this task from to Needs Triage.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Catrope triaged this task as High priority.Dec 19 2015, 1:31 AM

@jcrespo said he would like to:

  1. Take a backup.
  1. Add more (non-Flow) pages to Beta Cluster (these will go in cluster1). He mentioned wanting to make the script runtime a little more realistic. However, I don't think non-Flow content will have much impact on the runtime, so if we really care about having a longer runtime, I think we need to switch $wgFlowExternalStore back and create more Flow content.
  1. Review the FlowExternalStoreMoveCluster.php script again.

and that he would post here when this is done (of course, ping me anytime if you want to discuss something)

We also want to check with @greg. I pinged him on IRC, but I think he was away.

greg added a comment.May 12 2016, 6:25 PM

I'm working half-time right now (parental leave), so IRC is really laggy on my end :)

What should I know about the script running? Locking db/read-only?

I'm working half-time right now (parental leave), so IRC is really laggy on my end :)

What should I know about the script running? Locking db/read-only?

No, it shouldn't do either of those. There might be a slight impact due to a large number of inserts to the new External Store, but I doubt noticeable.

greg added a comment.May 12 2016, 9:21 PM

go ahead from me

Mentioned in SAL [2016-05-24T10:32:00Z] <jynus> creating backup of beta database just in case T119567

jcrespo moved this task from Triage to In progress on the DBA board.
jcrespo moved this task from In progress to Done on the DBA board.May 25 2016, 11:53 AM

The backup on /mnt/mysql_backup.sql.bz2 has finally finished (there were issues with labswiki). You can continue with the script.

Really sorry, I accidentally started running it without the dry run option. So it did it for real for some of the rows.

It completed aawiki, cawiki, commonswiki, deploymentwiki (this does not have Flow), dewiki, en_rtlwiki, and started enwiki.

1​foreachwiki extensions/Flow/maintenance/FlowExternalStoreMoveCluster.php --from=cluster1 --to=flow_cluster1|tee /home/mattflaschen/external_store_dry_run_all_wikis_2016-06-08.txt
2​-----------------------------------------------------------------
3​aawiki
4​-----------------------------------------------------------------
5​aawiki: Completed
6​-----------------------------------------------------------------
7​arwiki
8​-----------------------------------------------------------------
9​arwiki: Completed
10​-----------------------------------------------------------------
11​cawiki
12​-----------------------------------------------------------------
13​cawiki: Processing 18 rows
14​cawiki: Completed
15​-----------------------------------------------------------------
16​commonswiki
17​-----------------------------------------------------------------
18​commonswiki: Completed
19​-----------------------------------------------------------------
20​deploymentwiki
21​-----------------------------------------------------------------
22​deploymentwiki:
23​deploymentwiki: Fatal error: Class 'Flow\Container' not found in /mnt/srv/mediawiki/php-master/extensions/Flow/maintenance/FlowExternalStoreMoveCluster.php on line 269
24​-----------------------------------------------------------------
25​dewiki
26​-----------------------------------------------------------------
27​dewiki: Completed
28​-----------------------------------------------------------------
29​en_rtlwiki
30​-----------------------------------------------------------------
31​en_rtlwiki: Completed
32​-----------------------------------------------------------------
33​enwiki
34​-----------------------------------------------------------------
35​enwiki: Processing 300 rows
36​enwiki: Processing 300 rows
37​enwiki: Processing 300 rows
38​enwiki: Processing 300 rows
39​enwiki: Processing 300 rows
40​enwiki: Processing 300 rows
41​enwiki: Processing 300 rows
42​enwiki: Processing 300 rows

Will re-run correctly now.

Dry run completed: P3226

Skimming it, it looks good.

@jcrespo Let me know when I should proceed to finishing the real run on Beta.

Go for it. You may want to have some kind of check/flowdb list usage maybe, thinking in production? I think that only needs foreachwikiin flowdb.dblist, assuming that exists? Maybe that doesn't exist in beta and it was the intention all the time.

Go for it. You may want to have some kind of check/flowdb list usage maybe, thinking in production? I think that only needs foreachwikiin flowdb.dblist, assuming that exists? Maybe that doesn't exist in beta and it was the intention all the time.

This is a good reminder, thanks. It doesn't exist in Beta, but does in production.

Looks fine in vagrant both for dry-run and real run.

vagrant@mediawiki-vagrant:~$ foreachwiki extensions/Flow/maintenance/FlowExternalStoreMoveCluster.php --from=cluster1 --to=flow_cluster1 --dry-run
-----------------------------------------------------------------
centralauthtestwiki
-----------------------------------------------------------------
centralauthtestwiki:  Starting dry run
centralauthtestwiki:  
centralauthtestwiki:  Dry run completed
-----------------------------------------------------------------
frwiki
-----------------------------------------------------------------
frwiki:  Starting dry run
frwiki:  
frwiki:  Dry run completed
-----------------------------------------------------------------
frwiktionarywiki
-----------------------------------------------------------------
frwiktionarywiki:  Starting dry run
frwiktionarywiki:  
frwiktionarywiki:  Dry run completed
-----------------------------------------------------------------
hewiki
-----------------------------------------------------------------
hewiki:  Starting dry run
hewiki:  
hewiki:  Dry run completed
-----------------------------------------------------------------
loginwiki
-----------------------------------------------------------------
loginwiki:  Starting dry run
loginwiki:  
loginwiki:  Dry run completed
-----------------------------------------------------------------
wiki
-----------------------------------------------------------------
wiki:  Starting dry run
wiki:  
wiki:  Dry run completed
-----------------------------------------------------------------
zhwikivoyagewiki
-----------------------------------------------------------------
zhwikivoyagewiki:  Starting dry run
zhwikivoyagewiki:  
zhwikivoyagewiki:  Dry run completed
Mattflaschen-WMF added a comment.EditedJun 17 2016, 12:26 AM

It's not doing anything in Vagrant because there is only one External Store configured, and the name is 'external', not 'cluster1'.

I could show you how to set up multiple External Store on Vagrant if you want, but it's probably enough to review the Beta logs.

Thx! I've set up External Store on vagrant.

jmatazzoni closed this task as Resolved.Jun 22 2016, 10:48 AM