Page MenuHomePhabricator

Set up Flow-specific External Store cluster on Beta (secondary to the main one)
Closed, ResolvedPublic

Description

There needs to be a second External Store "cluster" so we can test cutting over.

I think we can just use the main wiki databases, but with a different 'blobs table' parameter.

Use blobs_flow1 (naming from https://phabricator.wikimedia.org/T107610#1506876)

Event Timeline

We have a first external store?

We do now (though it's not being used yet, pending https://gerrit.wikimedia.org/r/#/c/282440/ ). However, there is a missing blocker here.

Created second one (I used a less clear name the first time so I had to fix and re-run):

export MEDIAWIKI_STAGING_DIR; ../../extensions/WikimediaMaintenance/storage/make-all-blobs deployment-db1.deployment-prep.eqiad.wmflabs blobs_flow1

Patch coming soon to cut over (needs to be enabled before we can run the dry-run script).

Mentioned in SAL [2016-05-11T01:44:18Z] <matt_flaschen> Created Flow-specific External Store tables (blobs_flow1) on all wiki databases on Beta Cluster: T128417

Mattflaschen-WMF renamed this task from Set up second External Store cluster on Beta to Set up Flow-specific External Store cluster on Beta (secondary to the main one).May 11 2016, 1:47 AM

Change 288323 had a related patch set uploaded (by Mattflaschen):
Add Flow-specific External Store on Beta Cluster

https://gerrit.wikimedia.org/r/288323

Change 288323 merged by jenkins-bot:
Add Flow-specific External Store on Beta Cluster

https://gerrit.wikimedia.org/r/288323

There was a minor problem with that patch, so I had to do rOMWC1317ae266772: Fix Flow external store: Protocol was missing. However, it's done and works now. From tests on http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow :

mysql> SELECT HEX(rev_id), HEX(rev_type_id), rev_content, rev_flags FROM flow_revision ORDER BY rev_id DESC LIMIT 7;
+------------------------+------------------------+----------------------+------------------------------------------+
| HEX(rev_id)            | HEX(rev_type_id)       | rev_content          | rev_flags                                |
+------------------------+------------------------+----------------------+------------------------------------------+
| 0552975C385265DDB4696B | 0551CE4140A1E14F40AF18 | DB://flow_cluster1/6 | utf-8,gzip,html,external                 |
| 05529758A039E14F40AF18 | 05529758A039E14F40AF18 | DB://flow_cluster1/5 | utf-8,gzip,html,external                 |
| 055297579259E14F40AF18 | 055297579259E14F40AF18 | DB://flow_cluster1/4 | utf-8,gzip,html,external                 |
| 055297579245E24F40AF18 | 055297579245E14F40AF18 | DB://flow_cluster1/3 | utf-8,gzip,topic-title-wikitext,external |
| 055297550E49E14F40AF18 | 055297550E49E14F40AF18 | DB://flow_cluster1/2 | utf-8,gzip,html,external                 |
| 055297550D5DE14F40AF18 | 055297550BF1E14F40AF18 | DB://flow_cluster1/1 | utf-8,gzip,topic-title-wikitext,external |
| 0552897AA28A65DDB4696B | 0539E8BC9D23F1DDB4696B | DB://cluster1/11309  | utf-8,gzip,html,external                 |
+------------------------+------------------------+----------------------+------------------------------------------+
7 rows in set (0.00 sec)

mysql> SELECT * FROM blobs_flow1 LIMIT 10;
+---------+---------------------------------------------------------------------------------------------+
| blob_id | blob_text                                                                                   |
+---------+---------------------------------------------------------------------------------------------+
|       1 | �K-W(�/�L                                                                                     |
|       2 | �)PHI,I�-H,*��L�U�VJ).R��6�16�1�1��U�+�,�Pp��/�-.HM�L�LVp�(I-�K�.�/J��/�                                             |
|       3 | 
            J-I-.Q�K-W(�/�L                                                                              |
|       4 | �)PHI,I�-H,*��L�U�VJ).R��6�16�1�1��U�+�,�Pp��/�-.HM�L�LVH�(I-�K�Q(.�/J��/�                                             |
|       5 | ��@0 �_�t�,�/���ѓ6i�rWAĿ{�'p6ے,K
n(^��U�ݴ�h�~�8� �H��SP {�)���-�a�3�a#HN�U_��                                    |
|       6 | �)PHI,I�-H,*��L�U�VJ).R��6�14�1�1��U�su�
                                                    �Rp�O-V�H-J��/�                                                        |
+---------+---------------------------------------------------------------------------------------------+
6 rows in set (0.00 sec)

I also tested on one other Beta Cluster wiki (Catalan Wikipedia) just for completeness.

  • Re-run the above query for most recent entries
MariaDB [enwiki]> select HEX(rev_id), HEX(rev_type_id), rev_content, rev_flags  from flow_revision order by rev_mod_timestamp desc limit 7;
+------------------------+------------------------+--------------------------+---------------------------------------------------------------------+
| HEX(rev_id)            | HEX(rev_type_id)       | rev_content              | rev_flags                                                           |
+------------------------+------------------------+--------------------------+---------------------------------------------------------------------+
| 055581A78F3DE14F40AF18 | 0555819D8C29E14F40AF18 | DB://flow_cluster1/12125 | utf-8,gzip,topic-title-wikitext,external                            |
| 055581A44B6A65DDB4696B | 0555819E881665DDB4696B | DB://flow_cluster1/12124 | utf-8,gzip,topic-title-wikitext,external                            |
| 0555694F9B2DE14F40AF18 | 0550BEC83CE1E14F40AF18 | DB://flow_cluster1/7792  | utf-8,gzip,topic-title-wikitext,external,external,external,external |
| 0555694AEF1DE14F40AF18 | 05518E23E2A9E14F40AF18 | DB://flow_cluster1/7790  | utf-8,gzip,topic-title-wikitext,external,external                   |
| 0555693D406DE14F40AF18 | 0550BEC83CE1E14F40AF18 | DB://flow_cluster1/7789  | utf-8,gzip,topic-title-wikitext,external,external                   |
| 05556869E8C5E14F40AF18 | 0543CB7BDE46D2DDB4696B | DB://flow_cluster1/7782  | utf-8,gzip,wikitext,external                                        |
| 055568460D5265DDB4696B | 054B38A67DD354DDB4696B | DB://flow_cluster1/7774  | utf-8,gzip,topic-title-wikitext,external                            |
+------------------------+------------------------+--------------------------+---------------------------------------------------------------------+
7 rows in set (0.28 sec)
  • General regression testing is done in betalabs
  • Re-run the above query for most recent entries
MariaDB [enwiki]> select HEX(rev_id), HEX(rev_type_id), rev_content, rev_flags  from flow_revision order by rev_mod_timestamp desc limit 7;

Just FYI, rev_mod_timestamp is the timestamp of a moderation action, and is not always set (it can be null). If you just want to order by timestamp, use rev_id.

jmatazzoni closed this task as Resolved.Jun 22 2016, 10:51 AM

Change 321810 had a related patch set uploaded (by Mattflaschen):
Add Flow External Store to addWiki.php

https://gerrit.wikimedia.org/r/321810

Change 321810 merged by jenkins-bot:
Add Flow External Store to addWiki.php

https://gerrit.wikimedia.org/r/321810