Page MenuHomePhabricator

`pr_index` to be replicated to Labs public databases
Closed, ResolvedPublic

Description

In T113673, @Glaisher said

It looks like pr_index is not replicated to Labs public databases. It might be useful to request it to be available there on a new task. It doesn't look like there's private data on this table (it just stores integers only) so I think it will be accepted.

Details

Related Gerrit Patches:
operations/puppet : productionpr_index table is not private

Event Timeline

Billinghurst raised the priority of this task from to Needs Triage.
Billinghurst updated the task description. (Show Details)
Billinghurst added a project: Cloud-Services.
Billinghurst added subscribers: Billinghurst, Glaisher.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2015, 12:22 PM
Reedy set Security to None.Sep 26 2015, 12:28 PM
Reedy added a subscriber: Tpt.
chasemp triaged this task as Low priority.Nov 30 2015, 4:53 PM
chasemp added a subscriber: chasemp.
Tpt added a comment.Jan 8 2016, 9:35 PM

The table is already in the maintain-replicas.pl script but it doesn't seem to be replicated on labs.

See https://git.wikimedia.org/blob/operations%2Fsoftware.git/master/maintain-replicas%2Fmaintain-replicas.pl#L82

Tpt added a subscriber: Andrew.Feb 21 2016, 4:27 PM

Can you provide more context of what it is missing, and test it has not been added yet?

BTW, Labs is a very wide task- you should add it to the tracking ticket T150767 and not T50930, or otherwise this will be lost (this is probably a view/filtering issue, not a replication issue).

Can you provide more context of what it is missing

The pr_index table managed by the ProofreadPage extension and present on all Wikisources should be replicated on labs.

and test it has not been added yet?

mysql --defaults-file="${HOME}"/replica.my.cnf -h enwikisource.labsdb enwikisource_p
show tables;

does not contains pr_index in the list of tables.

BTW, Labs is a very wide task- you should add it to the tracking ticket T150767 and not T50930, or otherwise this will be lost (this is probably a view/filtering issue, not a replication issue).

Done. Yes it's definitely a view/filtering issue.

Reedy moved this task from Backlog to Wiki replicas on the Data-Services board.
Reedy added a subscriber: Reedy.

Can a DBA check why pr_index isn't being replicated? Thanks!

They are not being replicated because they are marked as private table: https://github.com/wikimedia/puppet/blob/production/manifests/realm.pp#L201
Why? That I don't know - I guess historic reasons?

Change 375347 had a related patch set uploaded (by Tpt; owner: Tpt):
[operations/puppet@production] pp_index table is not private

https://gerrit.wikimedia.org/r/375347

Security seems to agree on it being possibly public: https://phabricator.wikimedia.org/T103011#3536648

I will ask him to +1 that patch.

Once that is merged, it may take some time to deploy, as we have to create the tables copying it from production- apparently it is only used on wikisource, but it exists on all wikis.

Yeah - I thought about creating it on enwikisource to unblock this ticket first and then slowly create it on the big wikis and so forth

I thought about creating it on enwikisource to unblock this ticket first

Sadly, it is all or none, because replication will break otherwise (breaking wiksources and the other wikis, too).

Reedy added a comment.Sep 1 2017, 11:47 AM

Wow... It's on enwiki

https://github.com/wikimedia/mediawiki-extensions-ProofreadPage/blob/master/sql/ProofreadIndex.sql

I'd propose dropping it from anywhere that isn't in this list...

'wmgUseProofreadPage' => [
	'default' => false,
	'wikisource' => true,
	'sourceswiki' => true, // FIXME: should be a part of wikisource (see T91534)
	'test2wiki' => true,
	'frrwiki' => true, // T40023
],

Wow... It's on enwiki
https://github.com/wikimedia/mediawiki-extensions-ProofreadPage/blob/master/sql/ProofreadIndex.sql
I'd propose dropping it from anywhere that isn't in this list...

'wmgUseProofreadPage' => [
	'default' => false,
	'wikisource' => true,
	'sourceswiki' => true, // FIXME: should be a part of wikisource (see T91534)
	'test2wiki' => true,
	'frrwiki' => true, // T40023
],

More than happy to do that :-)

I'd propose dropping it from anywhere that isn't in this list...

I would support such an action, if it is an optional extension, it should not create "table spam" on other databases (it slows down metadata retrieving). It is not a huge problem, but it ends up adding up. I would create a separate task for "Deleting tables", as a child of the usual common task, and get this blocked on such a task.

Reedy added a comment.Sep 1 2017, 11:54 AM

I'm guessing someone messed up onetime running it over all wikis, rather than a subset. Or at some point, maybe we didn't have a way of making it easier

I'll create a subtask

Marostegui moved this task from Triage to Next on the DBA board.Sep 1 2017, 12:04 PM
Marostegui moved this task from Next to In progress on the DBA board.Sep 1 2017, 1:45 PM
jcrespo added a comment.EditedSep 5 2017, 3:04 PM

With T174782 solved, the next step is to, for each of the 7 shards:

  1. stop replication on sanitarium
  2. copy the tables, on existing wikis (wikisource and a few others) into sanitarium hosts and labs
  3. drop the filtering deploying https://gerrit.wikimedia.org/r/375347 and restarting the server would be the easiest way
  4. restart replication
  5. check no errors or unintended effects have been created

Then, setup the views with a followup patch.

Marostegui moved this task from In progress to Next on the DBA board.Oct 6 2017, 6:26 AM

Change 375347 merged by Jcrespo:
[operations/puppet@production] pr_index table is not private

https://gerrit.wikimedia.org/r/375347

jcrespo moved this task from Next to Done on the DBA board.Nov 30 2017, 10:50 AM

This is now done- we may need a rolling restart of sanitarium hosts, but the chance of a new proofread extension on a non-s3 wiki happening before the next natural restart is very low. The ball is now on cloud-services-team to add the appropriate view so it can be exposed to database users:

root@labsdb1009[enwikisource]> SELECT count(*) FROM pr_index;
+----------+
| count(*) |
+----------+
|    11402 |
+----------+
1 row in set (0.00 sec)

root@labsdb1009[enwikisource]> SELECT count(*) FROM enwiki_p.pr_index;
ERROR 1146 (42S02): Table 'enwiki_p.pr_index' doesn't exist
Reedy renamed this task from `pr_index`to be replicated to Labs public databases to `pr_index` to be replicated to Labs public databases.Nov 30 2017, 1:55 PM
Marostegui closed this task as Resolved.Dec 21 2018, 7:40 AM
Marostegui assigned this task to jcrespo.

This is all done and it is available on all the wikisources on the labs hosts:

root@cumin1001:~# for i in `mysql.py -BN -hlabsdb1010 -e "show databases like '%source_p';"`; do echo $i; mysql.py -hlabsdb1010 $i -e "select count(*) from pr_index;";done
angwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
arwikisource_p
+----------+
| count(*) |
+----------+
|        9 |
+----------+
aswikisource_p
+----------+
| count(*) |
+----------+
|       16 |
+----------+
azwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
bewikisource_p
+----------+
| count(*) |
+----------+
|      338 |
+----------+
bgwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
bnwikisource_p
+----------+
| count(*) |
+----------+
|     4076 |
+----------+
brwikisource_p
+----------+
| count(*) |
+----------+
|      321 |
+----------+
bswikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
cawikisource_p
+----------+
| count(*) |
+----------+
|      786 |
+----------+
cswikisource_p
+----------+
| count(*) |
+----------+
|      179 |
+----------+
cywikisource_p
+----------+
| count(*) |
+----------+
|        2 |
+----------+
dawikisource_p
+----------+
| count(*) |
+----------+
|      129 |
+----------+
dewikisource_p
+----------+
| count(*) |
+----------+
|     2483 |
+----------+
elwikisource_p
+----------+
| count(*) |
+----------+
|      177 |
+----------+
enwikisource_p
+----------+
| count(*) |
+----------+
|    13388 |
+----------+
eowikisource_p
+----------+
| count(*) |
+----------+
|       15 |
+----------+
eswikisource_p
+----------+
| count(*) |
+----------+
|     1551 |
+----------+
etwikisource_p
+----------+
| count(*) |
+----------+
|      104 |
+----------+
euwikisource_p
+----------+
| count(*) |
+----------+
|       76 |
+----------+
fawikisource_p
+----------+
| count(*) |
+----------+
|       47 |
+----------+
fiwikisource_p
+----------+
| count(*) |
+----------+
|       29 |
+----------+
fowikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
frwikisource_p
+----------+
| count(*) |
+----------+
|    16670 |
+----------+
glwikisource_p
+----------+
| count(*) |
+----------+
|        2 |
+----------+
guwikisource_p
+----------+
| count(*) |
+----------+
|       77 |
+----------+
hewikisource_p
+----------+
| count(*) |
+----------+
|       69 |
+----------+
hrwikisource_p
+----------+
| count(*) |
+----------+
|       19 |
+----------+
htwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
huwikisource_p
+----------+
| count(*) |
+----------+
|       19 |
+----------+
hywikisource_p
+----------+
| count(*) |
+----------+
|      372 |
+----------+
idwikisource_p
+----------+
| count(*) |
+----------+
|      388 |
+----------+
iswikisource_p
+----------+
| count(*) |
+----------+
|       17 |
+----------+
itwikisource_p
+----------+
| count(*) |
+----------+
|     2349 |
+----------+
jawikisource_p
+----------+
| count(*) |
+----------+
|       62 |
+----------+
knwikisource_p
+----------+
| count(*) |
+----------+
|      302 |
+----------+
kowikisource_p
+----------+
| count(*) |
+----------+
|       34 |
+----------+
lawikisource_p
+----------+
| count(*) |
+----------+
|      215 |
+----------+
liwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
ltwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
mkwikisource_p
+----------+
| count(*) |
+----------+
|        9 |
+----------+
mlwikisource_p
+----------+
| count(*) |
+----------+
|      266 |
+----------+
mrwikisource_p
+----------+
| count(*) |
+----------+
|       76 |
+----------+
nlwikisource_p
+----------+
| count(*) |
+----------+
|      230 |
+----------+
nowikisource_p
+----------+
| count(*) |
+----------+
|      680 |
+----------+
orwikisource_p
+----------+
| count(*) |
+----------+
|       84 |
+----------+
pawikisource_p
+----------+
| count(*) |
+----------+
|       93 |
+----------+
plwikisource_p
+----------+
| count(*) |
+----------+
|     3113 |
+----------+
pmswikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
ptwikisource_p
+----------+
| count(*) |
+----------+
|      540 |
+----------+
rowikisource_p
+----------+
| count(*) |
+----------+
|       32 |
+----------+
ruwikisource_p
+----------+
| count(*) |
+----------+
|     1501 |
+----------+
sahwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
sawikisource_p
+----------+
| count(*) |
+----------+
|      202 |
+----------+
skwikisource_p
+----------+
| count(*) |
+----------+
|        1 |
+----------+
slwikisource_p
+----------+
| count(*) |
+----------+
|       35 |
+----------+
srwikisource_p
+----------+
| count(*) |
+----------+
|       30 |
+----------+
svwikisource_p
+----------+
| count(*) |
+----------+
|      609 |
+----------+
tawikisource_p
+----------+
| count(*) |
+----------+
|     2186 |
+----------+
tewikisource_p
+----------+
| count(*) |
+----------+
|      408 |
+----------+
thwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
trwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
ukwikisource_p
+----------+
| count(*) |
+----------+
|      890 |
+----------+
vecwikisource_p
+----------+
| count(*) |
+----------+
|       96 |
+----------+
viwikisource_p
+----------+
| count(*) |
+----------+
|       20 |
+----------+
yiwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
zh_min_nanwikisource_p
+----------+
| count(*) |
+----------+
|        0 |
+----------+
zhwikisource_p
+----------+
| count(*) |
+----------+
|     4141 |
+----------+