Page MenuHomePhabricator

Add a link engineering: create tables in Wikimedia production
Closed, ResolvedPublic

Description

Create the GrowthExperiments Add Link tables in Wikimedia production:

The tables were added to createExtensionTables.php in 95f6e699 and 40f2e369, and to the private filter in b4fbbb75.

For now, only to testwiki: T277173: Deploy Add Link on testwiki

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@Tgr we're probably getting close enough to do this, I'll let you mark as unstalled and put into Ready for Development if you agree we're ready.

I'd delay this until we are ready to deploy, it saves the effort of having to do schema changes.

@Tgr do you think we're close enough to do this now, and then deploy the puppet change to run refreshLinkRecommendations.php? AIUI the search functionality for hasrecommendation:link is in place, and the DB schema seems unlikely to change at this moment.

I said this in Slack, but I want to note this here as well. Before creation, we need to decide whether the table should be replicated to Toolforge wiki replicas, and tell DBA to make sure correct filtering is set in place. See T272957 as an example of past task asking for a table to be marked as private.

Tgr moved this task from Incoming to In Progress on the Growth-Team (Current Sprint) board.

I'll look into sanitization, then create the tables.

Tgr changed the task status from Stalled to Open.Feb 21 2021, 9:49 PM

I'll look into sanitization, then create the tables.

For the record, adding to $private_tables in puppet should be enough. @Marostegui is the expert on that through :).

That's correct, they need to be added there, before they get added to production.
Once the patch is ready, add me as a reviewer so I can merge and restart mysql on sanitarium hosts, so they pick the new replicating filters excluding those tables.

Thank you!

Change 666216 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/puppet@production] Add GrowthExperiments tables to private_tables

https://gerrit.wikimedia.org/r/666216

Change 666216 merged by Kormat:
[operations/puppet@production] Add GrowthExperiments tables to private_tables

https://gerrit.wikimedia.org/r/666216

Mentioned in SAL (#wikimedia-operations) [2021-02-23T09:40:11Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on 6 hosts with reason: Restart mariadb to pick up config changes T266913

Mentioned in SAL (#wikimedia-operations) [2021-02-23T09:40:18Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on 6 hosts with reason: Restart mariadb to pick up config changes T266913

https://gerrit.wikimedia.org/r/666216 is deployed to all sanitarium hosts, and mariadb has been restarted to pick it up. Let me know when the tables are created, so i can double-check that they don't get replicated.

Tgr changed the task status from Open to Stalled.Feb 23 2021, 2:27 PM

I thought about this and I think we should leave table creation to the last moment. It's fast, schema changes in production are a hassle, and the recommendation caching logic is complicated, so let's wait until it passed product review on beta.

Tgr changed the task status from Stalled to Open.Mar 11 2021, 2:30 PM
Tgr updated the task description. (Show Details)
Tgr updated the task description. (Show Details)

Beta is broken all the time so we are better off deploying to testwiki at least.

Tgr changed the task status from Open to Stalled.Mar 11 2021, 9:26 PM

Done for testwiki in T277173: Deploy Add Link on testwiki. Will get back to it once we deploy to production wikis.

kostajh set Due Date to Apr 5 2021, 10:00 PM.

@Tgr Let's plan to do this on Tuesday? By that point our statsd monitoring code will be in place so we'll be visualize how long it takes to fill the task pool sizes.

If we let the maintenance script roll out with the train (and I don't think those two days would be worth the effort), then we can do the puppet change and the databases on Tuesday, and enable the link recommendations backend on group0/1/2 on Tuesday/Wednesday/Thursday in the post-train backport window.

Oh, sorry, you meant next Tuesday. We can do it tomorrow though, I already scheduled it and this task is trivial anyway.

Actually enabling the Add Link backend will be more complicated; I made T278710: Add a link: production deployment to track that.

Oh, sorry, you meant next Tuesday. We can do it tomorrow though, I already scheduled it and this task is trivial anyway.

Actually enabling the Add Link backend will be more complicated; I made T278710: Add a link: production deployment to track that.

I did mean next Tuesday, April 6. Sorry for the confusion.

The confusion is my fault, the Add Link board did have a date, I just failed to notice it.

kostajh raised the priority of this task from Low to Medium.Apr 30 2021, 10:49 AM

For the record: the tables are being created on new wikis with GrowthExpeirments deployed (as running createExtensionTables.php is a standard part of installing an extension that has database tables). The tables are only needed on the older target wikis.

@Urbanecm_WMF: do you have an example of a wiki with the tables created? I haven't been able to find any yet.

Ohh, i see where i went wrong. These tables are in x1 section, not in the normal mw sections. x1 doesn't get replicated to wikireplicas, so this shouldn't be an issue.

I think this has happened everywhere because the tables are part of the createExtensionTables.php script which was run for other reasons, even on existing wikis (T279853: Migrate mentor/mentee relationship to a separate database table on Wikimedia wikis, probably).

tgr@mwmaint1002:~$ for wiki in `cat /srv/mediawiki/dblists/growthexperiments.dblist | tail -n+2`; do sql $wiki --cluster=extension1 -- -e "show tables like 'growthexperiments%';"; done
+---------------------------------------+
| Tables_in_arwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_bnwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+----------------------------------------+
| Tables_in_cawiki (growthexperiments%)  |
+----------------------------------------+
| growthexperiments_link_recommendations |
| growthexperiments_link_submissions     |
| growthexperiments_mentee_data          |
| growthexperiments_mentor_mentee        |
+----------------------------------------+
+----------------------------------------+
| Tables_in_cswiki (growthexperiments%)  |
+----------------------------------------+
| growthexperiments_link_recommendations |
| growthexperiments_link_submissions     |
| growthexperiments_mentee_data          |
| growthexperiments_mentor_mentee        |
+----------------------------------------+
+---------------------------------------+
| Tables_in_dawiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+----------------------------------------+
| Tables_in_elwiki (growthexperiments%)  |
+----------------------------------------+
| growthexperiments_link_recommendations |
| growthexperiments_link_submissions     |
| growthexperiments_mentee_data          |
| growthexperiments_mentor_mentee        |
+----------------------------------------+
+---------------------------------------+
| Tables_in_eowiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_eswiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_euwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_fawiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_frwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------------+
| Tables_in_frwiktionary (growthexperiments%) |
+---------------------------------------------+
| growthexperiments_mentee_data               |
| growthexperiments_mentor_mentee             |
+---------------------------------------------+
+---------------------------------------+
| Tables_in_hewiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_hiwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_hrwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_huwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_hywiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_idwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_jawiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_kowiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_mswiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_nowiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_plwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_ptwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_rowiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_ruwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_shwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+-------------------------------------------+
| Tables_in_simplewiki (growthexperiments%) |
+-------------------------------------------+
| growthexperiments_mentee_data             |
| growthexperiments_mentor_mentee           |
+-------------------------------------------+
+---------------------------------------+
| Tables_in_sqwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_srwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_svwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_tawiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+-----------------------------------------+
| Tables_in_testwiki (growthexperiments%) |
+-----------------------------------------+
| growthexperiments_link_recommendations  |
| growthexperiments_link_submissions      |
| growthexperiments_mentee_data           |
| growthexperiments_mentor_mentee         |
+-----------------------------------------+
+---------------------------------------+
| Tables_in_tewiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_thwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_trwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+---------------------------------------+
| Tables_in_ukwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+
+----------------------------------------+
| Tables_in_urwiki (growthexperiments%)  |
+----------------------------------------+
| growthexperiments_link_recommendations |
| growthexperiments_link_submissions     |
| growthexperiments_mentee_data          |
| growthexperiments_mentor_mentee        |
+----------------------------------------+
+---------------------------------------+
| Tables_in_viwiki (growthexperiments%) |
+---------------------------------------+
| growthexperiments_mentee_data         |
| growthexperiments_mentor_mentee       |
+---------------------------------------+

So apparently not quite true.

Change 684416 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/WikimediaMaintenance@master] createExtensionTables.php: check whether tables exist

https://gerrit.wikimedia.org/r/684416

I think this has happened everywhere because the tables are part of the createExtensionTables.php script which was run for other reasons, even on existing wikis (T279853: Migrate mentor/mentee relationship to a separate database table on Wikimedia wikis, probably).

I personally run it only for new wikis, because I'm not sure what happens when I run it for wikis with some of the tables existing. For those wikis, I ran mwscript sql.php --wiki=xxwiki --cluster=extension1 /path/to/table/definition.sql.

Mentioned in SAL (#wikimedia-operations) [2021-05-04T11:10:29Z] <Urbanecm> Create growthexperiments_link_recommendations and growthexperiments_link_submissions on arwiki,bnwiki,viwiki x1 (T266913)

kostajh changed the task status from Stalled to Open.May 5 2021, 2:15 PM

Change 684416 merged by jenkins-bot:

[mediawiki/extensions/WikimediaMaintenance@master] createExtensionTables.php: check whether tables exist

https://gerrit.wikimedia.org/r/684416

Not yet done everywhere:

[urbanecm@mwmaint1002 ~]$ sql frwiki --cluster=extension1
wikiadmin@10.64.48.111(frwiki)> select * from growthexperiments_link_submissions  limit 1;
ERROR 1146 (42S02): Table 'frwiki.growthexperiments_link_submissions' doesn't exist
wikiadmin@10.64.48.111(frwiki)> use kowiki;
Database changed
wikiadmin@10.64.48.111(kowiki)> select * from growthexperiments_link_submissions  limit 1;
ERROR 1146 (42S02): Table 'kowiki.growthexperiments_link_submissions' doesn't exist
wikiadmin@10.64.48.111(kowiki)> use ptwiki
Database changed
wikiadmin@10.64.48.111(ptwiki)> select * from growthexperiments_link_submissions  limit 1;
ERROR 1146 (42S02): Table 'ptwiki.growthexperiments_link_submissions' doesn't exist
wikiadmin@10.64.48.111(ptwiki)>

This is only done on wikis we deployed to after 95f6e699 and 40f2e369 and on wikis with add a link enabled.

@kostajh @Tgr Any reason to _not_ create the tables on all wikis, regardless of add a link status there? I find having the tables only on some wikis confusing, and since we'd already have to go through prod schema change process if we want to change anything, it's IMO time to run the creation script on all GE wikis and resolve this.

Yes, it is very confusing to have tables on just a few wikis. Let's create them everywhere (it is fine if they stay empty)

Yes, it is very confusing to have tables on just a few wikis. Let's create them everywhere (it is fine if they stay empty)

Note the extension isn't currently deployed to most wikis -- I was proposing to create them on wikis in growthexperiments.dblist (which is all wikis the extension is currently enabled at). Would creating them only on wikis the extension is available at be ok?

I don't think it's unusual to only have a certain extension table on some wikis. That said, I don't have anything against creating it on all GE wikis or even all wikis. As you say, it won't make much difference for schema changes.

Mentioned in SAL (#wikimedia-operations) [2021-06-22T13:29:14Z] <urbanecm> urbanecm@mwmaint1002:~$ foreachwikiindblist growthexperiments extensions/WikimediaMaintenance/createExtensionTables.php growthexperiments # T266913

I took the liberty to finish this, and create all GrowthExperiments tables at all GrowthExperiments wikis per DBA recommendation. Script output is at P16692. Running for wiki in $(cat /srv/mediawiki/dblists/growthexperiments.dblist | tail -n+2); do sql $wiki --cluster=extension1 -- -e "show tables like 'growthexperiments%';"; done shows the tables are indeed live everywhere.

Closing.