Write script to backfill echo_unread_wikis table
Closed, ResolvedPublic

Description

We should have a script that will backfill the echo_unseen_wikis table instead of waiting on users to trigger cache invalidation through receiving or reading notifications.

Legoktm created this task.Nov 24 2015, 5:21 PM
Legoktm updated the task description. (Show Details)
Legoktm raised the priority of this task from to Normal.
Legoktm added a subscriber: Legoktm.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 24 2015, 5:21 PM
SBisson added a subscriber: SBisson.Dec 8 2015, 9:15 PM

I see that NotifUser::resetNotificationCount does that in different context.

Any reasons why looping on the users and calling it would be a bad idea?

It has unintended side-effects that we probably don't want like setting it in memcached and invalidating the user cache. Also if possible we should aim to do batch inserts.

Catrope renamed this task from Write script to backfill echo_unseen_wikis table to Write script to backfill echo_unread_wikis table.Dec 14 2015, 5:25 PM

Change 259265 had a related patch set uploaded (by Matthias Mullie):
Backfill echo_unread_wikis table

https://gerrit.wikimedia.org/r/259265

Change 259265 merged by jenkins-bot:
Backfill echo_unread_wikis table

https://gerrit.wikimedia.org/r/259265

wikishared.echo_unread_wikis table is properly updated for the following cases:

  1. when a user initiates an action that sends alerts/notifications to another user
  2. when a recipient of notifications from the previous step marks some(all) messages/alerts
  3. in flyout or
  4. go to Notifictions page.

Technically, that is testing the normal flow, not the backfill.

We should be able to test the backfill as part of the MediaWiki-Vagrant stuff. I'm working on a patch for this now (to auto-run it on provision).

Change 264257 had a related patch set uploaded (by Mattflaschen):
Backfill echo_unread_wikis table after it is created

https://gerrit.wikimedia.org/r/264257

@Etonkovidova If you want to test this on MediaWiki-Vagrant:

  1. DROP the echo_unread_wikis table in the wikishared DB (it will probably only exist if you tested the prior patch in this chain).
  2. Create whatever notifications you want to test.
  3. Checkout https://gerrit.wikimedia.org/r/264257
  4. vagrant provision

It will create and backfill the table. You can also run the script manually at any time if you want.

To connect to wikishared:

mysql -u extensionadmin wikishared -pextensionpassword
In T119523#1937102, @Mattflaschen wrote:

To connect to wikishared:

mysql -u extensionadmin wikishared -pextensionpassword

Can we add that to the welcome message when we access the VM with vagrant ssh?

Mattflaschen-WMF added a comment.EditedJan 15 2016, 8:53 PM

Sorry, when I posted the above, my brain was stuck on "test the users" mode, since part of the point was to have the PHP app use separated users.

However, the root user still exists (even if the app doesn't use it), so you can do:

mysql wikishared

I can still add that to MOTD if you think it's useful (there's a manifest, so it can easily be done only if Echo is installed).

Mattflaschen-WMF added a comment.EditedJan 15 2016, 10:52 PM

Nice catch on the Puppet ordering problem. Fixed in latest patch.

So, you can:

  1. DROP again.
  2. Create notifications.
  3. Checkout latest https://gerrit.wikimedia.org/r/#/c/264257
  4. vagrant provision

Rechecked locally actual backfill action with the steps given by @Mattflaschen

  1. wikishatred.echo_unread_wikis is created upon Notifcations are triggered
  2. e.g. euw_user=6 has Alerts form frwiki and Messages from wiki, It is be recorded in wikishatred.echo_unread_wikis as the following:

root@localhost:[wikishared]> select * from echo_unread_wikis;

euw_ideuw_usereuw_wikieuw_alertseuw_alerts_tseuw_messageseuw_messages_ts
21wiki220150716223405000000000000000
36frwiki120160116002634000000000000000
46wiki000000000000000120160116003155

After euw_user=6 viewed Alerts in frwiki, the table is updated.

euw_ideuw_usereuw_wikieuw_alertseuw_alerts_tseuw_messageseuw_messages_ts
21wiki220150716223405000000000000000
46wiki000000000000000120160116003155
jmatazzoni closed this task as Resolved.Feb 16 2016, 11:48 PM

Change 264257 merged by jenkins-bot:
Backfill echo_unread_wikis table after it is created

https://gerrit.wikimedia.org/r/264257