Page MenuHomePhabricator

Migrate OAuth data from mediawikiwiki to metawiki
Closed, ResolvedPublic

Description

As part of T59336: Make metawiki the central OAuth wiki, OAuth data (oauth_registered_consumer table, oauth_accepted_consumer table and OAuth records in the log table) needs to be moved to the Meta database. The data consist of about 250 consumers, 16000 acceptances (authorizations) and 500 OAuth log records.

Beta migration (done)

  • set $wgMWOAuthReadOnly = true for all beta wikis
  • run mwscript sql.php --wiki=metawiki php-master/extensions/OAuth/backend/schema/mysql/OAuth.sql on deployment-bastion
  • run mwscript sql.php --wiki=metawiki php-master/extensions/OAuth/backend/schema/mysql/callback_is_prefix.sql
  • run mwscript sql.php --wiki=metawiki php-master/extensions/OAuth/backend/schema/mysql/developer_agreement.sql
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=deploymentwiki --target=metawiki --table=oauth_registered_consumer
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=deploymentwiki --target=metawiki --table=oauth_accepted_consumer
  • set $wgMWOAuthCentralWiki = 'metawiki' for all wikis
  • set $wgMWOAuthReadOnly = false for all wikis
  • run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=metawiki --old=deploymentwiki

Test migration (done)

  • run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=aawiki --old=mediawikiwiki

(no need to test migration of OAuth tables - they don't exist on Meta now so it's low risk)

Real migration

  • set $wgMWOAuthReadOnly = true for all wikis gerrit 233608
  • run mwscript sql.php --wiki=metawiki php-1.26wmf20/extensions/OAuth/backend/schema/mysql/OAuth.sql on terbium
  • run mwscript sql.php --wiki=metawiki php-1.26wmf20/extensions/OAuth/backend/schema/mysql/callback_is_prefix.sql
  • run mwscript sql.php --wiki=metawiki php-1.26wmf20/extensions/OAuth/backend/schema/mysql/developer_agreement.sql
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=mediawikiwiki --target=metawiki --table=oauth_registered_consumer
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=mediawikiwiki --target=metawiki --table=oauth_accepted_consumer
  • set $wgMWOAuthCentralWiki = 'metawiki' for all wikis gerrit 233609
  • set $wgMWOAuthReadOnly = false for all wikis gerrit 233610
  • run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=metawiki --old=mediawikiwiki

Testing

  • authorize Hello World app before migration, verify it's still authorized afterwards
  • test deauthorizing/reauthorizing it
  • register a new consumer, test userinfo API via mwoauth

Event Timeline

Tgr claimed this task.
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr added subscribers: Tgr, bd808, csteipp.
Tgr set Security to None.
Tgr added subscribers: jcrespo, Springle.

@Springle/@jcrespo, any comments/concerns about this?

  • Creating an empty table and their indexes: OK to execute on master
  • Setting OAuth in read-only mode: OK because it avoids race conditions
  • Inserting entries 1 by one: OK because it avoids lag. If this/these was/were a very large table, monitor lag during the process and introduce some sleep every some rows if the multiple commits (fsyncs) cause any issue (it shouldn't unless it is a multi million row table).
  • Please provide details about the visibility of this table for labs (I suppose it shouldn't be on it at all)
  • If you want a sanity check, and the table is not too large, you can check that the copy was successful by mysql -BN -h host -e "SELECT * FROM old_table/new_table ORDER BY same_column" | md5sum. Select only identical columns (auto_increments may vary).

There is a wfWaitForSlaves() call after every 200 row insertions. The oauth_* tables are not visible on labs.

On the beta cluster $wgMWOAuthCentralWiki is currently deploymentwiki (due to T59403), I guess that should be migrated too. Or just discarded and recreated on Beta Meta.

It would be less confusing if the wiki was moved in beta too, but otherwise I don't think there's a reason to move it. Although it might help to run the whole process there as a final test before production.

Migration scheduled for next Monday (Aug 24) 14-16 PDT.

@csteipp am I correct in thinking that the only user-facing effect this is going to have is that the special pages related to listing, proposing and reviewing consumers will work on Meta instead of mw.org?

Also, do you think (some of the) OAuth documentation should be moved to Meta?

@csteipp am I correct in thinking that the only user-facing effect this is going to have is that the special pages related to listing, proposing and reviewing consumers will work on Meta instead of mw.org?

Post migration, yes, I think that is all. During the migration some end users will not be able to use OAuth, but I think the process has minimized the time that will take to a few minutes tops.

Also, do you think (some of the) OAuth documentation should be moved to Meta?

Anything WMF-specific or process related, yes, I think that should move to meta. Stuff about developing with OAuth, or the extension itself should stay on mw.ord, imho.

Tgr updated the task description. (Show Details)

Change 233606 had a related patch set uploaded (by Gergő Tisza):
Set OAuth readoonly on Beta for DB migration

https://gerrit.wikimedia.org/r/233606

Change 233607 had a related patch set uploaded (by Gergő Tisza):
Change OAuth central wiki on Beta to metawiki

https://gerrit.wikimedia.org/r/233607

Change 233608 had a related patch set uploaded (by Gergő Tisza):
Set OAuth readonly for DB migration

https://gerrit.wikimedia.org/r/233608

Change 233609 had a related patch set uploaded (by Gergő Tisza):
Change OAuth central wiki to metawiki

https://gerrit.wikimedia.org/r/233609

Change 233610 had a related patch set uploaded (by Gergő Tisza):
End OAuth migration; reenable writes

https://gerrit.wikimedia.org/r/233610

Tgr removed a project: Patch-For-Review.
Tgr removed a subscriber: gerritbot.

Change 233606 merged by jenkins-bot:
Set OAuth readoonly on Beta for DB migration

https://gerrit.wikimedia.org/r/233606

Change 233607 merged by jenkins-bot:
Change OAuth central wiki on Beta to metawiki

https://gerrit.wikimedia.org/r/233607

Test migration: run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=aawiki --old=mediawikiwiki

This failed with lots of the log user doesn't exist errors. I am going to assume that this is caused by aawiki being disabled well before the SUL finalization, and not a real error. Full log:

Beta migration and test migration done; postponing real one to another deploy window. I ran into a bunch of small problems and ran out of time.

Change 233608 merged by jenkins-bot:
Set OAuth readonly for DB migration

https://gerrit.wikimedia.org/r/233608

Migrating oauth_accepted_consumer gave a bunch of 'missing' results. I think this is normal behavior because the script walks through an id range by incrementing a number so revoked grants will result in this error. (If I am wrong about that - meh, these are just grants, losing some should be fine.)

Change 233609 merged by jenkins-bot:
Change OAuth central wiki to metawiki

https://gerrit.wikimedia.org/r/233609

Change 233610 merged by jenkins-bot:
End OAuth migration; reenable writes

https://gerrit.wikimedia.org/r/233610

Done and tested. There was one log migration failure. Thinking it through again, migration has to fail whenever the local user does not exist, so this could have ended badly, but then again it's hard to have a mediawiki.org account and not trigger crosswiki account creation (basically you must not use any other wiki and must never log in again).

Missed log id was 3445264, I'll see if I can move it by hand.

Ported the one missing log entry manually.