Page MenuHomePhabricator
Create Task
Maniphest T108648

Migrate OAuth data from mediawikiwiki to metawiki
Closed, ResolvedPublic
Actions

Description

As part of T59336: Make metawiki the central OAuth wiki, OAuth data (oauth_registered_consumer table, oauth_accepted_consumer table and OAuth records in the log table) needs to be moved to the Meta database. The data consist of about 250 consumers, 16000 acceptances (authorizations) and 500 OAuth log records.

Beta migration (done)

  • set $wgMWOAuthReadOnly = true for all beta wikis
  • run mwscript sql.php --wiki=metawiki php-master/extensions/OAuth/backend/schema/mysql/OAuth.sql on deployment-bastion
  • run mwscript sql.php --wiki=metawiki php-master/extensions/OAuth/backend/schema/mysql/callback_is_prefix.sql
  • run mwscript sql.php --wiki=metawiki php-master/extensions/OAuth/backend/schema/mysql/developer_agreement.sql
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=deploymentwiki --target=metawiki --table=oauth_registered_consumer
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=deploymentwiki --target=metawiki --table=oauth_accepted_consumer
  • set $wgMWOAuthCentralWiki = 'metawiki' for all wikis
  • set $wgMWOAuthReadOnly = false for all wikis
  • run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=metawiki --old=deploymentwiki

Test migration (done)

  • run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=aawiki --old=mediawikiwiki

(no need to test migration of OAuth tables - they don't exist on Meta now so it's low risk)

Real migration

  • set $wgMWOAuthReadOnly = true for all wikis gerrit 233608
  • run mwscript sql.php --wiki=metawiki php-1.26wmf20/extensions/OAuth/backend/schema/mysql/OAuth.sql on terbium
  • run mwscript sql.php --wiki=metawiki php-1.26wmf20/extensions/OAuth/backend/schema/mysql/callback_is_prefix.sql
  • run mwscript sql.php --wiki=metawiki php-1.26wmf20/extensions/OAuth/backend/schema/mysql/developer_agreement.sql
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=mediawikiwiki --target=metawiki --table=oauth_registered_consumer
  • run mwscript extensions/OAuth/maintenance/migrateCentralWiki.php --wiki=metawiki --old=mediawikiwiki --target=metawiki --table=oauth_accepted_consumer
  • set $wgMWOAuthCentralWiki = 'metawiki' for all wikis gerrit 233609
  • set $wgMWOAuthReadOnly = false for all wikis gerrit 233610
  • run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=metawiki --old=mediawikiwiki

Testing

  • authorize Hello World app before migration, verify it's still authorized afterwards
  • test deauthorizing/reauthorizing it
  • register a new consumer, test userinfo API via mwoauth

Details

SubjectRepoBranchLines +/-
End OAuth migration; reenable writesoperations/mediawiki-configmaster+0 -1
Change OAuth central wiki to metawikioperations/mediawiki-configmaster+1 -1
Set OAuth readonly for DB migrationoperations/mediawiki-configmaster+1 -0
Change OAuth central wiki on Beta to metawikioperations/mediawiki-configmaster+1 -2
Set OAuth readoonly on Beta for DB migrationoperations/mediawiki-configmaster+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Tgr created this task.Aug 10 2015, 11:40 PM
Tgr claimed this task.
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr added projects: MediaWiki-extensions-OAuth, DBA.
Tgr added subscribers: Tgr, bd808, • csteipp.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 10 2015, 11:40 PM
Tgr added a parent task: T59336: Make metawiki the central OAuth wiki.Aug 10 2015, 11:41 PM
Ricordisamoa subscribed.Aug 10 2015, 11:42 PM
Tgr updated the task description. (Show Details)Aug 10 2015, 11:44 PM
Tgr set Security to None.
Tgr added subscribers: jcrespo, • Springle.

@Springle/@jcrespo, any comments/concerns about this?

jcrespo added a comment.EditedAug 11 2015, 7:31 AM
  • Creating an empty table and their indexes: OK to execute on master
  • Setting OAuth in read-only mode: OK because it avoids race conditions
  • Inserting entries 1 by one: OK because it avoids lag. If this/these was/were a very large table, monitor lag during the process and introduce some sleep every some rows if the multiple commits (fsyncs) cause any issue (it shouldn't unless it is a multi million row table).
  • Please provide details about the visibility of this table for labs (I suppose it shouldn't be on it at all)
  • If you want a sanity check, and the table is not too large, you can check that the copy was successful by mysql -BN -h host -e "SELECT * FROM old_table/new_table ORDER BY same_column" | md5sum. Select only identical columns (auto_increments may vary).
Tgr added a comment.Aug 11 2015, 8:28 AM

There is a wfWaitForSlaves() call after every 200 row insertions. The oauth_* tables are not visible on labs.

Tgr added a comment.Aug 11 2015, 8:30 AM

On the beta cluster $wgMWOAuthCentralWiki is currently deploymentwiki (due to T59403), I guess that should be migrated too. Or just discarded and recreated on Beta Meta.

• csteipp added a comment.Aug 11 2015, 3:56 PM

It would be less confusing if the wiki was moved in beta too, but otherwise I don't think there's a reason to move it. Although it might help to run the whole process there as a final test before production.

Tgr added a comment.Aug 17 2015, 9:45 PM

Migration scheduled for next Monday (Aug 24) 14-16 PDT.

Tgr added a comment.Aug 17 2015, 10:10 PM

@csteipp am I correct in thinking that the only user-facing effect this is going to have is that the special pages related to listing, proposing and reviewing consumers will work on Meta instead of mw.org?

Tgr added a comment.Aug 17 2015, 10:12 PM

Also, do you think (some of the) OAuth documentation should be moved to Meta?

• csteipp added a comment.Aug 17 2015, 11:02 PM
In T108648#1547618, @Tgr wrote:

@csteipp am I correct in thinking that the only user-facing effect this is going to have is that the special pages related to listing, proposing and reviewing consumers will work on Meta instead of mw.org?

Post migration, yes, I think that is all. During the migration some end users will not be able to use OAuth, but I think the process has minimized the time that will take to a few minutes tops.

• csteipp added a comment.Aug 17 2015, 11:03 PM
In T108648#1547620, @Tgr wrote:

Also, do you think (some of the) OAuth documentation should be moved to Meta?

Anything WMF-specific or process related, yes, I think that should move to meta. Stuff about developing with OAuth, or the extension itself should stay on mw.ord, imho.

Tgr updated the task description. (Show Details)Aug 24 2015, 9:14 PM
Tgr updated the task description. (Show Details)
gerritbot subscribed.Aug 24 2015, 9:40 PM

Change 233606 had a related patch set uploaded (by Gergő Tisza):
Set OAuth readoonly on Beta for DB migration

https://gerrit.wikimedia.org/r/233606

gerritbot added a project: Patch-For-Review.Aug 24 2015, 9:40 PM

Change 233607 had a related patch set uploaded (by Gergő Tisza):
Change OAuth central wiki on Beta to metawiki

https://gerrit.wikimedia.org/r/233607

gerritbot added a comment.Aug 24 2015, 9:40 PM

Change 233608 had a related patch set uploaded (by Gergő Tisza):
Set OAuth readonly for DB migration

https://gerrit.wikimedia.org/r/233608

gerritbot added a comment.Aug 24 2015, 9:40 PM

Change 233609 had a related patch set uploaded (by Gergő Tisza):
Change OAuth central wiki to metawiki

https://gerrit.wikimedia.org/r/233609

gerritbot added a comment.Aug 24 2015, 9:40 PM

Change 233610 had a related patch set uploaded (by Gergő Tisza):
End OAuth migration; reenable writes

https://gerrit.wikimedia.org/r/233610

Tgr updated the task description. (Show Details)Aug 24 2015, 9:41 PM
Tgr removed a project: Patch-For-Review.
Tgr removed a subscriber: gerritbot.
gerritbot added a comment.Aug 24 2015, 9:52 PM

Change 233606 merged by jenkins-bot:
Set OAuth readoonly on Beta for DB migration

https://gerrit.wikimedia.org/r/233606

Tgr mentioned this in rOMWC9523b3b0e74d: Set OAuth readoonly on Beta for DB migration.Aug 24 2015, 9:52 PM
Tgr updated the task description. (Show Details)Aug 24 2015, 10:02 PM
Tgr updated the task description. (Show Details)Aug 24 2015, 10:14 PM
gerritbot added a comment.Aug 24 2015, 10:15 PM

Change 233607 merged by jenkins-bot:
Change OAuth central wiki on Beta to metawiki

https://gerrit.wikimedia.org/r/233607

Tgr mentioned this in rOMWC4d783e28a7e4: Change OAuth central wiki on Beta to metawiki.Aug 24 2015, 10:15 PM
Tgr updated the task description. (Show Details)Aug 24 2015, 10:17 PM
Tgr added a comment.EditedAug 24 2015, 10:22 PM

Test migration: run mwscript extensions/OAuth/maintenance/importCentralWikiLogs.php --wiki=aawiki --old=mediawikiwiki

This failed with lots of the log user doesn't exist errors. I am going to assume that this is caused by aawiki being disabled well before the SUL finalization, and not a real error. Full log:

oauth-aawiki-test-migration-log.txt38 KBDownload

Tgr updated the task description. (Show Details)Aug 24 2015, 10:39 PM

Beta migration and test migration done; postponing real one to another deploy window. I ran into a bunch of small problems and ran out of time.

Tgr updated the task description. (Show Details)Aug 24 2015, 10:46 PM
Tgr updated the task description. (Show Details)Aug 25 2015, 8:18 PM
gerritbot added a comment.Aug 25 2015, 8:22 PM

Change 233608 merged by jenkins-bot:
Set OAuth readonly for DB migration

https://gerrit.wikimedia.org/r/233608

Tgr mentioned this in rOMWC13832ca3beb6: Set OAuth readonly for DB migration.Aug 25 2015, 8:22 PM
Tgr added a comment.Aug 25 2015, 8:30 PM

Migrating oauth_accepted_consumer gave a bunch of 'missing' results. I think this is normal behavior because the script walks through an id range by incrementing a number so revoked grants will result in this error. (If I am wrong about that - meh, these are just grants, losing some should be fine.)

gerritbot added a comment.Aug 25 2015, 8:31 PM

Change 233609 merged by jenkins-bot:
Change OAuth central wiki to metawiki

https://gerrit.wikimedia.org/r/233609

Tgr mentioned this in rOMWC79ef9e065855: Change OAuth central wiki to metawiki.Aug 25 2015, 8:32 PM
gerritbot added a comment.Aug 25 2015, 8:33 PM

Change 233610 merged by jenkins-bot:
End OAuth migration; reenable writes

https://gerrit.wikimedia.org/r/233610

Tgr mentioned this in rOMWC4eb951711047: End OAuth migration; reenable writes.Aug 25 2015, 8:34 PM
Tgr closed this task as Resolved.Aug 25 2015, 8:59 PM

Done and tested. There was one log migration failure. Thinking it through again, migration has to fail whenever the local user does not exist, so this could have ended badly, but then again it's hard to have a mediawiki.org account and not trigger crosswiki account creation (basically you must not use any other wiki and must never log in again).

Missed log id was 3445264, I'll see if I can move it by hand.

Tgr added a comment.Aug 25 2015, 10:33 PM

Ported the one missing log entry manually.

Tgr mentioned this in T59336: Make metawiki the central OAuth wiki.Jan 6 2016, 9:50 PM
Framawiki mentioned this in T215725: Consider removing mediawikiwiki from wgCentralAuthAutoCreateWikis.Feb 10 2019, 2:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits