Page MenuHomePhabricator
Create Task
Maniphest T148111

DBQueryError when creating account on Beta, or logging into newly-created accounts
Closed, ResolvedPublic
Actions

Description

After creating an account on Beta (https://en.wikipedia.beta.wmflabs.org):

A database query error has occurred. This may indicate a bug in the software.[WABBNApEE4AAAEoHzP4AAAAG] 2016-10-14 02:21:42: Fatal exception of type "DBQueryError"

From fluorine:

2016-10-14 19:47:52 [WAE2ZwpEE4AAACQJnekAAAAQ] deployment-mediawiki04 enwiki 1.28.0-alpha exception ERROR: [WAE2ZwpEE4AAACQJnekAAAAQ] /wiki/Special:UserLogin   DBQueryError from line 1037 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: INSERT IGNORE INTO `localuser` (lu_wiki,lu_name,lu_attached_timestamp,lu_attached_method,lu_local_id,lu_global_id) VALUES ('enwiki','Flow test mover 2','20161014194752','password','12008','21997')
Function: CentralAuthUser::attach
Error: 1054 Unknown column 'lu_local_id' in 'field list' (10.68.23.30)
 {"exception_id":"WAE2ZwpEE4AAACQJnekAAAAQ"} 
[Exception DBQueryError] (/srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php:1037) A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: INSERT IGNORE INTO `localuser` (lu_wiki,lu_name,lu_attached_timestamp,lu_attached_method,lu_local_id,lu_global_id) VALUES ('enwiki','Flow test mover 2','20161014194752','password','12008','21997')
Function: CentralAuthUser::attach
Error: 1054 Unknown column 'lu_local_id' in 'field list' (10.68.23.30)

  #0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(895): Database->reportQueryError(string, integer, string, string, boolean)
  #1 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(1481): Database->query(string, string)
  #2 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(1803): Database->insert(string, array, string, string)
  #3 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(1280): CentralAuthUser->attach(string, string)
  #4 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthPrimaryAuthenticationProvider.php(204): CentralAuthUser->attemptPasswordMigration(string)
  #5 /srv/mediawiki/php-master/includes/auth/AuthManager.php(451): CentralAuthPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
  #6 /srv/mediawiki/php-master/includes/auth/AuthManager.php(381): MediaWiki\Auth\AuthManager->continueAuthentication(array)
  #7 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(355): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
  #8 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(484): AuthManagerSpecialPage->performAuthenticationStep(string, array)
  #9 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(656): AuthManagerSpecialPage->handleFormSubmit(array, VFormHTMLForm)
  #10 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(418): HTMLForm->trySubmit()
  #11 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(297): AuthManagerSpecialPage->trySubmit()
  #12 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(522): LoginSignupSpecialPage->execute(NULL)
  #13 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(583): SpecialPage->run(NULL)
  #14 /srv/mediawiki/php-master/includes/MediaWiki.php(283): SpecialPageFactory::executePath(Title, RequestContext)
  #15 /srv/mediawiki/php-master/includes/MediaWiki.php(861): MediaWiki->performRequest()
  #16 /srv/mediawiki/php-master/includes/MediaWiki.php(522): MediaWiki->main()
  #17 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
  #18 /srv/mediawiki/w/index.php(3): include(string)
  #19 {main}

Related Objects
Search...

Event Timeline

• Mattflaschen-WMF created this task.Oct 14 2016, 2:23 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 14 2016, 2:23 AM
• Mattflaschen-WMF added a subtask: T146723: deployment-fluorine02 does not have logs.Oct 14 2016, 2:27 AM
• Mattflaschen-WMF removed a subtask: T146723: deployment-fluorine02 does not have logs.Oct 14 2016, 2:32 AM
• Mattflaschen-WMF updated the task description. (Show Details)Oct 14 2016, 2:37 AM
• Mattflaschen-WMF added a subscriber: bd808.

@bd808, it doesn't seem like the actual DB error is available on logstash, just that there *was* a DB error. Am I missing something?

• Mattflaschen-WMF renamed this task from DBQueryError when creating account on Beta to DBQueryError when creating account on Beta, or logging into newly-created accounts.Oct 14 2016, 3:10 AM
• Mattflaschen-WMF added a comment.Oct 14 2016, 3:21 AM

The same happens when I try to log into newly created account on Beta (both created in the web UI, or created with createAndPromote.php). The one below is from:

mwscript createAndPromote.php --wiki=enwiki 'Flow test mover 2' 'M81gjn82lJSLI'

Username: Flow test mover 2
Password: M81gjn82lJSLI

However, I can still log into Mattflaschen.

• Mattflaschen-WMF added a subtask: T146723: deployment-fluorine02 does not have logs.Oct 14 2016, 3:22 AM
• Mattflaschen-WMF mentioned this in T147942: Move a Flow board is not possible.
Etonkovidova subscribed.Oct 14 2016, 4:08 PM

One more thing to add:
The error message on Special:CreateAccount page for attempting to create an account for an existing user will expose HTML markup:

The requested username is already taken in the <a href="http://meta.wikimedia.org/wiki/Help:Unified_login">unified login system</a>. Please choose a different name.

However, it happens only with usernames that displayed ' Fatal exception of type "DBQueryError" ' (with status code: 500) upon creation; old username are fine.

bd808 added a comment.Oct 14 2016, 4:25 PM
In T148111#2715267, @Mattflaschen-WMF wrote:

@bd808, it doesn't seem like the actual DB error is available on logstash, just that there *was* a DB error. Am I missing something?

I'm not seeing the query logged either: https://logstash-beta.wmflabs.org/goto/753266272fe5e69bae477eee06ed8cfd

phuedx subscribed.Oct 14 2016, 5:19 PM
• Mattflaschen-WMF added a comment.Oct 14 2016, 7:59 PM
In T148111#2717186, @bd808 wrote:
In T148111#2715267, @Mattflaschen-WMF wrote:

@bd808, it doesn't seem like the actual DB error is available on logstash, just that there *was* a DB error. Am I missing something?

I'm not seeing the query logged either: https://logstash-beta.wmflabs.org/goto/753266272fe5e69bae477eee06ed8cfd

Apparently it's in the message field, but truncated. I noticed that it is somewhere else more readable, https://logstash-beta.wmflabs.org/app/kibana#/doc/logstash-*/logstash-2016.10.14/mediawiki?id=AVfA_rxYYZVhfGLdiMRW&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15m,mode:quick,to:now)) . The latter has two relevant fields, error and sql1line, that together have everything.

It's also on fluorine now.

• Mattflaschen-WMF claimed this task.Oct 14 2016, 8:00 PM
• Mattflaschen-WMF updated the task description. (Show Details)
• Mattflaschen-WMF added a project: MediaWiki-extensions-CentralAuth.
• Mattflaschen-WMF added a project: Android-app-goals-2016-Oct-Dec-Q2.
Restricted Application added a project: Wikipedia-Android-App-Backlog. · View Herald TranscriptOct 14 2016, 8:00 PM
• Mattflaschen-WMF edited projects, added Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016); removed Wikipedia-Android-App-Backlog, Android-app-goals-2016-Oct-Dec-Q2.Oct 14 2016, 8:04 PM
bd808 added a subscriber: Niharika.Oct 14 2016, 8:05 PM

The lu_local_id stuff is new code CentralAuth from @Niharika that was merged yesterday. Looks like nobody has applied the database migration for it to the beta cluster. CA's migrations aren't done by update.php so it needs to be done manually.

• Mattflaschen-WMF added subscribers: Marostegui, kaldari.Oct 14 2016, 8:07 PM
In T148111#2718020, @bd808 wrote:

The lu_local_id stuff is new code CentralAuth from @Niharika that was merged yesterday. Looks like nobody has applied the database migration for it to the beta cluster. CA's migrations aren't done by update.php so it needs to be done manually.

Yeah, T141951: Add local_user_id and global_user_id fields to localuser table in centralauth database says it has been applied on Beta already. Looking into it now.

Stashbot subscribed.Oct 14 2016, 8:09 PM

Mentioned in SAL (#wikimedia-releng) [2016-10-14T20:09:13Z] <bd808> Applied CentralAuth's patch-lu_local_id.sql migration for T148111

• Mattflaschen-WMF merged a task: T144873: Make sure new CentralAuth database changes work on Beta Cluster.Oct 14 2016, 8:21 PM
• Mattflaschen-WMF added a parent task: T5525: Cross-wiki watchlists.
• Mattflaschen-WMF added a subscriber: • DannyH.
• Mattflaschen-WMF reassigned this task from • Mattflaschen-WMF to bd808.Oct 14 2016, 8:25 PM
• Mattflaschen-WMF mentioned this in T141951: Add local_user_id and global_user_id fields to localuser table in centralauth database.
• Mattflaschen-WMF removed a project: Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016).
Restricted Application added a project: User-bd808. · View Herald TranscriptOct 14 2016, 8:25 PM
Stashbot added a comment.Oct 14 2016, 8:37 PM

Mentioned in SAL (#wikimedia-releng) [2016-10-14T20:37:15Z] <matt_flaschen> Applied CentralAuth's patch-lu_local_id.sql migration for T148111, to sql --write

• Mattflaschen-WMF added a comment.Oct 14 2016, 8:48 PM
In T148111#2717050, @Etonkovidova wrote:

One more thing to add:
The error message on Special:CreateAccount page for attempting to create an account for an existing user will expose HTML markup:

Filed as T148247: Login errors show as raw HTML (on Beta)

• Mattflaschen-WMF mentioned this in T110115: Possible to run writes (e.g. UPDATE) on Beta Cluster replica.Oct 14 2016, 9:37 PM
kaldari added a comment.Oct 15 2016, 1:09 AM

I'm fairly sure I applied the database changes on the Beta Cluster, but maybe I did it wrong. @Mattflaschen-WMF, @bd808: Can you explain the steps that you took to apply the db patch on the Beta Cluster so I can be sure to do it correctly next time.

• Mattflaschen-WMF added a comment.EditedOct 15 2016, 1:15 AM

There were some problems with the way we did it this time (not the below).

But, a correct version of what we/I did this time:

  1. ssh deployment-tin.eqiad.wmflabs
  2. sql --write centralauth
  3. <Paste in SQL>

I'm pretty sure you can also pipe it to the same command, but I pasted it this time.

sql --write centralauth

ensures you are connecting to the master (the schema change will be replicated to the replicas automatically).

kaldari added a comment.Oct 15 2016, 8:18 AM

That's the same as what I did, except I didn't use "--write". Maybe it didn't get applied to the master somehow.

bd808 closed this task as Resolved.Oct 17 2016, 3:24 PM
bd808 reassigned this task from bd808 to • Mattflaschen-WMF.

New account creation on beta cluster wikis seems to be working again after @Mattflaschen-WMF's proper application of the patch to the master db and the subsequent fix of restarting the replication process that had died because of my patch to the wrong db.

bd808 moved this task from To Do to Archive on the User-bd808 board.Oct 17 2016, 3:25 PM
hashar closed subtask T146723: deployment-fluorine02 does not have logs as Resolved.Nov 21 2016, 12:53 PM
MarcoAurelio moved this task from Backlog to Done on the MediaWiki-extensions-CentralAuth board.Dec 15 2016, 4:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL