Page MenuHomePhabricator

DBQueryError when creating account on Beta, or logging into newly-created accounts
Closed, ResolvedPublic

Description

After creating an account on Beta (https://en.wikipedia.beta.wmflabs.org):

A database query error has occurred. This may indicate a bug in the software.[WABBNApEE4AAAEoHzP4AAAAG] 2016-10-14 02:21:42: Fatal exception of type "DBQueryError"

From fluorine:

2016-10-14 19:47:52 [WAE2ZwpEE4AAACQJnekAAAAQ] deployment-mediawiki04 enwiki 1.28.0-alpha exception ERROR: [WAE2ZwpEE4AAACQJnekAAAAQ] /wiki/Special:UserLogin   DBQueryError from line 1037 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: INSERT IGNORE INTO `localuser` (lu_wiki,lu_name,lu_attached_timestamp,lu_attached_method,lu_local_id,lu_global_id) VALUES ('enwiki','Flow test mover 2','20161014194752','password','12008','21997')
Function: CentralAuthUser::attach
Error: 1054 Unknown column 'lu_local_id' in 'field list' (10.68.23.30)
 {"exception_id":"WAE2ZwpEE4AAACQJnekAAAAQ"} 
[Exception DBQueryError] (/srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php:1037) A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: INSERT IGNORE INTO `localuser` (lu_wiki,lu_name,lu_attached_timestamp,lu_attached_method,lu_local_id,lu_global_id) VALUES ('enwiki','Flow test mover 2','20161014194752','password','12008','21997')
Function: CentralAuthUser::attach
Error: 1054 Unknown column 'lu_local_id' in 'field list' (10.68.23.30)

  #0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(895): Database->reportQueryError(string, integer, string, string, boolean)
  #1 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(1481): Database->query(string, string)
  #2 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(1803): Database->insert(string, array, string, string)
  #3 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(1280): CentralAuthUser->attach(string, string)
  #4 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthPrimaryAuthenticationProvider.php(204): CentralAuthUser->attemptPasswordMigration(string)
  #5 /srv/mediawiki/php-master/includes/auth/AuthManager.php(451): CentralAuthPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
  #6 /srv/mediawiki/php-master/includes/auth/AuthManager.php(381): MediaWiki\Auth\AuthManager->continueAuthentication(array)
  #7 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(355): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
  #8 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(484): AuthManagerSpecialPage->performAuthenticationStep(string, array)
  #9 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(656): AuthManagerSpecialPage->handleFormSubmit(array, VFormHTMLForm)
  #10 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(418): HTMLForm->trySubmit()
  #11 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(297): AuthManagerSpecialPage->trySubmit()
  #12 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(522): LoginSignupSpecialPage->execute(NULL)
  #13 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(583): SpecialPage->run(NULL)
  #14 /srv/mediawiki/php-master/includes/MediaWiki.php(283): SpecialPageFactory::executePath(Title, RequestContext)
  #15 /srv/mediawiki/php-master/includes/MediaWiki.php(861): MediaWiki->performRequest()
  #16 /srv/mediawiki/php-master/includes/MediaWiki.php(522): MediaWiki->main()
  #17 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
  #18 /srv/mediawiki/w/index.php(3): include(string)
  #19 {main}

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 14 2016, 2:23 AM
Mattflaschen-WMF added a subscriber: bd808.

@bd808, it doesn't seem like the actual DB error is available on logstash, just that there *was* a DB error. Am I missing something?

Mattflaschen-WMF renamed this task from DBQueryError when creating account on Beta to DBQueryError when creating account on Beta, or logging into newly-created accounts.Oct 14 2016, 3:10 AM

The same happens when I try to log into newly created account on Beta (both created in the web UI, or created with createAndPromote.php). The one below is from:

mwscript createAndPromote.php --wiki=enwiki 'Flow test mover 2' 'M81gjn82lJSLI'

Username: Flow test mover 2
Password: M81gjn82lJSLI

However, I can still log into Mattflaschen.

One more thing to add:
The error message on Special:CreateAccount page for attempting to create an account for an existing user will expose HTML markup:

The requested username is already taken in the <a href="http://meta.wikimedia.org/wiki/Help:Unified_login">unified login system</a>. Please choose a different name.

However, it happens only with usernames that displayed ' Fatal exception of type "DBQueryError" ' (with status code: 500) upon creation; old username are fine.

bd808 added a comment.Oct 14 2016, 4:25 PM

@bd808, it doesn't seem like the actual DB error is available on logstash, just that there *was* a DB error. Am I missing something?

I'm not seeing the query logged either: https://logstash-beta.wmflabs.org/goto/753266272fe5e69bae477eee06ed8cfd

phuedx added a subscriber: phuedx.Oct 14 2016, 5:19 PM

@bd808, it doesn't seem like the actual DB error is available on logstash, just that there *was* a DB error. Am I missing something?

I'm not seeing the query logged either: https://logstash-beta.wmflabs.org/goto/753266272fe5e69bae477eee06ed8cfd

Apparently it's in the message field, but truncated. I noticed that it is somewhere else more readable, https://logstash-beta.wmflabs.org/app/kibana#/doc/logstash-*/logstash-2016.10.14/mediawiki?id=AVfA_rxYYZVhfGLdiMRW&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15m,mode:quick,to:now)) . The latter has two relevant fields, error and sql1line, that together have everything.

It's also on fluorine now.

bd808 added a subscriber: Niharika.Oct 14 2016, 8:05 PM

The lu_local_id stuff is new code CentralAuth from @Niharika that was merged yesterday. Looks like nobody has applied the database migration for it to the beta cluster. CA's migrations aren't done by update.php so it needs to be done manually.

The lu_local_id stuff is new code CentralAuth from @Niharika that was merged yesterday. Looks like nobody has applied the database migration for it to the beta cluster. CA's migrations aren't done by update.php so it needs to be done manually.

Yeah, T141951: Add local_user_id and global_user_id fields to localuser table in centralauth database says it has been applied on Beta already. Looking into it now.

Mentioned in SAL (#wikimedia-releng) [2016-10-14T20:09:13Z] <bd808> Applied CentralAuth's patch-lu_local_id.sql migration for T148111

Restricted Application added a project: User-bd808. · View Herald TranscriptOct 14 2016, 8:25 PM

Mentioned in SAL (#wikimedia-releng) [2016-10-14T20:37:15Z] <matt_flaschen> Applied CentralAuth's patch-lu_local_id.sql migration for T148111, to sql --write

One more thing to add:
The error message on Special:CreateAccount page for attempting to create an account for an existing user will expose HTML markup:

Filed as T148247: Login errors show as raw HTML (on Beta)

I'm fairly sure I applied the database changes on the Beta Cluster, but maybe I did it wrong. @Mattflaschen-WMF, @bd808: Can you explain the steps that you took to apply the db patch on the Beta Cluster so I can be sure to do it correctly next time.

Mattflaschen-WMF added a comment.EditedOct 15 2016, 1:15 AM

There were some problems with the way we did it this time (not the below).

But, a correct version of what we/I did this time:

  1. ssh deployment-tin.eqiad.wmflabs
  2. sql --write centralauth
  3. <Paste in SQL>

I'm pretty sure you can also pipe it to the same command, but I pasted it this time.

sql --write centralauth

ensures you are connecting to the master (the schema change will be replicated to the replicas automatically).

That's the same as what I did, except I didn't use "--write". Maybe it didn't get applied to the master somehow.

bd808 closed this task as Resolved.Oct 17 2016, 3:24 PM
bd808 reassigned this task from bd808 to Mattflaschen-WMF.

New account creation on beta cluster wikis seems to be working again after @Mattflaschen-WMF's proper application of the patch to the master db and the subsequent fix of restarting the replication process that had died because of my patch to the wrong db.

bd808 moved this task from To Do to Archive on the User-bd808 board.Oct 17 2016, 3:25 PM