Global account manager (Special:CentralAuth) errors out on beta
Closed, ResolvedPublic

Description

Does not affect production; someone probably forgot to write an update script for a schema change.

https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Tgr

2017-02-08 23:18:19 [WJunOwpEE4AAAAeY6NEAAAAM] deployment-mediawiki04 enwiki 1.29.0-alpha exception ERROR: [WJunOwpEE4AAAAeY6NEAAAAM] /w/index.php?title=Special%3ACentralAuth&target=Tgr   DBQueryError from line 1057 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SELECT  ug_group  FROM `user_groups`    WHERE ug_user = '300' AND (ug_expiry IS NULL OR ug_expiry >= '20170208231819')  
Function: CentralAuthUser::localUserData
Error: 1054 Unknown column 'ug_expiry' in 'where clause' (10.68.18.35)
 {"exception_id":"WJunOwpEE4AAAAeY6NEAAAAM","caught_by":"mwe_handler"} 
[Exception DBQueryError] (/srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php:1057) A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SELECT  ug_group  FROM `user_groups`    WHERE ug_user = '300' AND (ug_expiry IS NULL OR ug_expiry >= '20170208231819')  
Function: CentralAuthUser::localUserData
Error: 1054 Unknown column 'ug_expiry' in 'where clause' (10.68.18.35)

  #0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(915): Database->reportQueryError(string, integer, string, string, boolean)
  #1 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(1257): Database->query(string, string)
  #2 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2426): Database->select(string, array, array, string)
  #3 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2264): CentralAuthUser->localUserData(string)
  #4 /srv/mediawiki/php-master/extensions/CentralAuth/includes/specials/SpecialCentralAuth.php(112): CentralAuthUser->queryAttached()
  #5 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(522): SpecialCentralAuth->execute(NULL)
  #6 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(576): SpecialPage->run(NULL)
  #7 /srv/mediawiki/php-master/includes/MediaWiki.php(285): SpecialPageFactory::executePath(Title, RequestContext)
  #8 /srv/mediawiki/php-master/includes/MediaWiki.php(860): MediaWiki->performRequest()
  #9 /srv/mediawiki/php-master/includes/MediaWiki.php(521): MediaWiki->main()
  #10 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
  #11 /srv/mediawiki/w/index.php(3): include(string)
  #12 {main}

(CentralAuth browser tests caught this. Yay!)

Tgr created this task.Feb 8 2017, 11:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 8 2017, 11:31 PM
Tgr edited the task description. (Show Details)Feb 8 2017, 11:40 PM
Legoktm assigned this task to TTO.Feb 9 2017, 1:35 AM
TTO added a comment.Feb 9 2017, 1:44 AM

Schema changes are supposed to be automatic on the beta cluster, and I know for a fact that ug_expiry exists on beta's enwiki at least. Unfortunately the error message doesn't tell us which DB is missing the ug_expiry column.

TTO added a comment.Feb 9 2017, 1:51 AM

https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore_beta_44 works, but https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore does not. Could someone with DB access on the beta cluster see which wikis Addshore exists on that Addshore_beta_44 does not?

Tgr edited the task description. (Show Details)Feb 9 2017, 3:06 AM
Tgr added a comment.Feb 9 2017, 3:24 AM

Weird.

tgr@deployment-tin:~$ foreachwiki sql.php --query "select count(*) from information_schema.COLUMNS where TABLE_SCHEMA = database() and TABLE_NAME = 'user_groups' and COLUMN_NAME = 'ug_expiry';" | grep count 2>/dev/null
aawiki:      [count(*)] => 1
arwiki:      [count(*)] => 1
cawiki:      [count(*)] => 1
commonswiki:      [count(*)] => 1
deploymentwiki:      [count(*)] => 1
dewiki:      [count(*)] => 1
dewiktionary:      [count(*)] => 1
en_rtlwiki:      [count(*)] => 1
enwiki:      [count(*)] => 1
enwikibooks:      [count(*)] => 1
enwikinews:      [count(*)] => 1
enwikiquote:      [count(*)] => 1
enwikisource:      [count(*)] => 1
enwikiversity:      [count(*)] => 1
enwikivoyage:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
enwiktionary:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
eowiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
eswiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
fawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
hewiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
hiwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
jawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
kowiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
loginwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
metawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
nlwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
ruwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
simplewiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
sqwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
testwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
ukwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
wikidatawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
zerowiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
zhwiki:      [count(*)] => 1

(Filed T157656 about the notice spam.)

Tgr added a comment.Feb 9 2017, 3:42 AM

https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore_beta_44 works, but https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore does not. Could someone with DB access on the beta cluster see which wikis Addshore exists on that Addshore_beta_44 does not?

tgr@deployment-tin:~$ foreachwiki sql.php --query "select u1.user_name is not null and u2.user_name is null as foo from (select 1)i left join user u1 on u1.user_name = 'Addshore' left join user u2 on u2.user_name = 'Addshore_beta_44';" 2>/dev/null | grep foo
aawiki:      [foo] => 0
arwiki:      [foo] => 0
cawiki:      [foo] => 0
commonswiki:      [foo] => 1
deploymentwiki:      [foo] => 1
dewiki:      [foo] => 0
dewiktionary:      [foo] => 0
en_rtlwiki:      [foo] => 0
enwiki:      [foo] => 1
enwikibooks:      [foo] => 1
enwikinews:      [foo] => 1
enwikiquote:      [foo] => 1
enwikisource:      [foo] => 1
enwikiversity:      [foo] => 1
enwikivoyage:      [foo] => 0
enwiktionary:      [foo] => 1
eowiki:      [foo] => 0
eswiki:      [foo] => 0
fawiki:      [foo] => 0
hewiki:      [foo] => 0
hiwiki:      [foo] => 0
jawiki:      [foo] => 0
kowiki:      [foo] => 0
loginwiki:      [foo] => 1
metawiki:      [foo] => 1
nlwiki:      [foo] => 0
ruwiki:      [foo] => 0
simplewiki:      [foo] => 0
sqwiki:      [foo] => 0
testwiki:      [foo] => 1
ukwiki:      [foo] => 0
wikidatawiki:      [foo] => 1
zerowiki:      [foo] => 0
zhwiki:      [foo] => 0
Tgr added a comment.Feb 9 2017, 4:03 AM

Uh, don't mind me. The database is right there in the extra log data: ee_prototypewiki. Which is apparently not in ąll-labs.db but accessed nevertheless. And indeed does not have ug_expiry.

So it seems the updater and Special:CentralAuth use different methods to iterate the wikis and those can get out of sync. (I'm pretty sure we had a bug about this for Vagrant; but then the multiwiki logic for Vagrant is a lot more fragile.)

Legoktm added a subscriber: Legoktm.

Special:CentralAuth will query any db table in the localuser/globaluser tables, so I think ee_prototypewiki probably got shut down, but since it's still linked to accounts, CA will try and query it. Given that https://meta.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix doesn't list ee_prototypewiki, we should probably delete those rows out of CA.

Tgr added a comment.Feb 9 2017, 6:28 AM

Oh, I remember now, I wrote https://gerrit.wikimedia.org/r/#/c/297519/ to deal with this issue in Vagrant. I'll just go run it.

Tgr edited the task description. (Show Details)Feb 9 2017, 6:29 AM

Although maybe Special:CA should just do a sanity check like the one added in https://gerrit.wikimedia.org/r/#/c/297519/

zeljkofilipin moved this task from Backlog to Watching o.O on the User-zeljkofilipin board.
hashar added a subscriber: hashar.Feb 9 2017, 4:07 PM

We had tasks in the paste with user no more being able to login because their central auth account still referred to a non existent wiki. So probably want to clean up the CentralAuth database?

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptFeb 9 2017, 4:27 PM
greg added a subscriber: greg.Feb 9 2017, 6:21 PM

Oh, I remember now, I wrote https://gerrit.wikimedia.org/r/#/c/297519/ to deal with this issue in Vagrant. I'll just go run it.

Although maybe Special:CA should just do a sanity check like the one added in https://gerrit.wikimedia.org/r/#/c/297519/

Status?

Change 336861 had a related patch set uploaded (by Gergő Tisza):
Handle non-existing wikis more gracefully

https://gerrit.wikimedia.org/r/336861

Tgr added a comment.Feb 9 2017, 6:52 PM

The patch should fix the errors (and log warnings instead). The script to clean up the database is checkLocalUser.php --allwikis --delete --delete-nowiki (but I want to test the patch first).

This is now blocking next week's train, most of our selenium tests are broken.

Tgr added a subscriber: Reedy.Feb 9 2017, 9:40 PM

I think @Reedy deleted the offending rows?

Reedy added a comment.Feb 9 2017, 9:42 PM

I think @Reedy deleted the offending rows?

I did

greg added a comment.Feb 9 2017, 9:44 PM

Soooo, status? :)

Reedy added a comment.Feb 9 2017, 9:50 PM

This is now blocking next week's train, most of our selenium tests are broken.

About an hour ago, which was about 2 hours after I did the DB deletes...

[20:42:11] <wmf-insecte> Yippee, build fixed!
[20:42:11] <wmf-insecte> Yippee, build fixed!
[20:42:12] <wmf-insecte> Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #299: FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/299/
[20:42:12] <wmf-insecte> Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #299: FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/299/
greg added a comment.Feb 9 2017, 9:54 PM

Re-running some others now...

greg closed this task as "Resolved".Feb 9 2017, 9:58 PM

w00t

21:55 < wmf-insec> Yippee, build fixed!
21:55 < wmf-insec> Project selenium-CentralAuth » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #299: FIXED in 1 min 50 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/299/

Mentioned in SAL (#wikimedia-releng) [2017-02-09T22:22:30Z] <greg-g> manually kicked off a bunch of selenium tests after tgr and Reedy fixed T157636

Change 336861 merged by jenkins-bot:
Handle non-existing wikis more gracefully

https://gerrit.wikimedia.org/r/336861