Page MenuHomePhabricator
Create Task
Maniphest T157636

Global account manager (Special:CentralAuth) errors out on beta
Closed, ResolvedPublic
Actions

Description

Does not affect production; someone probably forgot to write an update script for a schema change.

https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Tgr

2017-02-08 23:18:19 [WJunOwpEE4AAAAeY6NEAAAAM] deployment-mediawiki04 enwiki 1.29.0-alpha exception ERROR: [WJunOwpEE4AAAAeY6NEAAAAM] /w/index.php?title=Special%3ACentralAuth&target=Tgr   DBQueryError from line 1057 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SELECT  ug_group  FROM `user_groups`    WHERE ug_user = '300' AND (ug_expiry IS NULL OR ug_expiry >= '20170208231819')  
Function: CentralAuthUser::localUserData
Error: 1054 Unknown column 'ug_expiry' in 'where clause' (10.68.18.35)
 {"exception_id":"WJunOwpEE4AAAAeY6NEAAAAM","caught_by":"mwe_handler"} 
[Exception DBQueryError] (/srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php:1057) A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SELECT  ug_group  FROM `user_groups`    WHERE ug_user = '300' AND (ug_expiry IS NULL OR ug_expiry >= '20170208231819')  
Function: CentralAuthUser::localUserData
Error: 1054 Unknown column 'ug_expiry' in 'where clause' (10.68.18.35)

  #0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(915): Database->reportQueryError(string, integer, string, string, boolean)
  #1 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(1257): Database->query(string, string)
  #2 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2426): Database->select(string, array, array, string)
  #3 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUser.php(2264): CentralAuthUser->localUserData(string)
  #4 /srv/mediawiki/php-master/extensions/CentralAuth/includes/specials/SpecialCentralAuth.php(112): CentralAuthUser->queryAttached()
  #5 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(522): SpecialCentralAuth->execute(NULL)
  #6 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(576): SpecialPage->run(NULL)
  #7 /srv/mediawiki/php-master/includes/MediaWiki.php(285): SpecialPageFactory::executePath(Title, RequestContext)
  #8 /srv/mediawiki/php-master/includes/MediaWiki.php(860): MediaWiki->performRequest()
  #9 /srv/mediawiki/php-master/includes/MediaWiki.php(521): MediaWiki->main()
  #10 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
  #11 /srv/mediawiki/w/index.php(3): include(string)
  #12 {main}

(CentralAuth browser tests caught this. Yay!)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Handle non-existing wikis more gracefullymediawiki/extensions/CentralAuthmaster+16 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Tgr created this task.Feb 8 2017, 11:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 8 2017, 11:31 PM
Tgr updated the task description. (Show Details)Feb 8 2017, 11:40 PM
Tgr added a comment.Feb 8 2017, 11:45 PM

See also T157641: Database error messages are not useful in logstash

Legoktm assigned this task to TTO.Feb 9 2017, 1:35 AM
TTO added a comment.Feb 9 2017, 1:44 AM

Schema changes are supposed to be automatic on the beta cluster, and I know for a fact that ug_expiry exists on beta's enwiki at least. Unfortunately the error message doesn't tell us which DB is missing the ug_expiry column.

TTO added a comment.Feb 9 2017, 1:51 AM

https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore_beta_44 works, but https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore does not. Could someone with DB access on the beta cluster see which wikis Addshore exists on that Addshore_beta_44 does not?

Tgr updated the task description. (Show Details)Feb 9 2017, 3:06 AM
Tgr added a comment.Feb 9 2017, 3:24 AM

Weird.

tgr@deployment-tin:~$ foreachwiki sql.php --query "select count(*) from information_schema.COLUMNS where TABLE_SCHEMA = database() and TABLE_NAME = 'user_groups' and COLUMN_NAME = 'ug_expiry';" | grep count 2>/dev/null
aawiki:      [count(*)] => 1
arwiki:      [count(*)] => 1
cawiki:      [count(*)] => 1
commonswiki:      [count(*)] => 1
deploymentwiki:      [count(*)] => 1
dewiki:      [count(*)] => 1
dewiktionary:      [count(*)] => 1
en_rtlwiki:      [count(*)] => 1
enwiki:      [count(*)] => 1
enwikibooks:      [count(*)] => 1
enwikinews:      [count(*)] => 1
enwikiquote:      [count(*)] => 1
enwikisource:      [count(*)] => 1
enwikiversity:      [count(*)] => 1
enwikivoyage:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
enwiktionary:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
eowiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
eswiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
fawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
hewiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
hiwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
jawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
kowiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
loginwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
metawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
nlwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
ruwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
simplewiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
sqwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
testwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
ukwiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
wikidatawiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
zerowiki:      [count(*)] => 1
PHP Notice:  Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki/php-master/includes/SiteConfiguration.php on line 312
zhwiki:      [count(*)] => 1

(Filed T157656 about the notice spam.)

Liuxinyu970226 subscribed.Feb 9 2017, 3:33 AM
Tgr added a comment.Feb 9 2017, 3:42 AM
In T157636#3011885, @TTO wrote:

https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore_beta_44 works, but https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAuth/Addshore does not. Could someone with DB access on the beta cluster see which wikis Addshore exists on that Addshore_beta_44 does not?

tgr@deployment-tin:~$ foreachwiki sql.php --query "select u1.user_name is not null and u2.user_name is null as foo from (select 1)i left join user u1 on u1.user_name = 'Addshore' left join user u2 on u2.user_name = 'Addshore_beta_44';" 2>/dev/null | grep foo
aawiki:      [foo] => 0
arwiki:      [foo] => 0
cawiki:      [foo] => 0
commonswiki:      [foo] => 1
deploymentwiki:      [foo] => 1
dewiki:      [foo] => 0
dewiktionary:      [foo] => 0
en_rtlwiki:      [foo] => 0
enwiki:      [foo] => 1
enwikibooks:      [foo] => 1
enwikinews:      [foo] => 1
enwikiquote:      [foo] => 1
enwikisource:      [foo] => 1
enwikiversity:      [foo] => 1
enwikivoyage:      [foo] => 0
enwiktionary:      [foo] => 1
eowiki:      [foo] => 0
eswiki:      [foo] => 0
fawiki:      [foo] => 0
hewiki:      [foo] => 0
hiwiki:      [foo] => 0
jawiki:      [foo] => 0
kowiki:      [foo] => 0
loginwiki:      [foo] => 1
metawiki:      [foo] => 1
nlwiki:      [foo] => 0
ruwiki:      [foo] => 0
simplewiki:      [foo] => 0
sqwiki:      [foo] => 0
testwiki:      [foo] => 1
ukwiki:      [foo] => 0
wikidatawiki:      [foo] => 1
zerowiki:      [foo] => 0
zhwiki:      [foo] => 0
Tgr added a comment.Feb 9 2017, 4:03 AM

Uh, don't mind me. The database is right there in the extra log data: ee_prototypewiki. Which is apparently not in ąll-labs.db but accessed nevertheless. And indeed does not have ug_expiry.

So it seems the updater and Special:CentralAuth use different methods to iterate the wikis and those can get out of sync. (I'm pretty sure we had a bug about this for Vagrant; but then the multiwiki logic for Vagrant is a lot more fragile.)

Legoktm added a project: Beta-Cluster-Infrastructure.Feb 9 2017, 4:16 AM
Legoktm subscribed.

Special:CentralAuth will query any db table in the localuser/globaluser tables, so I think ee_prototypewiki probably got shut down, but since it's still linked to accounts, CA will try and query it. Given that https://meta.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix doesn't list ee_prototypewiki, we should probably delete those rows out of CA.

Tgr mentioned this in T157656: "Undefined index: wmgExtraLanguageNames" on beta.Feb 9 2017, 6:17 AM
Tgr added a comment.Feb 9 2017, 6:28 AM

Oh, I remember now, I wrote https://gerrit.wikimedia.org/r/#/c/297519/ to deal with this issue in Vagrant. I'll just go run it.

Tgr updated the task description. (Show Details)Feb 9 2017, 6:29 AM

Although maybe Special:CA should just do a sanity check like the one added in https://gerrit.wikimedia.org/r/#/c/297519/

zeljkofilipin triaged this task as High priority.EditedFeb 9 2017, 4:05 PM
zeljkofilipin added projects: User-zeljkofilipin, Browser-Tests-Infrastructure.
zeljkofilipin subscribed.

This broke almost all browser tests, not just CentralAuth, see T157665: Selenium Jenkins jobs fail with: Database query error. (internal_api_error_DBQueryError) (MediawikiApi::ApiError). :(

zeljkofilipin moved this task from Inbox to Waiting on the Browser-Tests-Infrastructure board.Feb 9 2017, 4:06 PM
zeljkofilipin moved this task from Backlog 🪒 to Watching 📺 on the User-zeljkofilipin board.
hashar subscribed.Feb 9 2017, 4:07 PM

We had tasks in the paste with user no more being able to login because their central auth account still referred to a non existent wiki. So probably want to clean up the CentralAuth database?

zeljkofilipin raised the priority of this task from High to Unbreak Now!.Feb 9 2017, 4:26 PM
zeljkofilipin added a subtask: T157665: Selenium Jenkins jobs fail with: Database query error. (internal_api_error_DBQueryError) (MediawikiApi::ApiError).
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptFeb 9 2017, 4:27 PM
greg subscribed.Feb 9 2017, 6:21 PM
In T157636#3012192, @Tgr wrote:

Oh, I remember now, I wrote https://gerrit.wikimedia.org/r/#/c/297519/ to deal with this issue in Vagrant. I'll just go run it.

In T157636#3012193, @Tgr wrote:

Although maybe Special:CA should just do a sanity check like the one added in https://gerrit.wikimedia.org/r/#/c/297519/

Status?

gerritbot subscribed.Feb 9 2017, 6:48 PM

Change 336861 had a related patch set uploaded (by Gergő Tisza):
Handle non-existing wikis more gracefully

https://gerrit.wikimedia.org/r/336861

gerritbot added a project: Patch-For-Review.Feb 9 2017, 6:48 PM
Tgr added a comment.Feb 9 2017, 6:52 PM

The patch should fix the errors (and log warnings instead). The script to clean up the database is checkLocalUser.php --allwikis --delete --delete-nowiki (but I want to test the patch first).

greg added a parent task: T155527: MW-1.29.0-wmf.12 deployment blockers.Feb 9 2017, 9:33 PM

This is now blocking next week's train, most of our selenium tests are broken.

Tgr added a subscriber: Reedy.Feb 9 2017, 9:40 PM

I think @Reedy deleted the offending rows?

Reedy added a comment.Feb 9 2017, 9:42 PM
In T157636#3014890, @Tgr wrote:

I think @Reedy deleted the offending rows?

I did

greg added a comment.Feb 9 2017, 9:44 PM

Soooo, status? :)

Reedy added a comment.Feb 9 2017, 9:50 PM
In T157636#3014847, @greg wrote:

This is now blocking next week's train, most of our selenium tests are broken.

About an hour ago, which was about 2 hours after I did the DB deletes...

[20:42:11] <wmf-insecte> Yippee, build fixed!
[20:42:11] <wmf-insecte> Yippee, build fixed!
[20:42:12] <wmf-insecte> Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #299: FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/299/
[20:42:12] <wmf-insecte> Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #299: FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/299/
greg added a comment.Feb 9 2017, 9:54 PM

Re-running some others now...

greg closed this task as Resolved.Feb 9 2017, 9:58 PM

w00t

21:55 < wmf-insec> Yippee, build fixed!
21:55 < wmf-insec> Project selenium-CentralAuth » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #299: FIXED in 1 min 50 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/299/
Stashbot subscribed.Feb 9 2017, 10:22 PM

Mentioned in SAL (#wikimedia-releng) [2017-02-09T22:22:30Z] <greg-g> manually kicked off a bunch of selenium tests after tgr and Reedy fixed T157636

Reedy merged a task: T157665: Selenium Jenkins jobs fail with: Database query error. (internal_api_error_DBQueryError) (MediawikiApi::ApiError).Feb 9 2017, 11:17 PM
Reedy added subscribers: Jdlrobson, dduvall.
Liuxinyu970226 unsubscribed.Feb 10 2017, 6:56 AM
jcrespo mentioned this in T126832: Recreate a wiki for Wikimedia Portugal.Feb 10 2017, 11:26 AM
Jdlrobson mentioned this in T157165: Deploying related pages broke our browser tests.Feb 10 2017, 6:15 PM
greg mentioned this in T157650: Can't log in to Beta Cluster wikis.Feb 10 2017, 10:48 PM
Aklapper merged a task: T157650: Can't log in to Beta Cluster wikis.Feb 12 2017, 5:19 AM
Aklapper added a subscriber: kaldari.
gerritbot added a comment.Feb 13 2017, 5:57 PM

Change 336861 merged by jenkins-bot:
Handle non-existing wikis more gracefully

https://gerrit.wikimedia.org/r/336861

ReleaseTaggerBot added a project: MW-1.29-release (WMF-deploy-2017-02-14_(1.29.0-wmf.12)).Feb 13 2017, 6:00 PM
MarcoAurelio moved this task from Backlog to Done on the MediaWiki-extensions-CentralAuth board.Nov 22 2017, 9:46 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits