Page MenuHomePhabricator

novaadmin removed from many keystone projects
Closed, ResolvedPublic

Description

novaadmin should be in every project. Yesterday it stopped being in lots of projects:

Roles for novaadmin are not set in these projects: set([u'phabricator', u'packaging', u'analytics', u'puppet', u'netdata', u'openstack', u'newsletter', u'maps-team', u'account-creation-assistance', u'servermon', u'testlabs', u'wikidata-dev', u'ores-staging', u'contributors', u'maps', u'huggle', u'tools', u'lizenzhinweisgenerator', u'integration', u'twl', u'wmt', u'kubernetes-testing', u'bastion', u'osmit', u'etcd', u'queryrapi', u'otrs', u'wikidata-federation', u'math', u'project-proxy', u'catgraph', u'wikidataconcepts', u'shinken', u'openocr', u'puppet3-diffs', u'search', u'scrumbugz', u'mwfileimport', u'services', u'wikidata-query', u'toolsbeta', u'wikidata-build', u'language', u'wikifactmine', u'deployment-prep', u'puppet-ca-replacement', u'gerrit', u'dumps', u'librarybase', u'testproject', u'bots', u'etherpad', u'ores'])

Easy to fix, but... what happened?

Details

Related Gerrit Patches:
mediawiki/extensions/LdapAuthentication : masterwhen user lookups fail, NULL out $this->userInfo

Event Timeline

Andrew created this task.Jul 21 2017, 3:01 PM
Restricted Application added subscribers: PokestarFan, Sadads, Aklapper. · View Herald TranscriptJul 21 2017, 3:01 PM

We've replaced novaadmin in deployment-prep; now it's missing from the following:

account-creation-assistance
analytics
bastion
bots
catgraph
contributors
dumps
etcd
etherpad
gerrit
huggle
integration
kubernetes-testing
language
librarybase
lizenzhinweisgenerator
maps
maps-team
math
mwfileimport
netdata
newsletter
openocr
openstack
ores
ores-staging
osmit
otrs
packaging
phabricator
project-proxy
puppet
puppet-ca-replacement
puppet3-diffs
queryrapi
scrumbugz
search
servermon
services
shinken
testlabs
testproject
tools
toolsbeta
twl
wikidata-build
wikidata-dev
wikidata-federation
wikidata-query
wikidataconcepts
wikifactmine
wmt

Andrew added a comment.EditedJul 21 2017, 3:23 PM

In total it was removed from 53 projects. I'm now checking to see if any one user is in all of those projects (other than novaadmin)

(update) Save novaadmin there's no one user account that has projectadmin in 53 projects, no even mine.

Andrew added a comment.EditedJul 21 2017, 3:35 PM

The first sign of trouble in the keystone log is:

(keystone.token.controllers): 2017-07-20 19:20:04,019 WARNING User novaadmin is unauthorized for tenant tools

Before that, keystone was restarted at 2017-07-20 18:14:20. That restart was probably due to the merge of https://gerrit.wikimedia.org/r/#/c/366025/ which happened at 18:12.

So currently I think this was caused by a misfire in OpenStackManager's removeUserFromBastionProject():

2017-07-20T19:22:17 BryanDavis (talk | contribs | block) changed group membership for WikiSysop from bureaucrat, cloud administrator, shell user and administrator to (none) (Rights cleanup per phab:T171090)
10:50 AM 2017-07-20T19:25:05 BryanDavis (talk | contribs | block) blocked WikiSysop (talk | contribs) with an expiration time of indefinite (account creation disabled, autoblock disabled) (Disabling old admin account. See phab:T171090) (unblock | change block)

In particular, the WikiSysop user doesn't have an ldap entry, and also is ID #1. Two corner cases.

Change 366973 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[mediawiki/extensions/LdapAuthentication@master] when user lookups fail, NULL out $this->userInfo

https://gerrit.wikimedia.org/r/366973

Change 366973 merged by jenkins-bot:
[mediawiki/extensions/LdapAuthentication@master] when user lookups fail, NULL out $this->userInfo

https://gerrit.wikimedia.org/r/366973

bd808 assigned this task to Andrew.Jul 27 2017, 3:03 AM
bd808 edited projects, added cloud-services-team (Kanban); removed Patch-For-Review.
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
Andrew closed this task as Resolved.Jul 27 2017, 2:56 PM

Attached patch should resolve the ultimate cause.