novaadmin removed from many keystone projects
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Andrew
	Jul 21 2017, 3:01 PM

Description

novaadmin should be in every project. Yesterday it stopped being in lots of projects:

Roles for novaadmin are not set in these projects: set([u'phabricator', u'packaging', u'analytics', u'puppet', u'netdata', u'openstack', u'newsletter', u'maps-team', u'account-creation-assistance', u'servermon', u'testlabs', u'wikidata-dev', u'ores-staging', u'contributors', u'maps', u'huggle', u'tools', u'lizenzhinweisgenerator', u'integration', u'twl', u'wmt', u'kubernetes-testing', u'bastion', u'osmit', u'etcd', u'queryrapi', u'otrs', u'wikidata-federation', u'math', u'project-proxy', u'catgraph', u'wikidataconcepts', u'shinken', u'openocr', u'puppet3-diffs', u'search', u'scrumbugz', u'mwfileimport', u'services', u'wikidata-query', u'toolsbeta', u'wikidata-build', u'language', u'wikifactmine', u'deployment-prep', u'puppet-ca-replacement', u'gerrit', u'dumps', u'librarybase', u'testproject', u'bots', u'etherpad', u'ores'])

Easy to fix, but... what happened?

Details

	Subject	Repo	Branch	Lines +/-
	when user lookups fail, NULL out $this->userInfo	mediawiki/extensions/LdapAuthentication	master	+3 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Andrew	T171280 wikitech api list=novainstances not returning list of instances
		Resolved		Andrew	T171313 novaadmin removed from many keystone projects

Event Timeline

Andrew created this task.Jul 21 2017, 3:01 PM

Restricted Application added subscribers: PokestarFan, Sadads, Aklapper. · View Herald TranscriptJul 21 2017, 3:01 PM

We've replaced novaadmin in deployment-prep; now it's missing from the following:

account-creation-assistance
analytics
bastion
bots
catgraph
contributors
dumps
etcd
etherpad
gerrit
huggle
integration
kubernetes-testing
language
librarybase
lizenzhinweisgenerator
maps
maps-team
math
mwfileimport
netdata
newsletter
openocr
openstack
ores
ores-staging
osmit
otrs
packaging
phabricator
project-proxy
puppet
puppet-ca-replacement
puppet3-diffs
queryrapi
scrumbugz
search
servermon
services
shinken
testlabs
testproject
tools
toolsbeta
twl
wikidata-build
wikidata-dev
wikidata-federation
wikidata-query
wikidataconcepts
wikifactmine
wmt

In total it was removed from 53 projects. I'm now checking to see if any one user is in all of those projects (other than novaadmin)

(update) Save novaadmin there's no one user account that has projectadmin in 53 projects, no even mine.

The first sign of trouble in the keystone log is:

(keystone.token.controllers): 2017-07-20 19:20:04,019 WARNING User novaadmin is unauthorized for tenant tools

Before that, keystone was restarted at 2017-07-20 18:14:20. That restart was probably due to the merge of https://gerrit.wikimedia.org/r/#/c/366025/ which happened at 18:12.

Reedy added a project: wikitech.wikimedia.org.Jul 21 2017, 4:04 PM

So currently I think this was caused by a misfire in OpenStackManager's removeUserFromBastionProject():

2017-07-20T19:22:17 BryanDavis (talk | contribs | block) changed group membership for WikiSysop from bureaucrat, cloud administrator, shell user and administrator to (none) (Rights cleanup per phab:T171090)
10:50 AM 2017-07-20T19:25:05 BryanDavis (talk | contribs | block) blocked WikiSysop (talk | contribs) with an expiration time of indefinite (account creation disabled, autoblock disabled) (Disabling old admin account. See phab:T171090) (unblock | change block)

In particular, the WikiSysop user doesn't have an ldap entry, and also is ID #1. Two corner cases.

Andrew added a parent task: T171280: wikitech api list=novainstances not returning list of instances.Jul 21 2017, 9:52 PM

Change 366973 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[mediawiki/extensions/LdapAuthentication@master] when user lookups fail, NULL out $this->userInfo

https://gerrit.wikimedia.org/r/366973

gerritbot added a project: Patch-For-Review.Jul 21 2017, 9:53 PM

Change 366973 merged by jenkins-bot:
[mediawiki/extensions/LdapAuthentication@master] when user lookups fail, NULL out $this->userInfo

https://gerrit.wikimedia.org/r/366973

ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)).Jul 21 2017, 11:00 PM

bd808 assigned this task to Andrew.Jul 27 2017, 3:03 AM

bd808 edited projects, added cloud-services-team (Kanban); removed Patch-For-Review.

bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Attached patch should resolve the ultimate cause.

bd808 moved this task from Doing to Done on the cloud-services-team (Kanban) board.Jul 28 2017, 11:23 PM