contintcloud instance refuses to launch due to "Maximum number of fixed ips exceeded
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Jul 20 2017, 11:01 AM

Description

Since 2017-07-20 01:37:46 bunch of Nodepool instances refuse to launch with a:

LaunchStatusException: Server beb9cb86-5a32-4f5b-8bea-75a883ce2030 for node id: 748489 status: ERROR

I caught one using openstack server show which reported:

Build of instance bb4dfe79-4c14-4b0e-a75a-d58633448984 aborted:
Failed to allocate the network(s) with error Maximum number of fixed ips exceeded, not rescheduling.
'code': 500, u'created': u'2017-07-20T10:50:20Z'

Field	Value
OS-DCF:diskConfig	MANUAL
OS-EXT-AZ:availability_zone
OS-EXT-STS:power_state	0
OS-EXT-STS:task_state	None
OS-EXT-STS:vm_state	error
OS-SRV-USG:launched_at	None
OS-SRV-USG:terminated_at	None
accessIPv4
accessIPv6
addresses
config_drive
created	2017-07-20T10:50:17Z
fault	{u'message': u'Build of instance bb4dfe79-4c14-4b0e-a75a-d58633448984 aborted: Failed to allocate the network(s) with error Maximum number of fixed ips exceeded, not rescheduling.', u'code': 500, u'created': u'2017-07-20T10:50:20Z'}
flavor	m1.medium (3)
hostId
id	bb4dfe79-4c14-4b0e-a75a-d58633448984
image	snapshot-ci-jessie-1500473642 (f7967e47-8b5b-4b0b-8da2-1b75d9273a8a)
key_name	None
name	ci-jessie-wikimedia-749798
os-extended-volumes:volumes_attached	[]
project_id	contintcloud
properties
security_groups	[{u'name': u'default'}]
status	ERROR
updated	2017-07-20T10:50:20Z
user_id	nodepoolmanager

nova absolute-limits for contintcloud:

Name	Used	Max
Cores	19	58
FloatingIps	-1	0
ImageMeta	-	128
Instances	9	29
Keypairs	-	100
Personality	-	5
Personality Size	-	10240
RAM	40959	118784
SecurityGroupRules	-	20
SecurityGroups	-2	10
Server Meta	-	128
ServerGroupMembers	-	10
ServerGroups	0	10

Related Objects

Mentioned In: T242332: CloudVPS: wrong nova quota usage for a project detected
T158350: contintcloud project thinks it is using 206 fixed-ip quota errantly
Mentioned Here: T158350: contintcloud project thinks it is using 206 fixed-ip quota errantly

Event Timeline

hashar created this task.Jul 20 2017, 11:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 20 2017, 11:01 AM

hashar updated the task description. (Show Details)Jul 20 2017, 11:01 AM

Paladox subscribed.Jul 20 2017, 11:13 AM

labnet1001.eqiad.wmnet has a lot of such errors in /var/log/nova/nova-network.log*

The first suspicious one:

2017-07-20 01:34:42.679 14215 WARNING nova.network.manager
[req-e87b9df8-ec96-4e72-9fc4-b2fde54d3d8f nodepoolmanager contintcloud - - -]
Error cleaning up fixed ip allocation. Manual cleanup may be required

And there is some ValueError: Circular reference detected.

They pill up until the quota of 200 ip has been reached:

2017-07-20 06:23:18.797 15836 WARNING nova.network.manager
[req-a8c1e602-2d89-42ea-bc86-99c3ce15b017 nodepoolmanager contintcloud - - -]
[instance: 3b6d46aa-c2b6-4347-8706-fd8b13f96bc3]
Quota exceeded for project contintcloud, tried to allocate fixed IP.
200 of 200 are in use or are already reserved.

Which raises FixedIpLimitExceeded: Maximum number of fixed ips exceeded.

That is the same issue as T158350 which got fixed via:

In T158350#3036994, @Andrew wrote:

I restarted nova-network and it looks like nova is cleaning up those leaks now. I'll keep an eye out, but I've reduced the quota to 200 and there's some slack now.

In T158350#3037145, @Andrew wrote:

I cleaned up about 100 leaks, like this:

update fixed_ips a, instances b set a.instance_uuid=NULL where a.instance_uuid = b.uuid and project_id='contintcloud' and b.deleted!='0';

After that, unowned ips in contintcloud are staying in the single-digits an seem to be getting cleaned up regularly.

hashar mentioned this in T158350: contintcloud project thinks it is using 206 fixed-ip quota errantly.Jul 20 2017, 11:16 AM

The Nodepool launch errors https://grafana.wikimedia.org/dashboard/db/nodepool?panelId=12&fullscreen&orgId=1&from=now-20h&to=now

openstack_errors.png (447×818 px, 54 KB)

Seems the nova database is on m5-master.eqiad.wmnet db name nova.

Luke081515 triaged this task as High priority.Jul 20 2017, 12:01 PM

I resolved this by running the query in https://ask.openstack.org/en/question/494/how-to-reset-incorrect-quota-count/

BUT

the 'reserved' quota was also broken, so I also ran

update quota_usages set reserved='0' where project_id='contintcloud';

Reserved started to increment right after that so I think we're good.

I can confirm that resolved the issue completely. Thank you!

https://wikitech.wikimedia.org/wiki/Incident_documentation/20170719-ldap#CI.2Fbeta

• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:44 PM

• aborrero mentioned this in T242332: CloudVPS: wrong nova quota usage for a project detected.Jan 9 2020, 11:07 AM

	F8801087: openstack_errors.png
	Jul 20 2017, 11:24 AM

contintcloud instance refuses to launch due to "Maximum number of fixed ips exceededClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

contintcloud instance refuses to launch due to "Maximum number of fixed ips exceeded
Closed, ResolvedPublic
Actions