Page MenuHomePhabricator

Trove volume size limit of 31Gb
Closed, ResolvedPublic

Description

I've been noticing a pattern of failure with standing up large trove databases. I just now did a binary search and have learned that the build stage always fails if the requested volume size is greater than 31 GB.

The volume is created, and the instance is created, and the attachment is created, but the large volumes aren't properly mounted. Could be a guest-agent bug, or an Ubuntu edge case, or a config setting that I've overlooked.

Event Timeline

I've logged this as upstream bug https://storyboard.openstack.org/#!/story/2010468

The issue is a timeout that happens during volume format. On instance stand-up the guest-agent runs

$ /usr/sbin/mkfs --type ext3 -m 5 /dev/sdb

If that takes longer than two minutes, the instance is doomed forever.

Change 867279 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Trove: increase volume_format_timeout to half an hour.

https://gerrit.wikimedia.org/r/867279

Change 867279 merged by Andrew Bogott:

[operations/puppet@production] Trove: increase volume_format_timeout to half an hour.

https://gerrit.wikimedia.org/r/867279

Change 867284 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Trove: copy format timeout to trove-guestagent.conf

https://gerrit.wikimedia.org/r/867284

Change 867284 merged by Andrew Bogott:

[operations/puppet@production] Trove: copy format timeout to trove-guestagent.conf

https://gerrit.wikimedia.org/r/867284

Change 867288 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] trove-guestagent.conf: remove some commented-out sections

https://gerrit.wikimedia.org/r/867288

Change 867288 merged by Andrew Bogott:

[operations/puppet@production] trove-guestagent.conf: remove some commented-out sections

https://gerrit.wikimedia.org/r/867288

I adjusted several limits and now I an stand up a 2TB trove instance.