I've been noticing a pattern of failure with standing up large trove databases. I just now did a binary search and have learned that the build stage always fails if the requested volume size is greater than 31 GB.
The volume is created, and the instance is created, and the attachment is created, but the large volumes aren't properly mounted. Could be a guest-agent bug, or an Ubuntu edge case, or a config setting that I've overlooked.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Bstorm | T216208 ToolsDB overload and cleanup | |||
| Resolved | • Bstorm | T216441 Evaluate transferring the non-replicated tables to the new toolsdb server | |||
| Resolved | fnegri | T236101 Find a way to remove non-replicated tables from ToolsDB | |||
| Resolved | dcaro | T301951 toolsdb: full disk on clouddb1001 broke clouddb1002 (secondary) replication | |||
| Open | None | T301967 toolsdb: evaluate storage usage by some tools | |||
| Open | None | T291782 Migrate largest ToolsDB users to Trove | |||
| Resolved | Andrew | T323502 Move some of magnus's tools to Trove databases (was: Request increased quota for mix-n-match Toolforge tool) | |||
| Resolved | Andrew | T324984 Trove volume size limit of 31Gb |
I've logged this as upstream bug https://storyboard.openstack.org/#!/story/2010468
The issue is a timeout that happens during volume format. On instance stand-up the guest-agent runs
$ /usr/sbin/mkfs --type ext3 -m 5 /dev/sdbIf that takes longer than two minutes, the instance is doomed forever.
Change 867279 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Trove: increase volume_format_timeout to half an hour.
Change 867279 merged by Andrew Bogott:
[operations/puppet@production] Trove: increase volume_format_timeout to half an hour.
Change 867284 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Trove: copy format timeout to trove-guestagent.conf
Change 867284 merged by Andrew Bogott:
[operations/puppet@production] Trove: copy format timeout to trove-guestagent.conf
Change 867288 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] trove-guestagent.conf: remove some commented-out sections
Change 867288 merged by Andrew Bogott:
[operations/puppet@production] trove-guestagent.conf: remove some commented-out sections