db1024 (s2 master) will run out of disk space in ~4 months
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	jcrespo
	Dec 21 2015, 1:37 PM

Description

Status	Assigned	Task
Resolved	None	T122048 db1024 (s2 master) will run out of disk space in ~4 months
		Unknown Object (Task)
		Unknown Object (Task)
		Unknown Object (Task)
		Unknown Object (Task)
Resolved	jcrespo	T124681 Architecture decision to solve the need larger serves (for better capacity and consolidation) vs. more, smaller servers (for high availability)
Resolved	jcrespo	T125215 Prepare db1018 and s2-slaves for s2 master failover

jcrespo raised the priority of this task from to High.

jcrespo updated the task description. (Show Details)

jcrespo added projects: DBA, SRE.

jcrespo subscribed.

jcrespo mentioned this in Unknown Object (Task).Dec 21 2015, 1:40 PM

jcrespo added a subtask: Unknown Object (Task).

While extending the Logical volume was successful, extending the actual xfs partition fails with:

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Cannot allocate memory

and dmesg says:

[60444789.323562] XFS (dm-0): _xfs_buf_find: Block out of range: block 0x95fff801, EOFS 0x95fff800

Versions:

Linux db1024 3.2.0-54-generic #82-Ubuntu
xfs_growfs version 3.1.7

Adding mark and robh as requested.

I'm... confused. So is this just a problem with XFS not growing to the new LV size, or do we actually need larger hardware? Or both?

Both. Extending the LVM will only be a temporary patch. The fact that we cannot apply it makes it more urgent.

Please note that some temporary tables (implicit) are not written to the tmp partition, so enough room is needed for performing ALTER tables.

This has accelerated in the last week, it may happen the 20 of February, scheduling a master failover soon.

jcrespo closed this task as Resolved.Feb 25 2016, 10:08 AM

mark closed subtask Unknown Object (Task) as Resolved.Apr 15 2016, 10:07 AM