Page MenuHomePhabricator

puppetize turning off reserved space for cassandra /srv
Closed, ResolvedPublic

Description

Some of the machines in the RESTBase Cassandra cluster have had the reserved space disabled (set to 0) on the volume where Cassandra storage happens, others have the standard 5%, which on a ~5T volume amounts to ~231G. Since Cassandra is the only thing using these volumes, it would seem sensible to remove the reserved space, and to make this a standard (and documented) part of the configuration. I'd be interested to hear if there are any objections from Ops on the former, as well as any suggestions about how best to do the latter (Puppet?).

Event Timeline

Eevans created this task.Apr 13 2016, 9:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 13 2016, 9:29 PM
Eevans triaged this task as Low priority.Apr 13 2016, 9:29 PM

FTR it was agreed to puppetize tune2fs to set /srv reserved space to zero

fgiunchedi renamed this task from Filesystem reserved space? to puppetize turning off reserved space for cassandra /srv.Apr 29 2016, 8:33 AM

@fgiunchedi, has this already happened?

GWicke moved this task from Backlog to watching on the Services board.Jul 11 2017, 7:54 PM
GWicke edited projects, added Services (watching); removed Services.

@Eevans @fgiunchedi should we go ahead with this?

@Eevans @fgiunchedi should we go ahead with this?

I don't think there was ever any controversy over whether we should do this. AFAIK, it didn't happen because it wasn't obvious (to me at least) how, and because of a general lack of time (or relative priority, etc).

@Eevans @fgiunchedi should we go ahead with this?

I don't think there was ever any controversy over whether we should do this. AFAIK, it didn't happen because it wasn't obvious (to me at least) how, and because of a general lack of time (or relative priority, etc).

Indeed this has never been prioritized, partly for lack of time too.

Eevans moved this task from Backlog to In-Progress on the User-Eevans board.Aug 28 2019, 6:53 PM

Interestingly, reserved space on the main data volumes in the production cluster already have zero reserved blocks.

1restbase1016.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
2restbase1016.eqiad.wmnet: /dev/sda4 Reserved block count: 0
3restbase1016.eqiad.wmnet: /dev/sdd4 Reserved block count: 0
4restbase1016.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
5restbase1016.eqiad.wmnet: /dev/md2 Reserved block count: 487859
6restbase1019.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
7restbase1019.eqiad.wmnet: /dev/sda4 Reserved block count: 0
8restbase1019.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
9restbase1019.eqiad.wmnet: /dev/md2 Reserved block count: 548902
10restbase1020.eqiad.wmnet: /dev/sda4 Reserved block count: 0
11restbase1020.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
12restbase1020.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
13restbase1020.eqiad.wmnet: /dev/md2 Reserved block count: 548902
14restbase1021.eqiad.wmnet: /dev/sda4 Reserved block count: 0
15restbase1021.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
16restbase1021.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
17restbase1021.eqiad.wmnet: /dev/md2 Reserved block count: 548902
18restbase1017.eqiad.wmnet: /dev/md2 Reserved block count: 487859
19restbase1017.eqiad.wmnet: /dev/sda4 Reserved block count: 0
20restbase1017.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
21restbase1017.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
22restbase1017.eqiad.wmnet: /dev/sdd4 Reserved block count: 0
23restbase1022.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
24restbase1022.eqiad.wmnet: /dev/sda4 Reserved block count: 0
25restbase1022.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
26restbase1022.eqiad.wmnet: /dev/md2 Reserved block count: 548902
27restbase1023.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
28restbase1023.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
29restbase1023.eqiad.wmnet: /dev/sda4 Reserved block count: 0
30restbase1023.eqiad.wmnet: /dev/md2 Reserved block count: 548902
31restbase1024.eqiad.wmnet: /dev/sda4 Reserved block count: 0
32restbase1024.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
33restbase1024.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
34restbase1024.eqiad.wmnet: /dev/md2 Reserved block count: 548902
35restbase1018.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
36restbase1018.eqiad.wmnet: /dev/sda4 Reserved block count: 0
37restbase1018.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
38restbase1018.eqiad.wmnet: /dev/sdd4 Reserved block count: 0
39restbase1018.eqiad.wmnet: /dev/md2 Reserved block count: 487859
40restbase1025.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
41restbase1025.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
42restbase1025.eqiad.wmnet: /dev/sda4 Reserved block count: 0
43restbase1025.eqiad.wmnet: /dev/md2 Reserved block count: 548902
44restbase1026.eqiad.wmnet: /dev/sda4 Reserved block count: 0
45restbase1026.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
46restbase1026.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
47restbase1026.eqiad.wmnet: /dev/md2 Reserved block count: 548902
48restbase1027.eqiad.wmnet: /dev/sdb4 Reserved block count: 0
49restbase1027.eqiad.wmnet: /dev/sda4 Reserved block count: 0
50restbase1027.eqiad.wmnet: /dev/sdc4 Reserved block count: 0
51restbase1027.eqiad.wmnet: /dev/md2 Reserved block count: 548902
52restbase2011.codfw.wmnet: /dev/sda4 Reserved block count: 0
53restbase2011.codfw.wmnet: /dev/sdb4 Reserved block count: 0
54restbase2011.codfw.wmnet: /dev/sdd4 Reserved block count: 0
55restbase2011.codfw.wmnet: /dev/sdc4 Reserved block count: 0
56restbase2011.codfw.wmnet: /dev/md2 Reserved block count: 487859
57restbase2015.codfw.wmnet: /dev/sdc4 Reserved block count: 0
58restbase2015.codfw.wmnet: /dev/sdb4 Reserved block count: 0
59restbase2015.codfw.wmnet: /dev/sda4 Reserved block count: 0
60restbase2015.codfw.wmnet: /dev/md2 Reserved block count: 548902
61restbase2016.codfw.wmnet: /dev/sdc4 Reserved block count: 0
62restbase2016.codfw.wmnet: /dev/sdb4 Reserved block count: 0
63restbase2016.codfw.wmnet: /dev/sda4 Reserved block count: 0
64restbase2016.codfw.wmnet: /dev/md2 Reserved block count: 548902
65restbase2020.codfw.wmnet: /dev/sda4 Reserved block count: 0
66restbase2020.codfw.wmnet: /dev/sdc4 Reserved block count: 0
67restbase2020.codfw.wmnet: /dev/sdb4 Reserved block count: 0
68restbase2020.codfw.wmnet: /dev/md2 Reserved block count: 548902
69restbase2010.codfw.wmnet: /dev/sda4 Reserved block count: 0
70restbase2010.codfw.wmnet: /dev/sdc4 Reserved block count: 0
71restbase2010.codfw.wmnet: /dev/sdb4 Reserved block count: 0
72restbase2010.codfw.wmnet: /dev/sdd4 Reserved block count: 0
73restbase2010.codfw.wmnet: /dev/md2 Reserved block count: 487859
74restbase2013.codfw.wmnet: /dev/sdb4 Reserved block count: 0
75restbase2013.codfw.wmnet: /dev/sda4 Reserved block count: 0
76restbase2013.codfw.wmnet: /dev/sdc4 Reserved block count: 0
77restbase2013.codfw.wmnet: /dev/md2 Reserved block count: 548902
78restbase2014.codfw.wmnet: /dev/sdb4 Reserved block count: 0
79restbase2014.codfw.wmnet: /dev/sda4 Reserved block count: 0
80restbase2014.codfw.wmnet: /dev/sdc4 Reserved block count: 0
81restbase2014.codfw.wmnet: /dev/md2 Reserved block count: 548902
82restbase2019.codfw.wmnet: /dev/sdb4 Reserved block count: 0
83restbase2019.codfw.wmnet: /dev/sda4 Reserved block count: 0
84restbase2019.codfw.wmnet: /dev/sdc4 Reserved block count: 0
85restbase2019.codfw.wmnet: /dev/md2 Reserved block count: 548902
86restbase2009.codfw.wmnet: /dev/sdc4 Reserved block count: 0
87restbase2009.codfw.wmnet: /dev/sdb4 Reserved block count: 0
88restbase2009.codfw.wmnet: /dev/sda4 Reserved block count: 0
89restbase2009.codfw.wmnet: /dev/sdd4 Reserved block count: 0
90restbase2009.codfw.wmnet: /dev/md2 Reserved block count: 487859
91restbase2012.codfw.wmnet: /dev/sda4 Reserved block count: 0
92restbase2012.codfw.wmnet: /dev/sdd4 Reserved block count: 0
93restbase2012.codfw.wmnet: /dev/sdb4 Reserved block count: 0
94restbase2012.codfw.wmnet: /dev/sdc4 Reserved block count: 0
95restbase2012.codfw.wmnet: /dev/md2 Reserved block count: 487859
96restbase2017.codfw.wmnet: /dev/sda4 Reserved block count: 0
97restbase2017.codfw.wmnet: /dev/sdb4 Reserved block count: 0
98restbase2017.codfw.wmnet: /dev/sdc4 Reserved block count: 0
99restbase2017.codfw.wmnet: /dev/md2 Reserved block count: 548902
100restbase2018.codfw.wmnet: /dev/sdb4 Reserved block count: 0
101restbase2018.codfw.wmnet: /dev/sda4 Reserved block count: 0
102restbase2018.codfw.wmnet: /dev/sdc4 Reserved block count: 0
103restbase2018.codfw.wmnet: /dev/md2 Reserved block count: 548902

The shared instance-data (/dev/md2) still has the standard 5%, though these partitions are smaller in comparison (5% amounts to < 2G), and we are never fully utilizing them (by design), so I do not think this is a problem.

NOTE: Collected using P8997
Eevans removed a project: User-Eevans.

I do not know how it came to pass that machines are getting setup without reserved space, but given how long this issue has been open (and since I'm still unsure how to best go about Puppetizing this), I think we should accept this as a gift and close the issue.

Looks like we effectively switched to zero reserved space for root when moving to JBOD (Iada7cb4e472b) so I'm +1 on resolving this

mobrovac closed this task as Resolved.Sep 2 2019, 9:31 AM