Some of the machines in the RESTBase Cassandra cluster have had the reserved space disabled (set to 0) on the volume where Cassandra storage happens, others have the standard 5%, which on a ~5T volume amounts to ~231G. Since Cassandra is the only thing using these volumes, it would seem sensible to remove the reserved space, and to make this a standard (and documented) part of the configuration. I'd be interested to hear if there are any objections from Ops on the former, as well as any suggestions about how best to do the latter (Puppet?).
Description
Related Objects
Event Timeline
In T132632#5023075, @mobrovac wrote:@Eevans @fgiunchedi should we go ahead with this?
I don't think there was ever any controversy over whether we should do this. AFAIK, it didn't happen because it wasn't obvious (to me at least) how, and because of a general lack of time (or relative priority, etc).
In T132632#5023911, @Eevans wrote:In T132632#5023075, @mobrovac wrote:@Eevans @fgiunchedi should we go ahead with this?
I don't think there was ever any controversy over whether we should do this. AFAIK, it didn't happen because it wasn't obvious (to me at least) how, and because of a general lack of time (or relative priority, etc).
Indeed this has never been prioritized, partly for lack of time too.
Interestingly, reserved space on the main data volumes in the production cluster already have zero reserved blocks.
1 | restbase1016.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
---|---|
2 | restbase1016.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
3 | restbase1016.eqiad.wmnet: /dev/sdd4 Reserved block count: 0 |
4 | restbase1016.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
5 | restbase1016.eqiad.wmnet: /dev/md2 Reserved block count: 487859 |
6 | restbase1019.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
7 | restbase1019.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
8 | restbase1019.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
9 | restbase1019.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
10 | restbase1020.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
11 | restbase1020.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
12 | restbase1020.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
13 | restbase1020.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
14 | restbase1021.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
15 | restbase1021.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
16 | restbase1021.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
17 | restbase1021.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
18 | restbase1017.eqiad.wmnet: /dev/md2 Reserved block count: 487859 |
19 | restbase1017.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
20 | restbase1017.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
21 | restbase1017.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
22 | restbase1017.eqiad.wmnet: /dev/sdd4 Reserved block count: 0 |
23 | restbase1022.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
24 | restbase1022.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
25 | restbase1022.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
26 | restbase1022.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
27 | restbase1023.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
28 | restbase1023.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
29 | restbase1023.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
30 | restbase1023.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
31 | restbase1024.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
32 | restbase1024.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
33 | restbase1024.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
34 | restbase1024.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
35 | restbase1018.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
36 | restbase1018.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
37 | restbase1018.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
38 | restbase1018.eqiad.wmnet: /dev/sdd4 Reserved block count: 0 |
39 | restbase1018.eqiad.wmnet: /dev/md2 Reserved block count: 487859 |
40 | restbase1025.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
41 | restbase1025.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
42 | restbase1025.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
43 | restbase1025.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
44 | restbase1026.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
45 | restbase1026.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
46 | restbase1026.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
47 | restbase1026.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
48 | restbase1027.eqiad.wmnet: /dev/sdb4 Reserved block count: 0 |
49 | restbase1027.eqiad.wmnet: /dev/sda4 Reserved block count: 0 |
50 | restbase1027.eqiad.wmnet: /dev/sdc4 Reserved block count: 0 |
51 | restbase1027.eqiad.wmnet: /dev/md2 Reserved block count: 548902 |
52 | restbase2011.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
53 | restbase2011.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
54 | restbase2011.codfw.wmnet: /dev/sdd4 Reserved block count: 0 |
55 | restbase2011.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
56 | restbase2011.codfw.wmnet: /dev/md2 Reserved block count: 487859 |
57 | restbase2015.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
58 | restbase2015.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
59 | restbase2015.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
60 | restbase2015.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
61 | restbase2016.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
62 | restbase2016.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
63 | restbase2016.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
64 | restbase2016.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
65 | restbase2020.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
66 | restbase2020.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
67 | restbase2020.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
68 | restbase2020.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
69 | restbase2010.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
70 | restbase2010.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
71 | restbase2010.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
72 | restbase2010.codfw.wmnet: /dev/sdd4 Reserved block count: 0 |
73 | restbase2010.codfw.wmnet: /dev/md2 Reserved block count: 487859 |
74 | restbase2013.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
75 | restbase2013.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
76 | restbase2013.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
77 | restbase2013.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
78 | restbase2014.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
79 | restbase2014.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
80 | restbase2014.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
81 | restbase2014.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
82 | restbase2019.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
83 | restbase2019.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
84 | restbase2019.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
85 | restbase2019.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
86 | restbase2009.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
87 | restbase2009.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
88 | restbase2009.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
89 | restbase2009.codfw.wmnet: /dev/sdd4 Reserved block count: 0 |
90 | restbase2009.codfw.wmnet: /dev/md2 Reserved block count: 487859 |
91 | restbase2012.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
92 | restbase2012.codfw.wmnet: /dev/sdd4 Reserved block count: 0 |
93 | restbase2012.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
94 | restbase2012.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
95 | restbase2012.codfw.wmnet: /dev/md2 Reserved block count: 487859 |
96 | restbase2017.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
97 | restbase2017.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
98 | restbase2017.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
99 | restbase2017.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
100 | restbase2018.codfw.wmnet: /dev/sdb4 Reserved block count: 0 |
101 | restbase2018.codfw.wmnet: /dev/sda4 Reserved block count: 0 |
102 | restbase2018.codfw.wmnet: /dev/sdc4 Reserved block count: 0 |
103 | restbase2018.codfw.wmnet: /dev/md2 Reserved block count: 548902 |
The shared instance-data (/dev/md2) still has the standard 5%, though these partitions are smaller in comparison (5% amounts to < 2G), and we are never fully utilizing them (by design), so I do not think this is a problem.
I do not know how it came to pass that machines are getting setup without reserved space, but given how long this issue has been open (and since I'm still unsure how to best go about Puppetizing this), I think we should accept this as a gift and close the issue.
Looks like we effectively switched to zero reserved space for root when moving to JBOD (Iada7cb4e472b) so I'm +1 on resolving this