This task is to test and document a way to achieve rack level HA.
Note that in codfw we don't really have the hosts in different racks, so we would not be able to test a real case, but we can experiment.
Note that C8 and D5 are together in codfw due to having only 3 hosts xd (in eqiad,
Deployment
Setting it manually, all at once
Reference: https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/
- dump current crushmap:
ceph osd getcrushmap -o crushmap.bin
- Decompile into text
crushtool -d crushmap.bin -o crushmap.txt
- Make a copy
cp crushmap.txt crushmap.$(date +%Y%m%d%H%M%S).before_rack_ha.txt
- Get the new prepared crushmap to a mon host from:
| 1 | # begin crush map |
|---|---|
| 2 | tunable choose_local_tries 0 |
| 3 | tunable choose_local_fallback_tries 0 |
| 4 | tunable choose_total_tries 50 |
| 5 | tunable chooseleaf_descend_once 1 |
| 6 | tunable chooseleaf_vary_r 1 |
| 7 | tunable chooseleaf_stable 1 |
| 8 | tunable straw_calc_version 1 |
| 9 | tunable allowed_bucket_algs 54 |
| 10 | |
| 11 | # devices |
| 12 | device 0 osd.0 class ssd |
| 13 | device 1 osd.1 class ssd |
| 14 | device 2 osd.2 class ssd |
| 15 | device 3 osd.3 class ssd |
| 16 | device 4 osd.4 class ssd |
| 17 | device 5 osd.5 class ssd |
| 18 | device 6 osd.6 class ssd |
| 19 | device 7 osd.7 class ssd |
| 20 | device 8 osd.8 class ssd |
| 21 | device 9 osd.9 class ssd |
| 22 | device 10 osd.10 class ssd |
| 23 | device 11 osd.11 class ssd |
| 24 | device 12 osd.12 class ssd |
| 25 | device 13 osd.13 class ssd |
| 26 | device 14 osd.14 class ssd |
| 27 | device 15 osd.15 class ssd |
| 28 | device 16 osd.16 class ssd |
| 29 | device 17 osd.17 class ssd |
| 30 | device 18 osd.18 class ssd |
| 31 | device 19 osd.19 class ssd |
| 32 | device 20 osd.20 class ssd |
| 33 | device 21 osd.21 class ssd |
| 34 | device 22 osd.22 class ssd |
| 35 | device 23 osd.23 class ssd |
| 36 | device 24 osd.24 class ssd |
| 37 | device 25 osd.25 class ssd |
| 38 | device 26 osd.26 class ssd |
| 39 | device 27 osd.27 class ssd |
| 40 | device 28 osd.28 class ssd |
| 41 | device 29 osd.29 class ssd |
| 42 | device 30 osd.30 class ssd |
| 43 | device 31 osd.31 class ssd |
| 44 | device 32 osd.32 class ssd |
| 45 | device 33 osd.33 class ssd |
| 46 | device 34 osd.34 class ssd |
| 47 | device 35 osd.35 class ssd |
| 48 | device 36 osd.36 class ssd |
| 49 | device 37 osd.37 class ssd |
| 50 | device 38 osd.38 class ssd |
| 51 | device 39 osd.39 class ssd |
| 52 | device 40 osd.40 class ssd |
| 53 | device 41 osd.41 class ssd |
| 54 | device 42 osd.42 class ssd |
| 55 | device 43 osd.43 class ssd |
| 56 | device 44 osd.44 class ssd |
| 57 | device 45 osd.45 class ssd |
| 58 | device 46 osd.46 class ssd |
| 59 | device 47 osd.47 class ssd |
| 60 | device 48 osd.48 class ssd |
| 61 | device 49 osd.49 class ssd |
| 62 | device 50 osd.50 class ssd |
| 63 | device 51 osd.51 class ssd |
| 64 | device 52 osd.52 class ssd |
| 65 | device 53 osd.53 class ssd |
| 66 | device 54 osd.54 class ssd |
| 67 | device 55 osd.55 class ssd |
| 68 | device 56 osd.56 class ssd |
| 69 | device 57 osd.57 class ssd |
| 70 | device 58 osd.58 class ssd |
| 71 | device 59 osd.59 class ssd |
| 72 | device 60 osd.60 class ssd |
| 73 | device 61 osd.61 class ssd |
| 74 | device 62 osd.62 class ssd |
| 75 | device 63 osd.63 class ssd |
| 76 | device 64 osd.64 class ssd |
| 77 | device 65 osd.65 class ssd |
| 78 | device 66 osd.66 class ssd |
| 79 | device 67 osd.67 class ssd |
| 80 | device 68 osd.68 class ssd |
| 81 | device 69 osd.69 class ssd |
| 82 | device 70 osd.70 class ssd |
| 83 | device 71 osd.71 class ssd |
| 84 | device 72 osd.72 class ssd |
| 85 | device 73 osd.73 class ssd |
| 86 | device 74 osd.74 class ssd |
| 87 | device 75 osd.75 class ssd |
| 88 | device 76 osd.76 class ssd |
| 89 | device 77 osd.77 class ssd |
| 90 | device 78 osd.78 class ssd |
| 91 | device 79 osd.79 class ssd |
| 92 | device 80 osd.80 class ssd |
| 93 | device 81 osd.81 class ssd |
| 94 | device 82 osd.82 class ssd |
| 95 | device 83 osd.83 class ssd |
| 96 | device 84 osd.84 class ssd |
| 97 | device 85 osd.85 class ssd |
| 98 | device 86 osd.86 class ssd |
| 99 | device 87 osd.87 class ssd |
| 100 | device 88 osd.88 class ssd |
| 101 | device 89 osd.89 class ssd |
| 102 | device 90 osd.90 class ssd |
| 103 | device 91 osd.91 class ssd |
| 104 | device 92 osd.92 class ssd |
| 105 | device 93 osd.93 class ssd |
| 106 | device 94 osd.94 class ssd |
| 107 | device 95 osd.95 class ssd |
| 108 | device 96 osd.96 class ssd |
| 109 | device 97 osd.97 class ssd |
| 110 | device 98 osd.98 class ssd |
| 111 | device 99 osd.99 class ssd |
| 112 | device 100 osd.100 class ssd |
| 113 | device 101 osd.101 class ssd |
| 114 | device 102 osd.102 class ssd |
| 115 | device 103 osd.103 class ssd |
| 116 | device 104 osd.104 class ssd |
| 117 | device 105 osd.105 class ssd |
| 118 | device 106 osd.106 class ssd |
| 119 | device 107 osd.107 class ssd |
| 120 | device 108 osd.108 class ssd |
| 121 | device 109 osd.109 class ssd |
| 122 | device 110 osd.110 class ssd |
| 123 | device 111 osd.111 class ssd |
| 124 | device 112 osd.112 class ssd |
| 125 | device 113 osd.113 class ssd |
| 126 | device 114 osd.114 class ssd |
| 127 | device 115 osd.115 class ssd |
| 128 | device 116 osd.116 class ssd |
| 129 | device 117 osd.117 class ssd |
| 130 | device 118 osd.118 class ssd |
| 131 | device 119 osd.119 class ssd |
| 132 | device 120 osd.120 class ssd |
| 133 | device 121 osd.121 class ssd |
| 134 | device 122 osd.122 class ssd |
| 135 | device 123 osd.123 class ssd |
| 136 | device 124 osd.124 class ssd |
| 137 | device 125 osd.125 class ssd |
| 138 | device 126 osd.126 class ssd |
| 139 | device 127 osd.127 class ssd |
| 140 | device 128 osd.128 class ssd |
| 141 | device 129 osd.129 class ssd |
| 142 | device 130 osd.130 class ssd |
| 143 | device 131 osd.131 class ssd |
| 144 | device 132 osd.132 class ssd |
| 145 | device 133 osd.133 class ssd |
| 146 | device 134 osd.134 class ssd |
| 147 | device 135 osd.135 class ssd |
| 148 | device 136 osd.136 class ssd |
| 149 | device 137 osd.137 class ssd |
| 150 | device 138 osd.138 class ssd |
| 151 | device 139 osd.139 class ssd |
| 152 | device 140 osd.140 class ssd |
| 153 | device 141 osd.141 class ssd |
| 154 | device 142 osd.142 class ssd |
| 155 | device 143 osd.143 class ssd |
| 156 | device 144 osd.144 class ssd |
| 157 | device 145 osd.145 class ssd |
| 158 | device 146 osd.146 class ssd |
| 159 | device 147 osd.147 class ssd |
| 160 | device 148 osd.148 class ssd |
| 161 | device 149 osd.149 class ssd |
| 162 | device 150 osd.150 class ssd |
| 163 | device 151 osd.151 class ssd |
| 164 | device 152 osd.152 class ssd |
| 165 | device 153 osd.153 class ssd |
| 166 | device 154 osd.154 class ssd |
| 167 | device 155 osd.155 class ssd |
| 168 | device 156 osd.156 class ssd |
| 169 | device 157 osd.157 class ssd |
| 170 | device 158 osd.158 class ssd |
| 171 | device 159 osd.159 class ssd |
| 172 | device 160 osd.160 class ssd |
| 173 | device 161 osd.161 class ssd |
| 174 | device 162 osd.162 class ssd |
| 175 | device 163 osd.163 class ssd |
| 176 | device 164 osd.164 class ssd |
| 177 | device 165 osd.165 class ssd |
| 178 | device 166 osd.166 class ssd |
| 179 | device 167 osd.167 class ssd |
| 180 | device 168 osd.168 class ssd |
| 181 | device 169 osd.169 class ssd |
| 182 | device 170 osd.170 class ssd |
| 183 | device 171 osd.171 class ssd |
| 184 | device 172 osd.172 class ssd |
| 185 | device 173 osd.173 class ssd |
| 186 | device 174 osd.174 class ssd |
| 187 | device 175 osd.175 class ssd |
| 188 | device 176 osd.176 class ssd |
| 189 | device 177 osd.177 class ssd |
| 190 | device 178 osd.178 class ssd |
| 191 | device 179 osd.179 class ssd |
| 192 | device 180 osd.180 class ssd |
| 193 | device 181 osd.181 class ssd |
| 194 | device 182 osd.182 class ssd |
| 195 | device 183 osd.183 class ssd |
| 196 | device 184 osd.184 class ssd |
| 197 | device 185 osd.185 class ssd |
| 198 | device 186 osd.186 class ssd |
| 199 | device 187 osd.187 class ssd |
| 200 | device 188 osd.188 class ssd |
| 201 | device 189 osd.189 class ssd |
| 202 | device 190 osd.190 class ssd |
| 203 | device 191 osd.191 class ssd |
| 204 | device 192 osd.192 class ssd |
| 205 | device 193 osd.193 class ssd |
| 206 | device 194 osd.194 class ssd |
| 207 | device 195 osd.195 class ssd |
| 208 | device 196 osd.196 class ssd |
| 209 | device 197 osd.197 class ssd |
| 210 | device 198 osd.198 class ssd |
| 211 | device 199 osd.199 class ssd |
| 212 | device 200 osd.200 class ssd |
| 213 | device 201 osd.201 class ssd |
| 214 | device 202 osd.202 class ssd |
| 215 | device 203 osd.203 class ssd |
| 216 | device 204 osd.204 class ssd |
| 217 | device 205 osd.205 class ssd |
| 218 | device 206 osd.206 class ssd |
| 219 | device 207 osd.207 class ssd |
| 220 | device 208 osd.208 class ssd |
| 221 | device 209 osd.209 class ssd |
| 222 | device 210 osd.210 class ssd |
| 223 | device 211 osd.211 class ssd |
| 224 | device 212 osd.212 class ssd |
| 225 | device 213 osd.213 class ssd |
| 226 | device 214 osd.214 class ssd |
| 227 | device 215 osd.215 class ssd |
| 228 | device 216 osd.216 class ssd |
| 229 | device 217 osd.217 class ssd |
| 230 | device 218 osd.218 class ssd |
| 231 | device 219 osd.219 class ssd |
| 232 | device 220 osd.220 class ssd |
| 233 | device 221 osd.221 class ssd |
| 234 | device 222 osd.222 class ssd |
| 235 | device 223 osd.223 class ssd |
| 236 | device 224 osd.224 class ssd |
| 237 | device 225 osd.225 class ssd |
| 238 | device 226 osd.226 class ssd |
| 239 | device 227 osd.227 class ssd |
| 240 | device 228 osd.228 class ssd |
| 241 | device 229 osd.229 class ssd |
| 242 | device 230 osd.230 class ssd |
| 243 | device 231 osd.231 class ssd |
| 244 | device 232 osd.232 class ssd |
| 245 | device 233 osd.233 class ssd |
| 246 | device 234 osd.234 class ssd |
| 247 | device 235 osd.235 class ssd |
| 248 | device 236 osd.236 class ssd |
| 249 | device 237 osd.237 class ssd |
| 250 | device 238 osd.238 class ssd |
| 251 | device 239 osd.239 class ssd |
| 252 | device 240 osd.240 class ssd |
| 253 | device 241 osd.241 class ssd |
| 254 | device 242 osd.242 class ssd |
| 255 | device 243 osd.243 class ssd |
| 256 | device 244 osd.244 class ssd |
| 257 | device 245 osd.245 class ssd |
| 258 | device 246 osd.246 class ssd |
| 259 | device 247 osd.247 class ssd |
| 260 | device 248 osd.248 class ssd |
| 261 | device 249 osd.249 class ssd |
| 262 | device 250 osd.250 class ssd |
| 263 | device 251 osd.251 class ssd |
| 264 | device 252 osd.252 class ssd |
| 265 | device 253 osd.253 class ssd |
| 266 | device 254 osd.254 class ssd |
| 267 | device 255 osd.255 class ssd |
| 268 | device 256 osd.256 class ssd |
| 269 | device 257 osd.257 class ssd |
| 270 | device 258 osd.258 class ssd |
| 271 | device 259 osd.259 class ssd |
| 272 | device 260 osd.260 class ssd |
| 273 | device 261 osd.261 class ssd |
| 274 | device 262 osd.262 class ssd |
| 275 | device 263 osd.263 class ssd |
| 276 | device 264 osd.264 class ssd |
| 277 | device 265 osd.265 class ssd |
| 278 | device 266 osd.266 class ssd |
| 279 | device 267 osd.267 class ssd |
| 280 | device 268 osd.268 class ssd |
| 281 | device 269 osd.269 class ssd |
| 282 | device 270 osd.270 class ssd |
| 283 | device 271 osd.271 class ssd |
| 284 | |
| 285 | # types |
| 286 | type 0 osd |
| 287 | type 1 host |
| 288 | type 2 chassis |
| 289 | type 3 rack |
| 290 | type 4 row |
| 291 | type 5 pdu |
| 292 | type 6 pod |
| 293 | type 7 room |
| 294 | type 8 datacenter |
| 295 | type 9 zone |
| 296 | type 10 region |
| 297 | type 11 root |
| 298 | |
| 299 | # buckets |
| 300 | host cloudcephosd1001 { |
| 301 | id -3 # do not change unnecessarily |
| 302 | id -4 class ssd # do not change unnecessarily |
| 303 | # weight 13.973 |
| 304 | alg straw2 |
| 305 | hash 0 # rjenkins1 |
| 306 | item osd.48 weight 1.747 |
| 307 | item osd.49 weight 1.747 |
| 308 | item osd.50 weight 1.747 |
| 309 | item osd.51 weight 1.747 |
| 310 | item osd.52 weight 1.747 |
| 311 | item osd.53 weight 1.747 |
| 312 | item osd.54 weight 1.747 |
| 313 | item osd.55 weight 1.747 |
| 314 | } |
| 315 | host cloudcephosd1002 { |
| 316 | id -5 # do not change unnecessarily |
| 317 | id -6 class ssd # do not change unnecessarily |
| 318 | # weight 13.973 |
| 319 | alg straw2 |
| 320 | hash 0 # rjenkins1 |
| 321 | item osd.56 weight 1.747 |
| 322 | item osd.57 weight 1.747 |
| 323 | item osd.58 weight 1.747 |
| 324 | item osd.59 weight 1.747 |
| 325 | item osd.60 weight 1.747 |
| 326 | item osd.61 weight 1.747 |
| 327 | item osd.62 weight 1.747 |
| 328 | item osd.63 weight 1.747 |
| 329 | } |
| 330 | host cloudcephosd1006 { |
| 331 | id -13 # do not change unnecessarily |
| 332 | id -14 class ssd # do not change unnecessarily |
| 333 | # weight 13.969 |
| 334 | alg straw2 |
| 335 | hash 0 # rjenkins1 |
| 336 | item osd.40 weight 1.746 |
| 337 | item osd.41 weight 1.746 |
| 338 | item osd.42 weight 1.746 |
| 339 | item osd.43 weight 1.746 |
| 340 | item osd.44 weight 1.746 |
| 341 | item osd.45 weight 1.746 |
| 342 | item osd.46 weight 1.746 |
| 343 | item osd.47 weight 1.746 |
| 344 | } |
| 345 | host cloudcephosd1007 { |
| 346 | id -22 # do not change unnecessarily |
| 347 | id -23 class ssd # do not change unnecessarily |
| 348 | # weight 13.969 |
| 349 | alg straw2 |
| 350 | hash 0 # rjenkins1 |
| 351 | item osd.0 weight 1.746 |
| 352 | item osd.1 weight 1.746 |
| 353 | item osd.2 weight 1.746 |
| 354 | item osd.3 weight 1.746 |
| 355 | item osd.4 weight 1.746 |
| 356 | item osd.5 weight 1.746 |
| 357 | item osd.6 weight 1.746 |
| 358 | item osd.7 weight 1.746 |
| 359 | } |
| 360 | host cloudcephosd1008 { |
| 361 | id -25 # do not change unnecessarily |
| 362 | id -26 class ssd # do not change unnecessarily |
| 363 | # weight 13.969 |
| 364 | alg straw2 |
| 365 | hash 0 # rjenkins1 |
| 366 | item osd.8 weight 1.746 |
| 367 | item osd.9 weight 1.746 |
| 368 | item osd.10 weight 1.746 |
| 369 | item osd.11 weight 1.746 |
| 370 | item osd.12 weight 1.746 |
| 371 | item osd.13 weight 1.746 |
| 372 | item osd.14 weight 1.746 |
| 373 | item osd.15 weight 1.746 |
| 374 | } |
| 375 | host cloudcephosd1009 { |
| 376 | id -28 # do not change unnecessarily |
| 377 | id -29 class ssd # do not change unnecessarily |
| 378 | # weight 13.969 |
| 379 | alg straw2 |
| 380 | hash 0 # rjenkins1 |
| 381 | item osd.16 weight 1.746 |
| 382 | item osd.17 weight 1.746 |
| 383 | item osd.18 weight 1.746 |
| 384 | item osd.19 weight 1.746 |
| 385 | item osd.20 weight 1.746 |
| 386 | item osd.21 weight 1.746 |
| 387 | item osd.22 weight 1.746 |
| 388 | item osd.23 weight 1.746 |
| 389 | } |
| 390 | host cloudcephosd1011 { |
| 391 | id -34 # do not change unnecessarily |
| 392 | id -35 class ssd # do not change unnecessarily |
| 393 | # weight 13.969 |
| 394 | alg straw2 |
| 395 | hash 0 # rjenkins1 |
| 396 | item osd.80 weight 1.746 |
| 397 | item osd.81 weight 1.746 |
| 398 | item osd.82 weight 1.746 |
| 399 | item osd.83 weight 1.746 |
| 400 | item osd.84 weight 1.746 |
| 401 | item osd.85 weight 1.746 |
| 402 | item osd.86 weight 1.746 |
| 403 | item osd.87 weight 1.746 |
| 404 | } |
| 405 | host cloudcephosd1012 { |
| 406 | id -37 # do not change unnecessarily |
| 407 | id -38 class ssd # do not change unnecessarily |
| 408 | # weight 13.969 |
| 409 | alg straw2 |
| 410 | hash 0 # rjenkins1 |
| 411 | item osd.88 weight 1.746 |
| 412 | item osd.89 weight 1.746 |
| 413 | item osd.90 weight 1.746 |
| 414 | item osd.91 weight 1.746 |
| 415 | item osd.92 weight 1.746 |
| 416 | item osd.93 weight 1.746 |
| 417 | item osd.94 weight 1.746 |
| 418 | item osd.95 weight 1.746 |
| 419 | } |
| 420 | host cloudcephosd1013 { |
| 421 | id -40 # do not change unnecessarily |
| 422 | id -41 class ssd # do not change unnecessarily |
| 423 | # weight 13.969 |
| 424 | alg straw2 |
| 425 | hash 0 # rjenkins1 |
| 426 | item osd.96 weight 1.746 |
| 427 | item osd.97 weight 1.746 |
| 428 | item osd.98 weight 1.746 |
| 429 | item osd.99 weight 1.746 |
| 430 | item osd.100 weight 1.746 |
| 431 | item osd.101 weight 1.746 |
| 432 | item osd.102 weight 1.746 |
| 433 | item osd.103 weight 1.746 |
| 434 | } |
| 435 | host cloudcephosd1014 { |
| 436 | id -43 # do not change unnecessarily |
| 437 | id -44 class ssd # do not change unnecessarily |
| 438 | # weight 13.969 |
| 439 | alg straw2 |
| 440 | hash 0 # rjenkins1 |
| 441 | item osd.104 weight 1.746 |
| 442 | item osd.105 weight 1.746 |
| 443 | item osd.106 weight 1.746 |
| 444 | item osd.107 weight 1.746 |
| 445 | item osd.108 weight 1.746 |
| 446 | item osd.109 weight 1.746 |
| 447 | item osd.110 weight 1.746 |
| 448 | item osd.111 weight 1.746 |
| 449 | } |
| 450 | host cloudcephosd1015 { |
| 451 | id -46 # do not change unnecessarily |
| 452 | id -47 class ssd # do not change unnecessarily |
| 453 | # weight 13.969 |
| 454 | alg straw2 |
| 455 | hash 0 # rjenkins1 |
| 456 | item osd.112 weight 1.746 |
| 457 | item osd.113 weight 1.746 |
| 458 | item osd.114 weight 1.746 |
| 459 | item osd.115 weight 1.746 |
| 460 | item osd.116 weight 1.746 |
| 461 | item osd.117 weight 1.746 |
| 462 | item osd.118 weight 1.746 |
| 463 | item osd.119 weight 1.746 |
| 464 | } |
| 465 | host cloudcephosd1016 { |
| 466 | id -9 # do not change unnecessarily |
| 467 | id -10 class ssd # do not change unnecessarily |
| 468 | # weight 13.973 |
| 469 | alg straw2 |
| 470 | hash 0 # rjenkins1 |
| 471 | item osd.120 weight 1.747 |
| 472 | item osd.121 weight 1.747 |
| 473 | item osd.122 weight 1.747 |
| 474 | item osd.123 weight 1.747 |
| 475 | item osd.124 weight 1.747 |
| 476 | item osd.125 weight 1.747 |
| 477 | item osd.126 weight 1.747 |
| 478 | item osd.127 weight 1.747 |
| 479 | } |
| 480 | host cloudcephosd1017 { |
| 481 | id -11 # do not change unnecessarily |
| 482 | id -12 class ssd # do not change unnecessarily |
| 483 | # weight 13.973 |
| 484 | alg straw2 |
| 485 | hash 0 # rjenkins1 |
| 486 | item osd.128 weight 1.747 |
| 487 | item osd.129 weight 1.747 |
| 488 | item osd.130 weight 1.747 |
| 489 | item osd.131 weight 1.747 |
| 490 | item osd.132 weight 1.747 |
| 491 | item osd.133 weight 1.747 |
| 492 | item osd.134 weight 1.747 |
| 493 | item osd.135 weight 1.747 |
| 494 | } |
| 495 | host cloudcephosd1019 { |
| 496 | id -15 # do not change unnecessarily |
| 497 | id -18 class ssd # do not change unnecessarily |
| 498 | # weight 13.973 |
| 499 | alg straw2 |
| 500 | hash 0 # rjenkins1 |
| 501 | item osd.136 weight 1.747 |
| 502 | item osd.137 weight 1.747 |
| 503 | item osd.138 weight 1.747 |
| 504 | item osd.139 weight 1.747 |
| 505 | item osd.140 weight 1.747 |
| 506 | item osd.141 weight 1.747 |
| 507 | item osd.142 weight 1.747 |
| 508 | item osd.143 weight 1.747 |
| 509 | } |
| 510 | host cloudcephosd1020 { |
| 511 | id -21 # do not change unnecessarily |
| 512 | id -24 class ssd # do not change unnecessarily |
| 513 | # weight 13.973 |
| 514 | alg straw2 |
| 515 | hash 0 # rjenkins1 |
| 516 | item osd.144 weight 1.747 |
| 517 | item osd.145 weight 1.747 |
| 518 | item osd.146 weight 1.747 |
| 519 | item osd.147 weight 1.747 |
| 520 | item osd.148 weight 1.747 |
| 521 | item osd.149 weight 1.747 |
| 522 | item osd.150 weight 1.747 |
| 523 | item osd.151 weight 1.747 |
| 524 | } |
| 525 | host cloudcephosd1018 { |
| 526 | id -27 # do not change unnecessarily |
| 527 | id -30 class ssd # do not change unnecessarily |
| 528 | # weight 13.973 |
| 529 | alg straw2 |
| 530 | hash 0 # rjenkins1 |
| 531 | item osd.152 weight 1.747 |
| 532 | item osd.153 weight 1.747 |
| 533 | item osd.154 weight 1.747 |
| 534 | item osd.155 weight 1.747 |
| 535 | item osd.156 weight 1.747 |
| 536 | item osd.157 weight 1.747 |
| 537 | item osd.158 weight 1.747 |
| 538 | item osd.159 weight 1.747 |
| 539 | } |
| 540 | host cloudcephosd1021 { |
| 541 | id -33 # do not change unnecessarily |
| 542 | id -36 class ssd # do not change unnecessarily |
| 543 | # weight 13.973 |
| 544 | alg straw2 |
| 545 | hash 0 # rjenkins1 |
| 546 | item osd.160 weight 1.747 |
| 547 | item osd.161 weight 1.747 |
| 548 | item osd.162 weight 1.747 |
| 549 | item osd.163 weight 1.747 |
| 550 | item osd.164 weight 1.747 |
| 551 | item osd.165 weight 1.747 |
| 552 | item osd.166 weight 1.747 |
| 553 | item osd.271 weight 1.747 |
| 554 | } |
| 555 | host cloudcephosd1022 { |
| 556 | id -39 # do not change unnecessarily |
| 557 | id -42 class ssd # do not change unnecessarily |
| 558 | # weight 13.973 |
| 559 | alg straw2 |
| 560 | hash 0 # rjenkins1 |
| 561 | item osd.167 weight 1.747 |
| 562 | item osd.168 weight 1.747 |
| 563 | item osd.169 weight 1.747 |
| 564 | item osd.170 weight 1.747 |
| 565 | item osd.171 weight 1.747 |
| 566 | item osd.172 weight 1.747 |
| 567 | item osd.173 weight 1.747 |
| 568 | item osd.174 weight 1.747 |
| 569 | } |
| 570 | host cloudcephosd1023 { |
| 571 | id -45 # do not change unnecessarily |
| 572 | id -48 class ssd # do not change unnecessarily |
| 573 | # weight 13.973 |
| 574 | alg straw2 |
| 575 | hash 0 # rjenkins1 |
| 576 | item osd.175 weight 1.747 |
| 577 | item osd.176 weight 1.747 |
| 578 | item osd.177 weight 1.747 |
| 579 | item osd.178 weight 1.747 |
| 580 | item osd.179 weight 1.747 |
| 581 | item osd.180 weight 1.747 |
| 582 | item osd.181 weight 1.747 |
| 583 | item osd.182 weight 1.747 |
| 584 | } |
| 585 | host cloudcephosd1024 { |
| 586 | id -49 # do not change unnecessarily |
| 587 | id -50 class ssd # do not change unnecessarily |
| 588 | # weight 13.973 |
| 589 | alg straw2 |
| 590 | hash 0 # rjenkins1 |
| 591 | item osd.183 weight 1.747 |
| 592 | item osd.184 weight 1.747 |
| 593 | item osd.185 weight 1.747 |
| 594 | item osd.186 weight 1.747 |
| 595 | item osd.187 weight 1.747 |
| 596 | item osd.188 weight 1.747 |
| 597 | item osd.189 weight 1.747 |
| 598 | item osd.190 weight 1.747 |
| 599 | } |
| 600 | host cloudcephosd1025 { |
| 601 | id -76 # do not change unnecessarily |
| 602 | id -77 class ssd # do not change unnecessarily |
| 603 | # weight 13.973 |
| 604 | alg straw2 |
| 605 | hash 0 # rjenkins1 |
| 606 | item osd.191 weight 1.747 |
| 607 | item osd.192 weight 1.747 |
| 608 | item osd.193 weight 1.747 |
| 609 | item osd.194 weight 1.747 |
| 610 | item osd.195 weight 1.747 |
| 611 | item osd.196 weight 1.747 |
| 612 | item osd.197 weight 1.747 |
| 613 | item osd.198 weight 1.747 |
| 614 | } |
| 615 | host cloudcephosd1026 { |
| 616 | id -51 # do not change unnecessarily |
| 617 | id -52 class ssd # do not change unnecessarily |
| 618 | # weight 13.973 |
| 619 | alg straw2 |
| 620 | hash 0 # rjenkins1 |
| 621 | item osd.199 weight 1.747 |
| 622 | item osd.200 weight 1.747 |
| 623 | item osd.201 weight 1.747 |
| 624 | item osd.202 weight 1.747 |
| 625 | item osd.203 weight 1.747 |
| 626 | item osd.204 weight 1.747 |
| 627 | item osd.205 weight 1.747 |
| 628 | item osd.206 weight 1.747 |
| 629 | } |
| 630 | host cloudcephosd1027 { |
| 631 | id -53 # do not change unnecessarily |
| 632 | id -54 class ssd # do not change unnecessarily |
| 633 | # weight 13.973 |
| 634 | alg straw2 |
| 635 | hash 0 # rjenkins1 |
| 636 | item osd.207 weight 1.747 |
| 637 | item osd.208 weight 1.747 |
| 638 | item osd.209 weight 1.747 |
| 639 | item osd.210 weight 1.747 |
| 640 | item osd.211 weight 1.747 |
| 641 | item osd.212 weight 1.747 |
| 642 | item osd.213 weight 1.747 |
| 643 | item osd.214 weight 1.747 |
| 644 | } |
| 645 | host cloudcephosd1028 { |
| 646 | id -55 # do not change unnecessarily |
| 647 | id -56 class ssd # do not change unnecessarily |
| 648 | # weight 13.973 |
| 649 | alg straw2 |
| 650 | hash 0 # rjenkins1 |
| 651 | item osd.215 weight 1.747 |
| 652 | item osd.216 weight 1.747 |
| 653 | item osd.217 weight 1.747 |
| 654 | item osd.218 weight 1.747 |
| 655 | item osd.219 weight 1.747 |
| 656 | item osd.220 weight 1.747 |
| 657 | item osd.221 weight 1.747 |
| 658 | item osd.222 weight 1.747 |
| 659 | } |
| 660 | host cloudcephosd1029 { |
| 661 | id -57 # do not change unnecessarily |
| 662 | id -58 class ssd # do not change unnecessarily |
| 663 | # weight 13.973 |
| 664 | alg straw2 |
| 665 | hash 0 # rjenkins1 |
| 666 | item osd.223 weight 1.747 |
| 667 | item osd.224 weight 1.747 |
| 668 | item osd.225 weight 1.747 |
| 669 | item osd.226 weight 1.747 |
| 670 | item osd.227 weight 1.747 |
| 671 | item osd.228 weight 1.747 |
| 672 | item osd.229 weight 1.747 |
| 673 | item osd.230 weight 1.747 |
| 674 | } |
| 675 | host cloudcephosd1030 { |
| 676 | id -59 # do not change unnecessarily |
| 677 | id -60 class ssd # do not change unnecessarily |
| 678 | # weight 13.973 |
| 679 | alg straw2 |
| 680 | hash 0 # rjenkins1 |
| 681 | item osd.239 weight 1.747 |
| 682 | item osd.240 weight 1.747 |
| 683 | item osd.241 weight 1.747 |
| 684 | item osd.242 weight 1.747 |
| 685 | item osd.243 weight 1.747 |
| 686 | item osd.244 weight 1.747 |
| 687 | item osd.245 weight 1.747 |
| 688 | item osd.246 weight 1.747 |
| 689 | } |
| 690 | host cloudcephosd1032 { |
| 691 | id -61 # do not change unnecessarily |
| 692 | id -62 class ssd # do not change unnecessarily |
| 693 | # weight 13.973 |
| 694 | alg straw2 |
| 695 | hash 0 # rjenkins1 |
| 696 | item osd.231 weight 1.747 |
| 697 | item osd.232 weight 1.747 |
| 698 | item osd.233 weight 1.747 |
| 699 | item osd.234 weight 1.747 |
| 700 | item osd.235 weight 1.747 |
| 701 | item osd.236 weight 1.747 |
| 702 | item osd.237 weight 1.747 |
| 703 | item osd.238 weight 1.747 |
| 704 | } |
| 705 | host cloudcephosd1031 { |
| 706 | id -63 # do not change unnecessarily |
| 707 | id -64 class ssd # do not change unnecessarily |
| 708 | # weight 13.973 |
| 709 | alg straw2 |
| 710 | hash 0 # rjenkins1 |
| 711 | item osd.247 weight 1.747 |
| 712 | item osd.248 weight 1.747 |
| 713 | item osd.249 weight 1.747 |
| 714 | item osd.250 weight 1.747 |
| 715 | item osd.251 weight 1.747 |
| 716 | item osd.252 weight 1.747 |
| 717 | item osd.253 weight 1.747 |
| 718 | item osd.254 weight 1.747 |
| 719 | } |
| 720 | host cloudcephosd1033 { |
| 721 | id -65 # do not change unnecessarily |
| 722 | id -66 class ssd # do not change unnecessarily |
| 723 | # weight 13.973 |
| 724 | alg straw2 |
| 725 | hash 0 # rjenkins1 |
| 726 | item osd.255 weight 1.747 |
| 727 | item osd.256 weight 1.747 |
| 728 | item osd.257 weight 1.747 |
| 729 | item osd.258 weight 1.747 |
| 730 | item osd.259 weight 1.747 |
| 731 | item osd.260 weight 1.747 |
| 732 | item osd.261 weight 1.747 |
| 733 | item osd.262 weight 1.747 |
| 734 | } |
| 735 | host cloudcephosd1034 { |
| 736 | id -67 # do not change unnecessarily |
| 737 | id -68 class ssd # do not change unnecessarily |
| 738 | # weight 13.973 |
| 739 | alg straw2 |
| 740 | hash 0 # rjenkins1 |
| 741 | item osd.263 weight 1.747 |
| 742 | item osd.264 weight 1.747 |
| 743 | item osd.265 weight 1.747 |
| 744 | item osd.266 weight 1.747 |
| 745 | item osd.267 weight 1.747 |
| 746 | item osd.268 weight 1.747 |
| 747 | item osd.269 weight 1.747 |
| 748 | item osd.270 weight 1.747 |
| 749 | } |
| 750 | host cloudcephosd1003 { |
| 751 | id -7 # do not change unnecessarily |
| 752 | id -8 class ssd # do not change unnecessarily |
| 753 | # weight 13.973 |
| 754 | alg straw2 |
| 755 | hash 0 # rjenkins1 |
| 756 | item osd.24 weight 1.747 |
| 757 | item osd.25 weight 1.747 |
| 758 | item osd.26 weight 1.747 |
| 759 | item osd.27 weight 1.747 |
| 760 | item osd.28 weight 1.747 |
| 761 | item osd.29 weight 1.747 |
| 762 | item osd.30 weight 1.747 |
| 763 | item osd.31 weight 1.747 |
| 764 | } |
| 765 | host cloudcephosd1004 { |
| 766 | id -19 # do not change unnecessarily |
| 767 | id -20 class ssd # do not change unnecessarily |
| 768 | # weight 13.973 |
| 769 | alg straw2 |
| 770 | hash 0 # rjenkins1 |
| 771 | item osd.64 weight 1.747 |
| 772 | item osd.65 weight 1.747 |
| 773 | item osd.66 weight 1.747 |
| 774 | item osd.67 weight 1.747 |
| 775 | item osd.68 weight 1.747 |
| 776 | item osd.69 weight 1.747 |
| 777 | item osd.70 weight 1.747 |
| 778 | item osd.71 weight 1.747 |
| 779 | } |
| 780 | host cloudcephosd1005 { |
| 781 | id -16 # do not change unnecessarily |
| 782 | id -17 class ssd # do not change unnecessarily |
| 783 | # weight 13.973 |
| 784 | alg straw2 |
| 785 | hash 0 # rjenkins1 |
| 786 | item osd.32 weight 1.747 |
| 787 | item osd.33 weight 1.747 |
| 788 | item osd.34 weight 1.747 |
| 789 | item osd.35 weight 1.747 |
| 790 | item osd.36 weight 1.747 |
| 791 | item osd.37 weight 1.747 |
| 792 | item osd.38 weight 1.747 |
| 793 | item osd.39 weight 1.747 |
| 794 | } |
| 795 | host cloudcephosd1010 { |
| 796 | id -31 # do not change unnecessarily |
| 797 | id -32 class ssd # do not change unnecessarily |
| 798 | # weight 13.973 |
| 799 | alg straw2 |
| 800 | hash 0 # rjenkins1 |
| 801 | item osd.72 weight 1.747 |
| 802 | item osd.73 weight 1.747 |
| 803 | item osd.74 weight 1.747 |
| 804 | item osd.75 weight 1.747 |
| 805 | item osd.76 weight 1.747 |
| 806 | item osd.77 weight 1.747 |
| 807 | item osd.78 weight 1.747 |
| 808 | item osd.79 weight 1.747 |
| 809 | } |
| 810 | rack E4 { |
| 811 | id -77 |
| 812 | id -78 class ssd |
| 813 | alg straw2 |
| 814 | hash 0 # rjenkins1 |
| 815 | item cloudcephosd1001 weight 13.973 |
| 816 | item cloudcephosd1002 weight 13.973 |
| 817 | item cloudcephosd1003 weight 13.973 |
| 818 | item cloudcephosd1025 weight 13.973 |
| 819 | item cloudcephosd1026 weight 13.973 |
| 820 | item cloudcephosd1027 weight 13.973 |
| 821 | item cloudcephosd1028 weight 13.973 |
| 822 | item cloudcephosd1029 weight 13.973 |
| 823 | } |
| 824 | rack F4 { |
| 825 | id -79 |
| 826 | id -80 class ssd |
| 827 | alg straw2 |
| 828 | hash 0 # rjenkins1 |
| 829 | item cloudcephosd1004 weight 13.973 |
| 830 | item cloudcephosd1005 weight 13.973 |
| 831 | item cloudcephosd1010 weight 13.973 |
| 832 | item cloudcephosd1030 weight 13.973 |
| 833 | item cloudcephosd1031 weight 13.973 |
| 834 | item cloudcephosd1032 weight 13.973 |
| 835 | item cloudcephosd1033 weight 13.973 |
| 836 | item cloudcephosd1034 weight 13.973 |
| 837 | } |
| 838 | rack D5 { |
| 839 | id -81 |
| 840 | id -82 class ssd |
| 841 | alg straw2 |
| 842 | hash 0 # rjenkins1 |
| 843 | item cloudcephosd1011 weight 13.969 |
| 844 | item cloudcephosd1012 weight 13.969 |
| 845 | item cloudcephosd1013 weight 13.969 |
| 846 | item cloudcephosd1014 weight 13.969 |
| 847 | item cloudcephosd1015 weight 13.969 |
| 848 | item cloudcephosd1019 weight 13.973 |
| 849 | item cloudcephosd1020 weight 13.973 |
| 850 | item cloudcephosd1023 weight 13.973 |
| 851 | item cloudcephosd1024 weight 13.973 |
| 852 | } |
| 853 | rack C8 { |
| 854 | id -83 |
| 855 | id -84 class ssd |
| 856 | alg straw2 |
| 857 | hash 0 # rjenkins1 |
| 858 | item cloudcephosd1006 weight 13.969 |
| 859 | item cloudcephosd1007 weight 13.969 |
| 860 | item cloudcephosd1008 weight 13.969 |
| 861 | item cloudcephosd1009 weight 13.969 |
| 862 | item cloudcephosd1016 weight 13.973 |
| 863 | item cloudcephosd1017 weight 13.973 |
| 864 | item cloudcephosd1018 weight 13.973 |
| 865 | item cloudcephosd1021 weight 13.973 |
| 866 | item cloudcephosd1022 weight 13.973 |
| 867 | } |
| 868 | root default { |
| 869 | id -1 # do not change unnecessarily |
| 870 | id -2 class ssd # do not change unnecessarily |
| 871 | # weight 475.038 |
| 872 | alg straw2 |
| 873 | hash 0 # rjenkins1 |
| 874 | item E4 weight 111.784 |
| 875 | item F4 weight 111.784 |
| 876 | item C8 weight 125.737 |
| 877 | item D5 weight 125.741 |
| 878 | } |
| 879 | |
| 880 | # rules |
| 881 | rule replicated_rule { |
| 882 | id 0 |
| 883 | type replicated |
| 884 | min_size 1 |
| 885 | max_size 10 |
| 886 | step take default |
| 887 | step chooseleaf firstn 0 type rack |
| 888 | step emit |
| 889 | } |
| 890 | |
| 891 | # end crush map |
Test results for the above crushmap:
| 1 | rule 0 (replicated_rule), x = 0..1023, numrep = 3..3 |
|---|---|
| 2 | rule 0 (replicated_rule) num_rep 3 result size == 3: 1024/1024 |
| 3 | device 0: stored : 14 expected : 11.2941 |
| 4 | device 1: stored : 11 expected : 11.2941 |
| 5 | device 2: stored : 12 expected : 11.2941 |
| 6 | device 3: stored : 10 expected : 11.2941 |
| 7 | device 4: stored : 10 expected : 11.2941 |
| 8 | device 5: stored : 10 expected : 11.2941 |
| 9 | device 6: stored : 14 expected : 11.2941 |
| 10 | device 7: stored : 10 expected : 11.2941 |
| 11 | device 8: stored : 12 expected : 11.2941 |
| 12 | device 9: stored : 7 expected : 11.2941 |
| 13 | device 10: stored : 9 expected : 11.2941 |
| 14 | device 11: stored : 13 expected : 11.2941 |
| 15 | device 12: stored : 11 expected : 11.2941 |
| 16 | device 13: stored : 16 expected : 11.2941 |
| 17 | device 14: stored : 13 expected : 11.2941 |
| 18 | device 15: stored : 11 expected : 11.2941 |
| 19 | device 16: stored : 9 expected : 11.2941 |
| 20 | device 17: stored : 14 expected : 11.2941 |
| 21 | device 18: stored : 17 expected : 11.2941 |
| 22 | device 19: stored : 11 expected : 11.2941 |
| 23 | device 20: stored : 9 expected : 11.2941 |
| 24 | device 21: stored : 10 expected : 11.2941 |
| 25 | device 22: stored : 9 expected : 11.2941 |
| 26 | device 23: stored : 5 expected : 11.2941 |
| 27 | device 24: stored : 13 expected : 11.2941 |
| 28 | device 25: stored : 9 expected : 11.2941 |
| 29 | device 26: stored : 17 expected : 11.2941 |
| 30 | device 27: stored : 19 expected : 11.2941 |
| 31 | device 28: stored : 14 expected : 11.2941 |
| 32 | device 29: stored : 10 expected : 11.2941 |
| 33 | device 30: stored : 16 expected : 11.2941 |
| 34 | device 31: stored : 17 expected : 11.2941 |
| 35 | device 32: stored : 16 expected : 11.2941 |
| 36 | device 33: stored : 13 expected : 11.2941 |
| 37 | device 34: stored : 7 expected : 11.2941 |
| 38 | device 35: stored : 12 expected : 11.2941 |
| 39 | device 36: stored : 15 expected : 11.2941 |
| 40 | device 37: stored : 11 expected : 11.2941 |
| 41 | device 38: stored : 12 expected : 11.2941 |
| 42 | device 39: stored : 9 expected : 11.2941 |
| 43 | device 40: stored : 12 expected : 11.2941 |
| 44 | device 41: stored : 11 expected : 11.2941 |
| 45 | device 42: stored : 13 expected : 11.2941 |
| 46 | device 43: stored : 9 expected : 11.2941 |
| 47 | device 44: stored : 7 expected : 11.2941 |
| 48 | device 45: stored : 8 expected : 11.2941 |
| 49 | device 46: stored : 8 expected : 11.2941 |
| 50 | device 47: stored : 9 expected : 11.2941 |
| 51 | device 48: stored : 14 expected : 11.2941 |
| 52 | device 49: stored : 17 expected : 11.2941 |
| 53 | device 50: stored : 17 expected : 11.2941 |
| 54 | device 51: stored : 14 expected : 11.2941 |
| 55 | device 52: stored : 11 expected : 11.2941 |
| 56 | device 53: stored : 14 expected : 11.2941 |
| 57 | device 54: stored : 17 expected : 11.2941 |
| 58 | device 55: stored : 9 expected : 11.2941 |
| 59 | device 56: stored : 11 expected : 11.2941 |
| 60 | device 57: stored : 6 expected : 11.2941 |
| 61 | device 58: stored : 15 expected : 11.2941 |
| 62 | device 59: stored : 11 expected : 11.2941 |
| 63 | device 60: stored : 12 expected : 11.2941 |
| 64 | device 61: stored : 12 expected : 11.2941 |
| 65 | device 62: stored : 8 expected : 11.2941 |
| 66 | device 63: stored : 15 expected : 11.2941 |
| 67 | device 64: stored : 20 expected : 11.2941 |
| 68 | device 65: stored : 10 expected : 11.2941 |
| 69 | device 66: stored : 7 expected : 11.2941 |
| 70 | device 67: stored : 7 expected : 11.2941 |
| 71 | device 68: stored : 17 expected : 11.2941 |
| 72 | device 69: stored : 13 expected : 11.2941 |
| 73 | device 70: stored : 9 expected : 11.2941 |
| 74 | device 71: stored : 13 expected : 11.2941 |
| 75 | device 72: stored : 10 expected : 11.2941 |
| 76 | device 73: stored : 9 expected : 11.2941 |
| 77 | device 74: stored : 15 expected : 11.2941 |
| 78 | device 75: stored : 12 expected : 11.2941 |
| 79 | device 76: stored : 17 expected : 11.2941 |
| 80 | device 77: stored : 10 expected : 11.2941 |
| 81 | device 78: stored : 14 expected : 11.2941 |
| 82 | device 79: stored : 14 expected : 11.2941 |
| 83 | device 80: stored : 15 expected : 11.2941 |
| 84 | device 81: stored : 9 expected : 11.2941 |
| 85 | device 82: stored : 12 expected : 11.2941 |
| 86 | device 83: stored : 8 expected : 11.2941 |
| 87 | device 84: stored : 12 expected : 11.2941 |
| 88 | device 85: stored : 14 expected : 11.2941 |
| 89 | device 86: stored : 8 expected : 11.2941 |
| 90 | device 87: stored : 15 expected : 11.2941 |
| 91 | device 88: stored : 8 expected : 11.2941 |
| 92 | device 89: stored : 15 expected : 11.2941 |
| 93 | device 90: stored : 14 expected : 11.2941 |
| 94 | device 91: stored : 9 expected : 11.2941 |
| 95 | device 92: stored : 8 expected : 11.2941 |
| 96 | device 93: stored : 8 expected : 11.2941 |
| 97 | device 94: stored : 11 expected : 11.2941 |
| 98 | device 95: stored : 14 expected : 11.2941 |
| 99 | device 96: stored : 10 expected : 11.2941 |
| 100 | device 97: stored : 12 expected : 11.2941 |
| 101 | device 98: stored : 10 expected : 11.2941 |
| 102 | device 99: stored : 8 expected : 11.2941 |
| 103 | device 100: stored : 8 expected : 11.2941 |
| 104 | device 101: stored : 11 expected : 11.2941 |
| 105 | device 102: stored : 9 expected : 11.2941 |
| 106 | device 103: stored : 5 expected : 11.2941 |
| 107 | device 104: stored : 5 expected : 11.2941 |
| 108 | device 105: stored : 9 expected : 11.2941 |
| 109 | device 106: stored : 16 expected : 11.2941 |
| 110 | device 107: stored : 9 expected : 11.2941 |
| 111 | device 108: stored : 13 expected : 11.2941 |
| 112 | device 109: stored : 13 expected : 11.2941 |
| 113 | device 110: stored : 9 expected : 11.2941 |
| 114 | device 111: stored : 12 expected : 11.2941 |
| 115 | device 112: stored : 9 expected : 11.2941 |
| 116 | device 113: stored : 14 expected : 11.2941 |
| 117 | device 114: stored : 10 expected : 11.2941 |
| 118 | device 115: stored : 12 expected : 11.2941 |
| 119 | device 116: stored : 8 expected : 11.2941 |
| 120 | device 117: stored : 13 expected : 11.2941 |
| 121 | device 118: stored : 10 expected : 11.2941 |
| 122 | device 119: stored : 10 expected : 11.2941 |
| 123 | device 120: stored : 9 expected : 11.2941 |
| 124 | device 121: stored : 18 expected : 11.2941 |
| 125 | device 122: stored : 7 expected : 11.2941 |
| 126 | device 123: stored : 11 expected : 11.2941 |
| 127 | device 124: stored : 12 expected : 11.2941 |
| 128 | device 125: stored : 9 expected : 11.2941 |
| 129 | device 126: stored : 6 expected : 11.2941 |
| 130 | device 127: stored : 11 expected : 11.2941 |
| 131 | device 128: stored : 12 expected : 11.2941 |
| 132 | device 129: stored : 12 expected : 11.2941 |
| 133 | device 130: stored : 14 expected : 11.2941 |
| 134 | device 131: stored : 16 expected : 11.2941 |
| 135 | device 132: stored : 13 expected : 11.2941 |
| 136 | device 133: stored : 12 expected : 11.2941 |
| 137 | device 134: stored : 7 expected : 11.2941 |
| 138 | device 135: stored : 9 expected : 11.2941 |
| 139 | device 136: stored : 8 expected : 11.2941 |
| 140 | device 137: stored : 9 expected : 11.2941 |
| 141 | device 138: stored : 12 expected : 11.2941 |
| 142 | device 139: stored : 11 expected : 11.2941 |
| 143 | device 140: stored : 7 expected : 11.2941 |
| 144 | device 141: stored : 12 expected : 11.2941 |
| 145 | device 142: stored : 9 expected : 11.2941 |
| 146 | device 143: stored : 11 expected : 11.2941 |
| 147 | device 144: stored : 6 expected : 11.2941 |
| 148 | device 145: stored : 10 expected : 11.2941 |
| 149 | device 146: stored : 10 expected : 11.2941 |
| 150 | device 147: stored : 9 expected : 11.2941 |
| 151 | device 148: stored : 11 expected : 11.2941 |
| 152 | device 149: stored : 16 expected : 11.2941 |
| 153 | device 150: stored : 9 expected : 11.2941 |
| 154 | device 151: stored : 17 expected : 11.2941 |
| 155 | device 152: stored : 11 expected : 11.2941 |
| 156 | device 153: stored : 16 expected : 11.2941 |
| 157 | device 154: stored : 7 expected : 11.2941 |
| 158 | device 155: stored : 14 expected : 11.2941 |
| 159 | device 156: stored : 11 expected : 11.2941 |
| 160 | device 157: stored : 13 expected : 11.2941 |
| 161 | device 158: stored : 9 expected : 11.2941 |
| 162 | device 159: stored : 14 expected : 11.2941 |
| 163 | device 160: stored : 7 expected : 11.2941 |
| 164 | device 161: stored : 14 expected : 11.2941 |
| 165 | device 162: stored : 6 expected : 11.2941 |
| 166 | device 163: stored : 9 expected : 11.2941 |
| 167 | device 164: stored : 14 expected : 11.2941 |
| 168 | device 165: stored : 13 expected : 11.2941 |
| 169 | device 166: stored : 8 expected : 11.2941 |
| 170 | device 167: stored : 13 expected : 11.2941 |
| 171 | device 168: stored : 16 expected : 11.2941 |
| 172 | device 169: stored : 15 expected : 11.2941 |
| 173 | device 170: stored : 4 expected : 11.2941 |
| 174 | device 171: stored : 13 expected : 11.2941 |
| 175 | device 172: stored : 9 expected : 11.2941 |
| 176 | device 173: stored : 2 expected : 11.2941 |
| 177 | device 174: stored : 10 expected : 11.2941 |
| 178 | device 175: stored : 12 expected : 11.2941 |
| 179 | device 176: stored : 15 expected : 11.2941 |
| 180 | device 177: stored : 17 expected : 11.2941 |
| 181 | device 178: stored : 7 expected : 11.2941 |
| 182 | device 179: stored : 11 expected : 11.2941 |
| 183 | device 180: stored : 14 expected : 11.2941 |
| 184 | device 181: stored : 9 expected : 11.2941 |
| 185 | device 182: stored : 14 expected : 11.2941 |
| 186 | device 183: stored : 13 expected : 11.2941 |
| 187 | device 184: stored : 12 expected : 11.2941 |
| 188 | device 185: stored : 9 expected : 11.2941 |
| 189 | device 186: stored : 9 expected : 11.2941 |
| 190 | device 187: stored : 15 expected : 11.2941 |
| 191 | device 188: stored : 11 expected : 11.2941 |
| 192 | device 189: stored : 8 expected : 11.2941 |
| 193 | device 190: stored : 13 expected : 11.2941 |
| 194 | device 191: stored : 10 expected : 11.2941 |
| 195 | device 192: stored : 11 expected : 11.2941 |
| 196 | device 193: stored : 9 expected : 11.2941 |
| 197 | device 194: stored : 9 expected : 11.2941 |
| 198 | device 195: stored : 10 expected : 11.2941 |
| 199 | device 196: stored : 6 expected : 11.2941 |
| 200 | device 197: stored : 7 expected : 11.2941 |
| 201 | device 198: stored : 10 expected : 11.2941 |
| 202 | device 199: stored : 5 expected : 11.2941 |
| 203 | device 200: stored : 10 expected : 11.2941 |
| 204 | device 201: stored : 10 expected : 11.2941 |
| 205 | device 202: stored : 14 expected : 11.2941 |
| 206 | device 203: stored : 11 expected : 11.2941 |
| 207 | device 204: stored : 13 expected : 11.2941 |
| 208 | device 205: stored : 14 expected : 11.2941 |
| 209 | device 206: stored : 11 expected : 11.2941 |
| 210 | device 207: stored : 21 expected : 11.2941 |
| 211 | device 208: stored : 6 expected : 11.2941 |
| 212 | device 209: stored : 10 expected : 11.2941 |
| 213 | device 210: stored : 10 expected : 11.2941 |
| 214 | device 211: stored : 12 expected : 11.2941 |
| 215 | device 212: stored : 19 expected : 11.2941 |
| 216 | device 213: stored : 8 expected : 11.2941 |
| 217 | device 214: stored : 9 expected : 11.2941 |
| 218 | device 215: stored : 17 expected : 11.2941 |
| 219 | device 216: stored : 11 expected : 11.2941 |
| 220 | device 217: stored : 9 expected : 11.2941 |
| 221 | device 218: stored : 11 expected : 11.2941 |
| 222 | device 219: stored : 12 expected : 11.2941 |
| 223 | device 220: stored : 10 expected : 11.2941 |
| 224 | device 221: stored : 14 expected : 11.2941 |
| 225 | device 222: stored : 6 expected : 11.2941 |
| 226 | device 223: stored : 7 expected : 11.2941 |
| 227 | device 224: stored : 13 expected : 11.2941 |
| 228 | device 225: stored : 8 expected : 11.2941 |
| 229 | device 226: stored : 18 expected : 11.2941 |
| 230 | device 227: stored : 18 expected : 11.2941 |
| 231 | device 228: stored : 6 expected : 11.2941 |
| 232 | device 229: stored : 17 expected : 11.2941 |
| 233 | device 230: stored : 10 expected : 11.2941 |
| 234 | device 231: stored : 16 expected : 11.2941 |
| 235 | device 232: stored : 13 expected : 11.2941 |
| 236 | device 233: stored : 9 expected : 11.2941 |
| 237 | device 234: stored : 4 expected : 11.2941 |
| 238 | device 235: stored : 15 expected : 11.2941 |
| 239 | device 236: stored : 12 expected : 11.2941 |
| 240 | device 237: stored : 11 expected : 11.2941 |
| 241 | device 238: stored : 14 expected : 11.2941 |
| 242 | device 239: stored : 9 expected : 11.2941 |
| 243 | device 240: stored : 9 expected : 11.2941 |
| 244 | device 241: stored : 17 expected : 11.2941 |
| 245 | device 242: stored : 14 expected : 11.2941 |
| 246 | device 243: stored : 11 expected : 11.2941 |
| 247 | device 244: stored : 11 expected : 11.2941 |
| 248 | device 245: stored : 12 expected : 11.2941 |
| 249 | device 246: stored : 14 expected : 11.2941 |
| 250 | device 247: stored : 11 expected : 11.2941 |
| 251 | device 248: stored : 8 expected : 11.2941 |
| 252 | device 249: stored : 12 expected : 11.2941 |
| 253 | device 250: stored : 11 expected : 11.2941 |
| 254 | device 251: stored : 9 expected : 11.2941 |
| 255 | device 252: stored : 14 expected : 11.2941 |
| 256 | device 253: stored : 18 expected : 11.2941 |
| 257 | device 254: stored : 16 expected : 11.2941 |
| 258 | device 255: stored : 12 expected : 11.2941 |
| 259 | device 256: stored : 15 expected : 11.2941 |
| 260 | device 257: stored : 10 expected : 11.2941 |
| 261 | device 258: stored : 8 expected : 11.2941 |
| 262 | device 259: stored : 10 expected : 11.2941 |
| 263 | device 260: stored : 14 expected : 11.2941 |
| 264 | device 261: stored : 9 expected : 11.2941 |
| 265 | device 262: stored : 12 expected : 11.2941 |
| 266 | device 263: stored : 14 expected : 11.2941 |
| 267 | device 264: stored : 13 expected : 11.2941 |
| 268 | device 265: stored : 13 expected : 11.2941 |
| 269 | device 266: stored : 7 expected : 11.2941 |
| 270 | device 267: stored : 13 expected : 11.2941 |
| 271 | device 268: stored : 5 expected : 11.2941 |
| 272 | device 269: stored : 9 expected : 11.2941 |
| 273 | device 270: stored : 6 expected : 11.2941 |
| 274 | device 271: stored : 9 expected : 11.2941 |
- Compile it:
crushtool -c new_crushmap.txt -o crushmap.bin
- Test that the rules still work well (check that there's no misplaced pgs, that is shows 1024/1024, and that there's more or less the same placements on each device, current output example P44923):
crushtool --test -i crushmap.bin --show-utilization --num-rep=3
- Load the new crush map and wait for the cluster to shift data around (will take a looong time)
ceph osd setcrushmap -i crushmap.bin
Tests
Followed the above procedure (adapted for codfw) to setup a "fake" rack ha there, with three virtual racks and one node on each rack. The cluster started rebalancing (shows as repair, and misplaced pgs) without loss of service or redundancy.
It took in the order of 3h hours to rebalance 6 osds