Page MenuHomePhabricator
Create Task
Maniphest T348643

cloudcephosd1021-1034: hard drive sector errors increasing
Open, In Progress, HighPublic
Actions

Description

We have noticed that 14 of our ceph hosts are having sector errors on all the hard drives:

P52904 Masterwork From Distant Lands
1cloudcephosd1021 /dev/sdc
2cloudcephosd1021 /dev/sdd
3cloudcephosd1021 /dev/sde
4cloudcephosd1021 /dev/sdf
5cloudcephosd1021 /dev/sdg
6cloudcephosd1021 /dev/sdh
7cloudcephosd1021 /dev/sdi
8cloudcephosd1021 /dev/sdj
9cloudcephosd1022 /dev/sdc
10cloudcephosd1022 /dev/sdd
11cloudcephosd1022 /dev/sde
12cloudcephosd1022 /dev/sdf
13cloudcephosd1022 /dev/sdg
14cloudcephosd1022 /dev/sdh
15cloudcephosd1022 /dev/sdi
16cloudcephosd1022 /dev/sdj
17cloudcephosd1023 /dev/sdc
18cloudcephosd1023 /dev/sdd
19cloudcephosd1023 /dev/sde
20cloudcephosd1023 /dev/sdf
21cloudcephosd1023 /dev/sdg
22cloudcephosd1023 /dev/sdh
23cloudcephosd1023 /dev/sdi
24cloudcephosd1023 /dev/sdj
25cloudcephosd1024 /dev/sdc
26cloudcephosd1024 /dev/sdd
27cloudcephosd1024 /dev/sde
28cloudcephosd1024 /dev/sdf
29cloudcephosd1024 /dev/sdg
30cloudcephosd1024 /dev/sdh
31cloudcephosd1024 /dev/sdi
32cloudcephosd1024 /dev/sdj
33cloudcephosd1025 /dev/sdb
34cloudcephosd1025 /dev/sdd
35cloudcephosd1025 /dev/sde
36cloudcephosd1025 /dev/sdf
37cloudcephosd1025 /dev/sdg
38cloudcephosd1025 /dev/sdh
39cloudcephosd1025 /dev/sdi
40cloudcephosd1025 /dev/sdj
41cloudcephosd1026 /dev/sdc
42cloudcephosd1026 /dev/sdd
43cloudcephosd1026 /dev/sde
44cloudcephosd1026 /dev/sdf
45cloudcephosd1026 /dev/sdg
46cloudcephosd1026 /dev/sdh
47cloudcephosd1026 /dev/sdi
48cloudcephosd1026 /dev/sdj
49cloudcephosd1027 /dev/sdc
50cloudcephosd1027 /dev/sdd
51cloudcephosd1027 /dev/sde
52cloudcephosd1027 /dev/sdf
53cloudcephosd1027 /dev/sdg
54cloudcephosd1027 /dev/sdh
55cloudcephosd1027 /dev/sdi
56cloudcephosd1027 /dev/sdj
57cloudcephosd1028 /dev/sdc
58cloudcephosd1028 /dev/sdd
59cloudcephosd1028 /dev/sde
60cloudcephosd1028 /dev/sdf
61cloudcephosd1028 /dev/sdg
62cloudcephosd1028 /dev/sdh
63cloudcephosd1028 /dev/sdi
64cloudcephosd1028 /dev/sdj
65cloudcephosd1029 /dev/sdc
66cloudcephosd1029 /dev/sdd
67cloudcephosd1029 /dev/sde
68cloudcephosd1029 /dev/sdf
69cloudcephosd1029 /dev/sdg
70cloudcephosd1029 /dev/sdh
71cloudcephosd1029 /dev/sdi
72cloudcephosd1029 /dev/sdj
73cloudcephosd1030 /dev/sdc
74cloudcephosd1030 /dev/sdd
75cloudcephosd1030 /dev/sde
76cloudcephosd1030 /dev/sdf
77cloudcephosd1030 /dev/sdg
78cloudcephosd1030 /dev/sdh
79cloudcephosd1030 /dev/sdi
80cloudcephosd1030 /dev/sdj
81cloudcephosd1031 /dev/sdc
82cloudcephosd1031 /dev/sdd
83cloudcephosd1031 /dev/sde
84cloudcephosd1031 /dev/sdf
85cloudcephosd1031 /dev/sdg
86cloudcephosd1031 /dev/sdh
87cloudcephosd1031 /dev/sdi
88cloudcephosd1031 /dev/sdj
89cloudcephosd1032 /dev/sdc
90cloudcephosd1032 /dev/sdd
91cloudcephosd1032 /dev/sde
92cloudcephosd1032 /dev/sdf
93cloudcephosd1032 /dev/sdg
94cloudcephosd1032 /dev/sdh
95cloudcephosd1032 /dev/sdi
96cloudcephosd1032 /dev/sdj
97cloudcephosd1033 /dev/sdb
98cloudcephosd1033 /dev/sdd
99cloudcephosd1033 /dev/sde
100cloudcephosd1033 /dev/sdf
101cloudcephosd1033 /dev/sdg
102cloudcephosd1033 /dev/sdh
103cloudcephosd1033 /dev/sdi
104cloudcephosd1033 /dev/sdj
105cloudcephosd1034 /dev/sdc
106cloudcephosd1034 /dev/sdd
107cloudcephosd1034 /dev/sde
108cloudcephosd1034 /dev/sdf
109cloudcephosd1034 /dev/sdg
110cloudcephosd1034 /dev/sdi
111cloudcephosd1034 /dev/sdj

This errors are increasing over time, so it seems the drives are degrading (see the changed lines):

P52903 Masterwork From Distant Lands
1===== NODE GROUP =====
2(1) cloudcephosd1030.eqiad.wmnet
3----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
4Oct 01 01:45:02 cloudcephosd1030 smart_failure[1384923]: Device: /dev/sdd [SAT], 712 Offline uncorrectable sectors
5Oct 01 02:15:02 cloudcephosd1030 smart_failure[1396279]: Device: /dev/sdj [SAT], 1168 Offline uncorrectable sectors
6Oct 01 02:45:02 cloudcephosd1030 smart_failure[1407428]: Device: /dev/sde [SAT], 1936 Offline uncorrectable sectors
7Oct 01 04:45:02 cloudcephosd1030 smart_failure[1452484]: Device: /dev/sdf [SAT], 2904 Offline uncorrectable sectors
8Oct 01 10:15:02 cloudcephosd1030 smart_failure[1576410]: Device: /dev/sdh [SAT], 2600 Offline uncorrectable sectors
9Oct 01 13:15:02 cloudcephosd1030 smart_failure[1644186]: Device: /dev/sdc [SAT], 2264 Offline uncorrectable sectors (changed +8)
10Oct 01 13:15:02 cloudcephosd1030 smart_failure[1644205]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
11Oct 01 16:15:02 cloudcephosd1030 smart_failure[1711798]: Device: /dev/sdg [SAT], 1392 Offline uncorrectable sectors
12Oct 02 01:45:02 cloudcephosd1030 smart_failure[1924241]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
13Oct 02 02:15:02 cloudcephosd1030 smart_failure[1935840]: Device: /dev/sdj [SAT], 1176 Offline uncorrectable sectors
14Oct 02 02:45:02 cloudcephosd1030 smart_failure[1946947]: Device: /dev/sde [SAT], 1936 Offline uncorrectable sectors
15Oct 02 04:45:02 cloudcephosd1030 smart_failure[1992109]: Device: /dev/sdf [SAT], 2912 Offline uncorrectable sectors
16Oct 02 10:15:02 cloudcephosd1030 smart_failure[2115614]: Device: /dev/sdh [SAT], 2600 Offline uncorrectable sectors
17Oct 02 13:15:02 cloudcephosd1030 smart_failure[2182941]: Device: /dev/sdc [SAT], 2264 Offline uncorrectable sectors
18Oct 02 13:15:02 cloudcephosd1030 smart_failure[2182960]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
19Oct 02 16:15:02 cloudcephosd1030 smart_failure[2250370]: Device: /dev/sdg [SAT], 1400 Offline uncorrectable sectors
20Oct 03 01:45:02 cloudcephosd1030 smart_failure[2455412]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
21Oct 03 02:15:02 cloudcephosd1030 smart_failure[2466805]: Device: /dev/sdj [SAT], 1176 Offline uncorrectable sectors
22Oct 03 02:45:02 cloudcephosd1030 smart_failure[2478029]: Device: /dev/sde [SAT], 1944 Offline uncorrectable sectors
23Oct 03 04:45:02 cloudcephosd1030 smart_failure[2523016]: Device: /dev/sdf [SAT], 2912 Offline uncorrectable sectors
24Oct 03 10:15:02 cloudcephosd1030 smart_failure[2646781]: Device: /dev/sdh [SAT], 2600 Offline uncorrectable sectors
25Oct 03 13:15:02 cloudcephosd1030 smart_failure[2715327]: Device: /dev/sdc [SAT], 2264 Offline uncorrectable sectors
26Oct 03 13:15:02 cloudcephosd1030 smart_failure[2715346]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
27Oct 03 16:15:02 cloudcephosd1030 smart_failure[2782722]: Device: /dev/sdg [SAT], 1408 Offline uncorrectable sectors
28Oct 04 01:45:02 cloudcephosd1030 smart_failure[2996161]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
29Oct 04 02:15:02 cloudcephosd1030 smart_failure[3007509]: Device: /dev/sdj [SAT], 1208 Offline uncorrectable sectors
30Oct 04 02:45:02 cloudcephosd1030 smart_failure[3018755]: Device: /dev/sde [SAT], 1944 Offline uncorrectable sectors
31Oct 04 04:45:02 cloudcephosd1030 smart_failure[3063745]: Device: /dev/sdf [SAT], 2944 Offline uncorrectable sectors
32Oct 04 10:15:02 cloudcephosd1030 smart_failure[3188825]: Device: /dev/sdh [SAT], 2600 Offline uncorrectable sectors
33Oct 04 13:15:02 cloudcephosd1030 smart_failure[3256447]: Device: /dev/sdc [SAT], 2264 Offline uncorrectable sectors
34Oct 04 13:15:02 cloudcephosd1030 smart_failure[3256466]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
35Oct 04 16:15:02 cloudcephosd1030 smart_failure[3324555]: Device: /dev/sdg [SAT], 1408 Offline uncorrectable sectors
36Oct 05 01:45:02 cloudcephosd1030 smart_failure[3538912]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
37Oct 05 02:15:02 cloudcephosd1030 smart_failure[3550065]: Device: /dev/sdj [SAT], 1208 Offline uncorrectable sectors
38Oct 05 02:45:02 cloudcephosd1030 smart_failure[3561206]: Device: /dev/sde [SAT], 1944 Offline uncorrectable sectors
39Oct 05 04:45:02 cloudcephosd1030 smart_failure[3605969]: Device: /dev/sdf [SAT], 2976 Offline uncorrectable sectors
40Oct 05 10:15:02 cloudcephosd1030 smart_failure[3730002]: Device: /dev/sdh [SAT], 2608 Offline uncorrectable sectors
41Oct 05 13:15:02 cloudcephosd1030 smart_failure[3809622]: Device: /dev/sdc [SAT], 2264 Offline uncorrectable sectors
42Oct 05 13:15:02 cloudcephosd1030 smart_failure[3809641]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
43Oct 05 16:15:02 cloudcephosd1030 smart_failure[3877380]: Device: /dev/sdg [SAT], 1424 Offline uncorrectable sectors
44Oct 06 01:45:02 cloudcephosd1030 smart_failure[4091308]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
45Oct 06 02:15:02 cloudcephosd1030 smart_failure[4102610]: Device: /dev/sdj [SAT], 1208 Offline uncorrectable sectors
46Oct 06 02:45:02 cloudcephosd1030 smart_failure[4113861]: Device: /dev/sde [SAT], 1944 Offline uncorrectable sectors
47Oct 06 04:45:02 cloudcephosd1030 smart_failure[4159497]: Device: /dev/sdf [SAT], 2976 Offline uncorrectable sectors
48Oct 06 10:15:02 cloudcephosd1030 smart_failure[90096]: Device: /dev/sdh [SAT], 2608 Offline uncorrectable sectors
49Oct 06 13:15:02 cloudcephosd1030 smart_failure[158069]: Device: /dev/sdc [SAT], 2272 Offline uncorrectable sectors
50Oct 06 13:15:02 cloudcephosd1030 smart_failure[158088]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
51Oct 06 16:15:02 cloudcephosd1030 smart_failure[226246]: Device: /dev/sdg [SAT], 1424 Offline uncorrectable sectors
52Oct 07 01:45:02 cloudcephosd1030 smart_failure[440224]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
53Oct 07 02:15:02 cloudcephosd1030 smart_failure[451415]: Device: /dev/sdj [SAT], 1216 Offline uncorrectable sectors
54Oct 07 02:45:02 cloudcephosd1030 smart_failure[462623]: Device: /dev/sde [SAT], 1952 Offline uncorrectable sectors
55Oct 07 04:45:03 cloudcephosd1030 smart_failure[507605]: Device: /dev/sdf [SAT], 2984 Offline uncorrectable sectors
56Oct 07 10:15:02 cloudcephosd1030 smart_failure[631776]: Device: /dev/sdh [SAT], 2616 Offline uncorrectable sectors
57Oct 07 13:15:02 cloudcephosd1030 smart_failure[699449]: Device: /dev/sdc [SAT], 2272 Offline uncorrectable sectors
58Oct 07 13:15:02 cloudcephosd1030 smart_failure[699469]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
59Oct 07 16:15:02 cloudcephosd1030 smart_failure[766980]: Device: /dev/sdg [SAT], 1424 Offline uncorrectable sectors
60Oct 08 01:45:02 cloudcephosd1030 smart_failure[981267]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
61Oct 08 02:15:02 cloudcephosd1030 smart_failure[992579]: Device: /dev/sdj [SAT], 1216 Offline uncorrectable sectors
62Oct 08 02:45:02 cloudcephosd1030 smart_failure[1004041]: Device: /dev/sde [SAT], 1952 Offline uncorrectable sectors
63Oct 08 05:15:02 cloudcephosd1030 smart_failure[1060303]: Device: /dev/sdf [SAT], 2984 Offline uncorrectable sectors
64Oct 08 10:15:02 cloudcephosd1030 smart_failure[1173008]: Device: /dev/sdh [SAT], 2616 Offline uncorrectable sectors
65Oct 08 13:15:03 cloudcephosd1030 smart_failure[1240338]: Device: /dev/sdc [SAT], 2272 Offline uncorrectable sectors
66Oct 08 13:15:03 cloudcephosd1030 smart_failure[1240357]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
67Oct 08 16:15:02 cloudcephosd1030 smart_failure[1308078]: Device: /dev/sdg [SAT], 1432 Offline uncorrectable sectors
68Oct 09 01:45:02 cloudcephosd1030 smart_failure[1522376]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
69Oct 09 02:15:02 cloudcephosd1030 smart_failure[1533713]: Device: /dev/sdj [SAT], 1224 Offline uncorrectable sectors
70Oct 09 02:45:02 cloudcephosd1030 smart_failure[1545051]: Device: /dev/sde [SAT], 1952 Offline uncorrectable sectors
71Oct 09 05:15:03 cloudcephosd1030 smart_failure[1601083]: Device: /dev/sdf [SAT], 2984 Offline uncorrectable sectors
72Oct 09 10:15:02 cloudcephosd1030 smart_failure[1713436]: Device: /dev/sdh [SAT], 2616 Offline uncorrectable sectors
73Oct 09 13:15:02 cloudcephosd1030 smart_failure[1781526]: Device: /dev/sdc [SAT], 2272 Offline uncorrectable sectors
74Oct 09 13:45:02 cloudcephosd1030 smart_failure[1792886]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
75Oct 09 16:15:02 cloudcephosd1030 smart_failure[1848892]: Device: /dev/sdg [SAT], 1448 Offline uncorrectable sectors
76Oct 10 01:45:02 cloudcephosd1030 smart_failure[2063321]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
77Oct 10 02:15:02 cloudcephosd1030 smart_failure[2074508]: Device: /dev/sdj [SAT], 1224 Offline uncorrectable sectors
78Oct 10 02:45:02 cloudcephosd1030 smart_failure[2085711]: Device: /dev/sde [SAT], 1952 Offline uncorrectable sectors
79Oct 10 05:15:02 cloudcephosd1030 smart_failure[2141993]: Device: /dev/sdf [SAT], 3000 Offline uncorrectable sectors
80Oct 10 10:15:02 cloudcephosd1030 smart_failure[2254994]: Device: /dev/sdh [SAT], 2624 Offline uncorrectable sectors
81Oct 10 13:15:03 cloudcephosd1030 smart_failure[2322239]: Device: /dev/sdc [SAT], 2280 Offline uncorrectable sectors
82Oct 10 13:45:02 cloudcephosd1030 smart_failure[2333507]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
83Oct 10 16:15:02 cloudcephosd1030 smart_failure[2389839]: Device: /dev/sdg [SAT], 1448 Offline uncorrectable sectors
84Oct 11 01:45:02 cloudcephosd1030 smart_failure[2605386]: Device: /dev/sdd [SAT], 720 Offline uncorrectable sectors
85Oct 11 02:15:02 cloudcephosd1030 smart_failure[2616555]: Device: /dev/sdj [SAT], 1232 Offline uncorrectable sectors
86Oct 11 02:45:02 cloudcephosd1030 smart_failure[2627846]: Device: /dev/sde [SAT], 1952 Offline uncorrectable sectors
87Oct 11 05:15:02 cloudcephosd1030 smart_failure[2684512]: Device: /dev/sdf [SAT], 3000 Offline uncorrectable sectors
88Oct 11 10:15:02 cloudcephosd1030 smart_failure[2797289]: Device: /dev/sdh [SAT], 2624 Offline uncorrectable sectors
89Sep 15 01:45:02 cloudcephosd1030 smart_failure[1116327]: Device: /dev/sdd [SAT], 624 Offline uncorrectable sectors
90Sep 15 02:15:02 cloudcephosd1030 smart_failure[1127611]: Device: /dev/sdj [SAT], 1040 Offline uncorrectable sectors (changed +8)
91Sep 15 02:45:02 cloudcephosd1030 smart_failure[1139031]: Device: /dev/sde [SAT], 1856 Offline uncorrectable sectors
92Sep 15 04:45:02 cloudcephosd1030 smart_failure[1183790]: Device: /dev/sdf [SAT], 2696 Offline uncorrectable sectors
93Sep 15 09:45:02 cloudcephosd1030 smart_failure[1296453]: Device: /dev/sdh [SAT], 2520 Offline uncorrectable sectors
94Sep 15 11:45:02 cloudcephosd1030 smart_failure[1341462]: Device: /dev/sdc [SAT], 2192 Offline uncorrectable sectors
95Sep 15 12:45:02 cloudcephosd1030 smart_failure[1363915]: Device: /dev/sdi [SAT], 1688 Offline uncorrectable sectors
96Sep 15 15:45:03 cloudcephosd1030 smart_failure[1431098]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
97Sep 16 01:45:02 cloudcephosd1030 smart_failure[1655783]: Device: /dev/sdd [SAT], 624 Offline uncorrectable sectors
98Sep 16 02:15:02 cloudcephosd1030 smart_failure[1666982]: Device: /dev/sdj [SAT], 1048 Offline uncorrectable sectors
99Sep 16 02:45:02 cloudcephosd1030 smart_failure[1678273]: Device: /dev/sde [SAT], 1856 Offline uncorrectable sectors
100Sep 16 04:45:02 cloudcephosd1030 smart_failure[1723143]: Device: /dev/sdf [SAT], 2728 Offline uncorrectable sectors
101Sep 16 09:45:02 cloudcephosd1030 smart_failure[1835225]: Device: /dev/sdh [SAT], 2528 Offline uncorrectable sectors
102Sep 16 11:45:02 cloudcephosd1030 smart_failure[1880134]: Device: /dev/sdc [SAT], 2192 Offline uncorrectable sectors
103Sep 16 12:45:02 cloudcephosd1030 smart_failure[1902551]: Device: /dev/sdi [SAT], 1688 Offline uncorrectable sectors
104Sep 16 16:15:02 cloudcephosd1030 smart_failure[1980579]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
105Sep 17 01:45:02 cloudcephosd1030 smart_failure[2193135]: Device: /dev/sdd [SAT], 624 Offline uncorrectable sectors
106Sep 17 02:15:02 cloudcephosd1030 smart_failure[2204385]: Device: /dev/sdj [SAT], 1048 Offline uncorrectable sectors
107Sep 17 02:45:02 cloudcephosd1030 smart_failure[2215568]: Device: /dev/sde [SAT], 1856 Offline uncorrectable sectors
108Sep 17 04:45:02 cloudcephosd1030 smart_failure[2260473]: Device: /dev/sdf [SAT], 2744 Offline uncorrectable sectors
109Sep 17 09:45:02 cloudcephosd1030 smart_failure[2372370]: Device: /dev/sdh [SAT], 2544 Offline uncorrectable sectors
110Sep 17 11:45:02 cloudcephosd1030 smart_failure[2417056]: Device: /dev/sdc [SAT], 2192 Offline uncorrectable sectors
111Sep 17 12:45:02 cloudcephosd1030 smart_failure[2439459]: Device: /dev/sdi [SAT], 1688 Offline uncorrectable sectors
112Sep 17 16:15:02 cloudcephosd1030 smart_failure[2517571]: Device: /dev/sdg [SAT], 1344 Offline uncorrectable sectors
113Sep 18 01:45:02 cloudcephosd1030 smart_failure[2730357]: Device: /dev/sdd [SAT], 624 Offline uncorrectable sectors
114Sep 18 02:15:02 cloudcephosd1030 smart_failure[2741574]: Device: /dev/sdj [SAT], 1048 Offline uncorrectable sectors
115Sep 18 02:45:02 cloudcephosd1030 smart_failure[2752821]: Device: /dev/sde [SAT], 1880 Offline uncorrectable sectors
116Sep 18 04:45:02 cloudcephosd1030 smart_failure[2797696]: Device: /dev/sdf [SAT], 2752 Offline uncorrectable sectors
117Sep 18 09:45:03 cloudcephosd1030 smart_failure[2909503]: Device: /dev/sdh [SAT], 2552 Offline uncorrectable sectors
118Sep 18 11:45:02 cloudcephosd1030 smart_failure[2954649]: Device: /dev/sdc [SAT], 2192 Offline uncorrectable sectors
119Sep 18 12:45:02 cloudcephosd1030 smart_failure[2977788]: Device: /dev/sdi [SAT], 1696 Offline uncorrectable sectors
120Sep 18 16:15:02 cloudcephosd1030 smart_failure[3056702]: Device: /dev/sdg [SAT], 1344 Offline uncorrectable sectors
121Sep 19 01:45:03 cloudcephosd1030 smart_failure[3269674]: Device: /dev/sdd [SAT], 640 Offline uncorrectable sectors
122Sep 19 02:15:02 cloudcephosd1030 smart_failure[3280976]: Device: /dev/sdj [SAT], 1048 Offline uncorrectable sectors
123Sep 19 02:45:02 cloudcephosd1030 smart_failure[3292193]: Device: /dev/sde [SAT], 1880 Offline uncorrectable sectors
124Sep 19 04:45:02 cloudcephosd1030 smart_failure[3337002]: Device: /dev/sdf [SAT], 2760 Offline uncorrectable sectors
125Sep 19 10:15:02 cloudcephosd1030 smart_failure[3461294]: Device: /dev/sdh [SAT], 2552 Offline uncorrectable sectors
126Sep 19 11:45:02 cloudcephosd1030 smart_failure[3494978]: Device: /dev/sdc [SAT], 2200 Offline uncorrectable sectors
127Sep 19 12:45:02 cloudcephosd1030 smart_failure[3517478]: Device: /dev/sdi [SAT], 1696 Offline uncorrectable sectors
128Sep 19 16:15:02 cloudcephosd1030 smart_failure[3597100]: Device: /dev/sdg [SAT], 1360 Offline uncorrectable sectors
129Sep 20 01:45:02 cloudcephosd1030 smart_failure[3814352]: Device: /dev/sdd [SAT], 640 Offline uncorrectable sectors
130Sep 20 02:15:02 cloudcephosd1030 smart_failure[3825850]: Device: /dev/sdj [SAT], 1056 Offline uncorrectable sectors
131Sep 20 02:45:02 cloudcephosd1030 smart_failure[3837231]: Device: /dev/sde [SAT], 1880 Offline uncorrectable sectors
132Sep 20 04:45:02 cloudcephosd1030 smart_failure[3882644]: Device: /dev/sdf [SAT], 2768 Offline uncorrectable sectors
133Sep 20 10:15:02 cloudcephosd1030 smart_failure[4007829]: Device: /dev/sdh [SAT], 2552 Offline uncorrectable sectors
134Sep 20 11:45:02 cloudcephosd1030 smart_failure[4041908]: Device: /dev/sdc [SAT], 2200 Offline uncorrectable sectors
135Sep 20 12:45:02 cloudcephosd1030 smart_failure[4064363]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
136Sep 20 16:15:02 cloudcephosd1030 smart_failure[4144255]: Device: /dev/sdg [SAT], 1360 Offline uncorrectable sectors
137Sep 21 01:45:02 cloudcephosd1030 smart_failure[166807]: Device: /dev/sdd [SAT], 640 Offline uncorrectable sectors
138Sep 21 02:15:02 cloudcephosd1030 smart_failure[178165]: Device: /dev/sdj [SAT], 1056 Offline uncorrectable sectors
139Sep 21 02:45:02 cloudcephosd1030 smart_failure[189415]: Device: /dev/sde [SAT], 1904 Offline uncorrectable sectors
140Sep 21 04:45:02 cloudcephosd1030 smart_failure[234584]: Device: /dev/sdf [SAT], 2792 Offline uncorrectable sectors
141Sep 21 10:15:02 cloudcephosd1030 smart_failure[359618]: Device: /dev/sdh [SAT], 2552 Offline uncorrectable sectors
142Sep 21 11:45:02 cloudcephosd1030 smart_failure[393093]: Device: /dev/sdc [SAT], 2200 Offline uncorrectable sectors
143Sep 21 12:45:02 cloudcephosd1030 smart_failure[415951]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
144Sep 21 16:15:02 cloudcephosd1030 smart_failure[495992]: Device: /dev/sdg [SAT], 1360 Offline uncorrectable sectors
145Sep 22 01:45:02 cloudcephosd1030 smart_failure[711405]: Device: /dev/sdd [SAT], 640 Offline uncorrectable sectors
146Sep 22 02:15:02 cloudcephosd1030 smart_failure[722881]: Device: /dev/sdj [SAT], 1072 Offline uncorrectable sectors
147Sep 22 02:45:02 cloudcephosd1030 smart_failure[734307]: Device: /dev/sde [SAT], 1904 Offline uncorrectable sectors
148Sep 22 04:45:02 cloudcephosd1030 smart_failure[779725]: Device: /dev/sdf [SAT], 2792 Offline uncorrectable sectors
149Sep 22 10:15:02 cloudcephosd1030 smart_failure[904949]: Device: /dev/sdh [SAT], 2552 Offline uncorrectable sectors
150Sep 22 11:45:03 cloudcephosd1030 smart_failure[938741]: Device: /dev/sdc [SAT], 2200 Offline uncorrectable sectors
151Sep 22 12:45:02 cloudcephosd1030 smart_failure[961300]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
152Sep 22 16:15:02 cloudcephosd1030 smart_failure[1040102]: Device: /dev/sdg [SAT], 1360 Offline uncorrectable sectors
153Sep 23 01:45:02 cloudcephosd1030 smart_failure[1253502]: Device: /dev/sdd [SAT], 640 Offline uncorrectable sectors
154Sep 23 02:15:02 cloudcephosd1030 smart_failure[1264806]: Device: /dev/sdj [SAT], 1104 Offline uncorrectable sectors
155Sep 23 02:45:02 cloudcephosd1030 smart_failure[1275893]: Device: /dev/sde [SAT], 1904 Offline uncorrectable sectors
156Sep 23 04:45:02 cloudcephosd1030 smart_failure[1320603]: Device: /dev/sdf [SAT], 2792 Offline uncorrectable sectors
157Sep 23 10:15:02 cloudcephosd1030 smart_failure[1444674]: Device: /dev/sdh [SAT], 2560 Offline uncorrectable sectors
158Sep 23 12:15:02 cloudcephosd1030 smart_failure[1489718]: Device: /dev/sdc [SAT], 2200 Offline uncorrectable sectors
159Sep 23 12:45:02 cloudcephosd1030 smart_failure[1501082]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
160Sep 23 16:15:02 cloudcephosd1030 smart_failure[1579938]: Device: /dev/sdg [SAT], 1360 Offline uncorrectable sectors
161Sep 24 01:45:02 cloudcephosd1030 smart_failure[1794212]: Device: /dev/sdd [SAT], 640 Offline uncorrectable sectors
162Sep 24 02:15:02 cloudcephosd1030 smart_failure[1805528]: Device: /dev/sdj [SAT], 1112 Offline uncorrectable sectors
163Sep 24 02:45:02 cloudcephosd1030 smart_failure[1816918]: Device: /dev/sde [SAT], 1904 Offline uncorrectable sectors
164Sep 24 04:45:02 cloudcephosd1030 smart_failure[1861942]: Device: /dev/sdf [SAT], 2800 Offline uncorrectable sectors
165Sep 24 10:15:02 cloudcephosd1030 smart_failure[1985742]: Device: /dev/sdh [SAT], 2568 Offline uncorrectable sectors
166Sep 24 12:15:03 cloudcephosd1030 smart_failure[2030401]: Device: /dev/sdc [SAT], 2208 Offline uncorrectable sectors
167Sep 24 12:45:02 cloudcephosd1030 smart_failure[2041326]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
168Sep 24 16:15:02 cloudcephosd1030 smart_failure[2119937]: Device: /dev/sdg [SAT], 1368 Offline uncorrectable sectors
169Sep 25 01:45:02 cloudcephosd1030 smart_failure[2333626]: Device: /dev/sdd [SAT], 648 Offline uncorrectable sectors
170Sep 25 02:15:02 cloudcephosd1030 smart_failure[2344954]: Device: /dev/sdj [SAT], 1112 Offline uncorrectable sectors
171Sep 25 02:45:02 cloudcephosd1030 smart_failure[2356147]: Device: /dev/sde [SAT], 1920 Offline uncorrectable sectors
172Sep 25 04:45:02 cloudcephosd1030 smart_failure[2400978]: Device: /dev/sdf [SAT], 2800 Offline uncorrectable sectors
173Sep 25 10:15:02 cloudcephosd1030 smart_failure[2524714]: Device: /dev/sdh [SAT], 2576 Offline uncorrectable sectors
174Sep 25 12:45:02 cloudcephosd1030 smart_failure[2580713]: Device: /dev/sdc [SAT], 2216 Offline uncorrectable sectors
175Sep 25 12:45:02 cloudcephosd1030 smart_failure[2580732]: Device: /dev/sdi [SAT], 1736 Offline uncorrectable sectors
176Sep 25 16:15:02 cloudcephosd1030 smart_failure[2659422]: Device: /dev/sdg [SAT], 1368 Offline uncorrectable sectors
177Sep 26 01:45:02 cloudcephosd1030 smart_failure[2874395]: Device: /dev/sdd [SAT], 648 Offline uncorrectable sectors
178Sep 26 02:15:02 cloudcephosd1030 smart_failure[2885926]: Device: /dev/sdj [SAT], 1112 Offline uncorrectable sectors
179Sep 26 02:45:02 cloudcephosd1030 smart_failure[2897294]: Device: /dev/sde [SAT], 1928 Offline uncorrectable sectors
180Sep 26 04:45:02 cloudcephosd1030 smart_failure[2942104]: Device: /dev/sdf [SAT], 2800 Offline uncorrectable sectors
181Sep 26 10:15:02 cloudcephosd1030 smart_failure[3066211]: Device: /dev/sdh [SAT], 2584 Offline uncorrectable sectors
182Sep 26 12:45:03 cloudcephosd1030 smart_failure[3122492]: Device: /dev/sdc [SAT], 2216 Offline uncorrectable sectors
183Sep 26 12:45:03 cloudcephosd1030 smart_failure[3122511]: Device: /dev/sdi [SAT], 1752 Offline uncorrectable sectors
184Sep 26 16:15:02 cloudcephosd1030 smart_failure[3201645]: Device: /dev/sdg [SAT], 1368 Offline uncorrectable sectors
185Sep 27 01:45:02 cloudcephosd1030 smart_failure[3416531]: Device: /dev/sdd [SAT], 648 Offline uncorrectable sectors
186Sep 27 02:15:02 cloudcephosd1030 smart_failure[3427742]: Device: /dev/sdj [SAT], 1112 Offline uncorrectable sectors
187Sep 27 02:45:02 cloudcephosd1030 smart_failure[3439110]: Device: /dev/sde [SAT], 1928 Offline uncorrectable sectors
188Sep 27 04:45:02 cloudcephosd1030 smart_failure[3484056]: Device: /dev/sdf [SAT], 2824 Offline uncorrectable sectors
189Sep 27 10:15:02 cloudcephosd1030 smart_failure[3607349]: Device: /dev/sdh [SAT], 2592 Offline uncorrectable sectors
190Sep 27 13:15:02 cloudcephosd1030 smart_failure[3675266]: Device: /dev/sdc [SAT], 2240 Offline uncorrectable sectors
191Sep 27 13:15:02 cloudcephosd1030 smart_failure[3675285]: Device: /dev/sdi [SAT], 1752 Offline uncorrectable sectors
192Sep 27 16:15:02 cloudcephosd1030 smart_failure[3742423]: Device: /dev/sdg [SAT], 1368 Offline uncorrectable sectors
193Sep 28 01:45:02 cloudcephosd1030 smart_failure[3956027]: Device: /dev/sdd [SAT], 648 Offline uncorrectable sectors
194Sep 28 02:15:02 cloudcephosd1030 smart_failure[3967495]: Device: /dev/sdj [SAT], 1112 Offline uncorrectable sectors
195Sep 28 02:45:02 cloudcephosd1030 smart_failure[3978987]: Device: /dev/sde [SAT], 1928 Offline uncorrectable sectors
196Sep 28 04:45:02 cloudcephosd1030 smart_failure[4023847]: Device: /dev/sdf [SAT], 2832 Offline uncorrectable sectors
197Sep 28 10:15:02 cloudcephosd1030 smart_failure[4147179]: Device: /dev/sdh [SAT], 2592 Offline uncorrectable sectors
198Sep 28 13:15:02 cloudcephosd1030 smart_failure[20719]: Device: /dev/sdc [SAT], 2248 Offline uncorrectable sectors
199Sep 28 13:15:03 cloudcephosd1030 smart_failure[20738]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
200Sep 28 16:15:02 cloudcephosd1030 smart_failure[88214]: Device: /dev/sdg [SAT], 1376 Offline uncorrectable sectors
201Sep 29 01:45:02 cloudcephosd1030 smart_failure[302443]: Device: /dev/sdd [SAT], 680 Offline uncorrectable sectors
202Sep 29 02:15:02 cloudcephosd1030 smart_failure[313735]: Device: /dev/sdj [SAT], 1144 Offline uncorrectable sectors
203Sep 29 02:45:02 cloudcephosd1030 smart_failure[324907]: Device: /dev/sde [SAT], 1928 Offline uncorrectable sectors
204Sep 29 04:45:02 cloudcephosd1030 smart_failure[370051]: Device: /dev/sdf [SAT], 2896 Offline uncorrectable sectors
205Sep 29 10:15:02 cloudcephosd1030 smart_failure[493756]: Device: /dev/sdh [SAT], 2592 Offline uncorrectable sectors
206Sep 29 13:15:02 cloudcephosd1030 smart_failure[561361]: Device: /dev/sdc [SAT], 2256 Offline uncorrectable sectors
207Sep 29 13:15:02 cloudcephosd1030 smart_failure[561380]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
208Sep 29 16:15:02 cloudcephosd1030 smart_failure[628981]: Device: /dev/sdg [SAT], 1392 Offline uncorrectable sectors
209Sep 30 01:45:02 cloudcephosd1030 smart_failure[842938]: Device: /dev/sdd [SAT], 712 Offline uncorrectable sectors
210Sep 30 02:15:02 cloudcephosd1030 smart_failure[854340]: Device: /dev/sdj [SAT], 1152 Offline uncorrectable sectors
211Sep 30 02:45:02 cloudcephosd1030 smart_failure[865808]: Device: /dev/sde [SAT], 1928 Offline uncorrectable sectors
212Sep 30 04:45:02 cloudcephosd1030 smart_failure[911012]: Device: /dev/sdf [SAT], 2896 Offline uncorrectable sectors
213Sep 30 10:15:02 cloudcephosd1030 smart_failure[1035248]: Device: /dev/sdh [SAT], 2600 Offline uncorrectable sectors
214Sep 30 13:15:02 cloudcephosd1030 smart_failure[1102857]: Device: /dev/sdc [SAT], 2256 Offline uncorrectable sectors
215Sep 30 13:15:02 cloudcephosd1030 smart_failure[1102876]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
216Sep 30 16:15:02 cloudcephosd1030 smart_failure[1170560]: Device: /dev/sdg [SAT], 1392 Offline uncorrectable sectors
217===== NODE GROUP =====
218(1) cloudcephosd1029.eqiad.wmnet
219----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
220Oct 01 00:27:05 cloudcephosd1029 smart_failure[1375666]: Device: /dev/sde [SAT], 2896 Offline uncorrectable sectors
221Oct 01 00:27:05 cloudcephosd1029 smart_failure[1375685]: Device: /dev/sdf [SAT], 1024 Offline uncorrectable sectors
222Oct 01 00:57:05 cloudcephosd1029 smart_failure[1386513]: Device: /dev/sdj [SAT], 2480 Offline uncorrectable sectors
223Oct 01 06:27:05 cloudcephosd1029 smart_failure[1509999]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
224Oct 01 06:27:05 cloudcephosd1029 smart_failure[1510056]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
225Oct 01 10:57:05 cloudcephosd1029 smart_failure[1610909]: Device: /dev/sdh [SAT], 1056 Offline uncorrectable sectors
226Oct 01 12:27:05 cloudcephosd1029 smart_failure[1644741]: Device: /dev/sdi [SAT], 960 Offline uncorrectable sectors
227Oct 01 19:57:05 cloudcephosd1029 smart_failure[1813227]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
228Oct 02 00:27:05 cloudcephosd1029 smart_failure[1913543]: Device: /dev/sde [SAT], 2912 Offline uncorrectable sectors
229Oct 02 00:27:05 cloudcephosd1029 smart_failure[1913591]: Device: /dev/sdf [SAT], 1032 Offline uncorrectable sectors
230Oct 02 00:57:05 cloudcephosd1029 smart_failure[1924554]: Device: /dev/sdj [SAT], 2488 Offline uncorrectable sectors
231Oct 02 06:27:05 cloudcephosd1029 smart_failure[2048181]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
232Oct 02 06:27:05 cloudcephosd1029 smart_failure[2048200]: Device: /dev/sdd [SAT], 1928 Offline uncorrectable sectors
233Oct 02 10:57:05 cloudcephosd1029 smart_failure[2148471]: Device: /dev/sdh [SAT], 1056 Offline uncorrectable sectors
234Oct 02 12:27:05 cloudcephosd1029 smart_failure[2182294]: Device: /dev/sdi [SAT], 960 Offline uncorrectable sectors
235Oct 02 19:57:05 cloudcephosd1029 smart_failure[2349669]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
236Oct 03 00:27:05 cloudcephosd1029 smart_failure[2444400]: Device: /dev/sde [SAT], 2928 Offline uncorrectable sectors
237Oct 03 00:27:05 cloudcephosd1029 smart_failure[2444419]: Device: /dev/sdf [SAT], 1032 Offline uncorrectable sectors
238Oct 03 00:57:05 cloudcephosd1029 smart_failure[2454114]: Device: /dev/sdj [SAT], 2488 Offline uncorrectable sectors
239Oct 03 06:27:05 cloudcephosd1029 smart_failure[2576177]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
240Oct 03 06:27:05 cloudcephosd1029 smart_failure[2576223]: Device: /dev/sdd [SAT], 1928 Offline uncorrectable sectors
241Oct 03 10:57:05 cloudcephosd1029 smart_failure[2677573]: Device: /dev/sdh [SAT], 1056 Offline uncorrectable sectors
242Oct 03 12:27:05 cloudcephosd1029 smart_failure[2711815]: Device: /dev/sdi [SAT], 960 Offline uncorrectable sectors
243Oct 03 19:57:05 cloudcephosd1029 smart_failure[2879076]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
244Oct 04 00:27:05 cloudcephosd1029 smart_failure[2979814]: Device: /dev/sde [SAT], 2936 Offline uncorrectable sectors
245Oct 04 00:27:05 cloudcephosd1029 smart_failure[2979857]: Device: /dev/sdf [SAT], 1032 Offline uncorrectable sectors
246Oct 04 00:57:05 cloudcephosd1029 smart_failure[2991026]: Device: /dev/sdj [SAT], 2504 Offline uncorrectable sectors
247Oct 04 06:27:05 cloudcephosd1029 smart_failure[3114419]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
248Oct 04 06:27:05 cloudcephosd1029 smart_failure[3114438]: Device: /dev/sdd [SAT], 1928 Offline uncorrectable sectors
249Oct 04 10:57:05 cloudcephosd1029 smart_failure[3215056]: Device: /dev/sdh [SAT], 1056 Offline uncorrectable sectors
250Oct 04 12:27:05 cloudcephosd1029 smart_failure[3248764]: Device: /dev/sdi [SAT], 960 Offline uncorrectable sectors
251Oct 04 19:57:05 cloudcephosd1029 smart_failure[3418157]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
252Oct 05 00:57:01 cloudcephosd1029 smart_failure[3530346]: Device: /dev/sde [SAT], 2944 Offline uncorrectable sectors
253Oct 05 00:57:01 cloudcephosd1029 smart_failure[3530392]: Device: /dev/sdf [SAT], 1032 Offline uncorrectable sectors
254Oct 05 01:27:01 cloudcephosd1029 smart_failure[3541714]: Device: /dev/sdj [SAT], 2504 Offline uncorrectable sectors
255Oct 05 06:57:01 cloudcephosd1029 smart_failure[3665213]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
256Oct 05 06:57:01 cloudcephosd1029 smart_failure[3665256]: Device: /dev/sdd [SAT], 1928 Offline uncorrectable sectors
257Oct 05 11:27:01 cloudcephosd1029 smart_failure[3767486]: Device: /dev/sdh [SAT], 1056 Offline uncorrectable sectors
258Oct 05 12:57:02 cloudcephosd1029 smart_failure[3813117]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
259Oct 05 20:27:01 cloudcephosd1029 smart_failure[3981412]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
260Oct 06 00:57:01 cloudcephosd1029 smart_failure[4082101]: Device: /dev/sde [SAT], 2944 Offline uncorrectable sectors
261Oct 06 00:57:01 cloudcephosd1029 smart_failure[4082149]: Device: /dev/sdf [SAT], 1032 Offline uncorrectable sectors
262Oct 06 01:27:01 cloudcephosd1029 smart_failure[4093413]: Device: /dev/sdj [SAT], 2504 Offline uncorrectable sectors
263Oct 06 06:57:01 cloudcephosd1029 smart_failure[23991]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
264Oct 06 06:57:01 cloudcephosd1029 smart_failure[24010]: Device: /dev/sdd [SAT], 1936 Offline uncorrectable sectors
265Oct 06 11:27:01 cloudcephosd1029 smart_failure[124764]: Device: /dev/sdh [SAT], 1056 Offline uncorrectable sectors
266Oct 06 13:27:01 cloudcephosd1029 smart_failure[169759]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
267Oct 06 20:27:01 cloudcephosd1029 smart_failure[327496]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
268Oct 07 00:57:01 cloudcephosd1029 smart_failure[429102]: Device: /dev/sde [SAT], 2944 Offline uncorrectable sectors
269Oct 07 00:57:01 cloudcephosd1029 smart_failure[429150]: Device: /dev/sdf [SAT], 1032 Offline uncorrectable sectors
270Oct 07 01:27:01 cloudcephosd1029 smart_failure[440491]: Device: /dev/sdj [SAT], 2528 Offline uncorrectable sectors
271Oct 07 06:57:01 cloudcephosd1029 smart_failure[564824]: Device: /dev/sdc [SAT], 1416 Offline uncorrectable sectors
272Oct 07 06:57:01 cloudcephosd1029 smart_failure[564869]: Device: /dev/sdd [SAT], 1936 Offline uncorrectable sectors
273Oct 07 11:27:01 cloudcephosd1029 smart_failure[666970]: Device: /dev/sdh [SAT], 1064 Offline uncorrectable sectors
274Oct 07 13:27:02 cloudcephosd1029 smart_failure[712297]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
275Oct 07 20:27:01 cloudcephosd1029 smart_failure[870672]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
276Oct 08 00:57:01 cloudcephosd1029 smart_failure[972835]: Device: /dev/sde [SAT], 2960 Offline uncorrectable sectors
277Oct 08 00:57:01 cloudcephosd1029 smart_failure[972879]: Device: /dev/sdf [SAT], 1032 Offline uncorrectable sectors
278Oct 08 01:27:01 cloudcephosd1029 smart_failure[984368]: Device: /dev/sdj [SAT], 2528 Offline uncorrectable sectors
279Oct 08 06:57:02 cloudcephosd1029 smart_failure[1109372]: Device: /dev/sdc [SAT], 1416 Offline uncorrectable sectors
280Oct 08 06:57:02 cloudcephosd1029 smart_failure[1109391]: Device: /dev/sdd [SAT], 1944 Offline uncorrectable sectors
281Oct 08 11:27:01 cloudcephosd1029 smart_failure[1210980]: Device: /dev/sdh [SAT], 1064 Offline uncorrectable sectors
282Oct 08 13:57:01 cloudcephosd1029 smart_failure[1267264]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
283Oct 08 20:27:01 cloudcephosd1029 smart_failure[1415033]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
284Oct 09 00:57:01 cloudcephosd1029 smart_failure[1516836]: Device: /dev/sde [SAT], 2960 Offline uncorrectable sectors
285Oct 09 00:57:01 cloudcephosd1029 smart_failure[1516879]: Device: /dev/sdf [SAT], 1040 Offline uncorrectable sectors
286Oct 09 01:27:01 cloudcephosd1029 smart_failure[1528371]: Device: /dev/sdj [SAT], 2528 Offline uncorrectable sectors
287Oct 09 07:27:01 cloudcephosd1029 smart_failure[1663926]: Device: /dev/sdc [SAT], 1416 Offline uncorrectable sectors
288Oct 09 07:27:01 cloudcephosd1029 smart_failure[1663970]: Device: /dev/sdd [SAT], 1952 Offline uncorrectable sectors
289Oct 09 11:27:01 cloudcephosd1029 smart_failure[1754859]: Device: /dev/sdh [SAT], 1064 Offline uncorrectable sectors
290Oct 09 13:57:02 cloudcephosd1029 smart_failure[1811474]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
291Oct 09 20:27:01 cloudcephosd1029 smart_failure[1958853]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
292Oct 10 00:57:01 cloudcephosd1029 smart_failure[2060685]: Device: /dev/sde [SAT], 2984 Offline uncorrectable sectors
293Oct 10 00:57:01 cloudcephosd1029 smart_failure[2060728]: Device: /dev/sdf [SAT], 1040 Offline uncorrectable sectors
294Oct 10 01:27:01 cloudcephosd1029 smart_failure[2072395]: Device: /dev/sdj [SAT], 2536 Offline uncorrectable sectors
295Oct 10 07:27:01 cloudcephosd1029 smart_failure[2208036]: Device: /dev/sdc [SAT], 1424 Offline uncorrectable sectors
296Oct 10 07:27:01 cloudcephosd1029 smart_failure[2208079]: Device: /dev/sdd [SAT], 1952 Offline uncorrectable sectors
297Oct 10 11:27:01 cloudcephosd1029 smart_failure[2299134]: Device: /dev/sdh [SAT], 1064 Offline uncorrectable sectors
298Oct 10 13:57:01 cloudcephosd1029 smart_failure[2355234]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
299Oct 10 20:27:01 cloudcephosd1029 smart_failure[2502673]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
300Oct 11 00:57:01 cloudcephosd1029 smart_failure[2603689]: Device: /dev/sde [SAT], 2984 Offline uncorrectable sectors
301Oct 11 00:57:01 cloudcephosd1029 smart_failure[2603732]: Device: /dev/sdf [SAT], 1040 Offline uncorrectable sectors
302Oct 11 01:27:01 cloudcephosd1029 smart_failure[2615210]: Device: /dev/sdj [SAT], 2544 Offline uncorrectable sectors
303Oct 11 07:27:01 cloudcephosd1029 smart_failure[2750424]: Device: /dev/sdc [SAT], 1424 Offline uncorrectable sectors
304Oct 11 07:27:01 cloudcephosd1029 smart_failure[2750468]: Device: /dev/sdd [SAT], 1952 Offline uncorrectable sectors
305Oct 11 11:27:01 cloudcephosd1029 smart_failure[2841042]: Device: /dev/sdh [SAT], 1064 Offline uncorrectable sectors
306Sep 08 05:57:05 cloudcephosd1029 smart_failure[1631346]: Device: /dev/sdc [SAT], 1312 Offline uncorrectable sectors
307Sep 08 06:27:05 cloudcephosd1029 smart_failure[1643045]: Device: /dev/sdd [SAT], 1872 Offline uncorrectable sectors
308Sep 08 10:27:05 cloudcephosd1029 smart_failure[1733333]: Device: /dev/sdh [SAT], 944 Offline uncorrectable sectors
309Sep 08 11:57:05 cloudcephosd1029 smart_failure[1767196]: Device: /dev/sdi [SAT], 936 Offline uncorrectable sectors
310Sep 08 19:27:05 cloudcephosd1029 smart_failure[1936126]: Device: /dev/sdg [SAT], 640 Offline uncorrectable sectors
311Sep 09 00:27:05 cloudcephosd1029 smart_failure[2049517]: Device: /dev/sde [SAT], 2744 Offline uncorrectable sectors
312Sep 09 00:27:05 cloudcephosd1029 smart_failure[2049536]: Device: /dev/sdf [SAT], 936 Offline uncorrectable sectors
313Sep 09 00:57:05 cloudcephosd1029 smart_failure[2060633]: Device: /dev/sdj [SAT], 2248 Offline uncorrectable sectors
314Sep 09 05:57:05 cloudcephosd1029 smart_failure[2173421]: Device: /dev/sdc [SAT], 1312 Offline uncorrectable sectors
315Sep 09 06:27:05 cloudcephosd1029 smart_failure[2184877]: Device: /dev/sdd [SAT], 1872 Offline uncorrectable sectors
316Sep 09 10:27:05 cloudcephosd1029 smart_failure[2274787]: Device: /dev/sdh [SAT], 976 Offline uncorrectable sectors
317Sep 09 11:57:05 cloudcephosd1029 smart_failure[2308232]: Device: /dev/sdi [SAT], 936 Offline uncorrectable sectors
318Sep 09 19:27:05 cloudcephosd1029 smart_failure[2477266]: Device: /dev/sdg [SAT], 640 Offline uncorrectable sectors
319Sep 10 00:27:05 cloudcephosd1029 smart_failure[2590513]: Device: /dev/sde [SAT], 2744 Offline uncorrectable sectors
320Sep 10 00:27:05 cloudcephosd1029 smart_failure[2590556]: Device: /dev/sdf [SAT], 936 Offline uncorrectable sectors
321Sep 10 00:57:05 cloudcephosd1029 smart_failure[2601821]: Device: /dev/sdj [SAT], 2256 Offline uncorrectable sectors
322Sep 10 05:57:05 cloudcephosd1029 smart_failure[2714485]: Device: /dev/sdc [SAT], 1312 Offline uncorrectable sectors
323Sep 10 06:27:05 cloudcephosd1029 smart_failure[2726127]: Device: /dev/sdd [SAT], 1880 Offline uncorrectable sectors (changed +8)
324Sep 10 10:27:05 cloudcephosd1029 smart_failure[2816141]: Device: /dev/sdh [SAT], 976 Offline uncorrectable sectors
325Sep 10 11:57:05 cloudcephosd1029 smart_failure[2849984]: Device: /dev/sdi [SAT], 936 Offline uncorrectable sectors
326Sep 10 19:27:05 cloudcephosd1029 smart_failure[3018959]: Device: /dev/sdg [SAT], 640 Offline uncorrectable sectors
327Sep 11 00:27:05 cloudcephosd1029 smart_failure[3131772]: Device: /dev/sde [SAT], 2752 Offline uncorrectable sectors (changed +8)
328Sep 11 00:27:05 cloudcephosd1029 smart_failure[3131791]: Device: /dev/sdf [SAT], 936 Offline uncorrectable sectors
329Sep 11 00:57:05 cloudcephosd1029 smart_failure[3142873]: Device: /dev/sdj [SAT], 2256 Offline uncorrectable sectors
330Sep 11 05:57:05 cloudcephosd1029 smart_failure[3255520]: Device: /dev/sdc [SAT], 1320 Offline uncorrectable sectors
331Sep 11 06:27:05 cloudcephosd1029 smart_failure[3267115]: Device: /dev/sdd [SAT], 1880 Offline uncorrectable sectors
332Sep 11 10:27:05 cloudcephosd1029 smart_failure[3357203]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
333Sep 11 11:57:05 cloudcephosd1029 smart_failure[3391469]: Device: /dev/sdi [SAT], 936 Offline uncorrectable sectors
334Sep 11 19:27:05 cloudcephosd1029 smart_failure[3560789]: Device: /dev/sdg [SAT], 648 Offline uncorrectable sectors
335Sep 12 00:27:05 cloudcephosd1029 smart_failure[3674245]: Device: /dev/sde [SAT], 2776 Offline uncorrectable sectors
336Sep 12 00:27:05 cloudcephosd1029 smart_failure[3674290]: Device: /dev/sdf [SAT], 944 Offline uncorrectable sectors
337Sep 12 00:57:05 cloudcephosd1029 smart_failure[3685669]: Device: /dev/sdj [SAT], 2264 Offline uncorrectable sectors
338Sep 12 05:57:05 cloudcephosd1029 smart_failure[3798642]: Device: /dev/sdc [SAT], 1320 Offline uncorrectable sectors
339Sep 12 06:27:05 cloudcephosd1029 smart_failure[3810433]: Device: /dev/sdd [SAT], 1888 Offline uncorrectable sectors
340Sep 12 10:27:05 cloudcephosd1029 smart_failure[3900679]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
341Sep 12 11:57:05 cloudcephosd1029 smart_failure[3934237]: Device: /dev/sdi [SAT], 936 Offline uncorrectable sectors
342Sep 12 19:27:05 cloudcephosd1029 smart_failure[4102562]: Device: /dev/sdg [SAT], 648 Offline uncorrectable sectors
343Sep 13 00:27:05 cloudcephosd1029 smart_failure[20843]: Device: /dev/sde [SAT], 2776 Offline uncorrectable sectors
344Sep 13 00:27:05 cloudcephosd1029 smart_failure[20886]: Device: /dev/sdf [SAT], 944 Offline uncorrectable sectors
345Sep 13 00:57:05 cloudcephosd1029 smart_failure[32025]: Device: /dev/sdj [SAT], 2264 Offline uncorrectable sectors
346Sep 13 05:57:05 cloudcephosd1029 smart_failure[143677]: Device: /dev/sdc [SAT], 1320 Offline uncorrectable sectors
347Sep 13 06:27:05 cloudcephosd1029 smart_failure[155157]: Device: /dev/sdd [SAT], 1904 Offline uncorrectable sectors
348Sep 13 10:27:05 cloudcephosd1029 smart_failure[244272]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
349Sep 13 11:57:05 cloudcephosd1029 smart_failure[277433]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
350Sep 13 19:27:05 cloudcephosd1029 smart_failure[445345]: Device: /dev/sdg [SAT], 656 Offline uncorrectable sectors
351Sep 14 00:27:05 cloudcephosd1029 smart_failure[557257]: Device: /dev/sde [SAT], 2776 Offline uncorrectable sectors
352Sep 14 00:27:06 cloudcephosd1029 smart_failure[557276]: Device: /dev/sdf [SAT], 944 Offline uncorrectable sectors
353Sep 14 00:57:05 cloudcephosd1029 smart_failure[568238]: Device: /dev/sdj [SAT], 2312 Offline uncorrectable sectors
354Sep 14 05:57:05 cloudcephosd1029 smart_failure[680940]: Device: /dev/sdc [SAT], 1328 Offline uncorrectable sectors
355Sep 14 06:27:05 cloudcephosd1029 smart_failure[692730]: Device: /dev/sdd [SAT], 1904 Offline uncorrectable sectors
356Sep 14 10:27:05 cloudcephosd1029 smart_failure[783278]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
357Sep 14 11:57:05 cloudcephosd1029 smart_failure[817216]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
358Sep 14 19:27:05 cloudcephosd1029 smart_failure[986085]: Device: /dev/sdg [SAT], 656 Offline uncorrectable sectors
359Sep 15 00:27:05 cloudcephosd1029 smart_failure[1098343]: Device: /dev/sde [SAT], 2776 Offline uncorrectable sectors
360Sep 15 00:27:05 cloudcephosd1029 smart_failure[1098387]: Device: /dev/sdf [SAT], 960 Offline uncorrectable sectors
361Sep 15 00:57:05 cloudcephosd1029 smart_failure[1109998]: Device: /dev/sdj [SAT], 2320 Offline uncorrectable sectors
362Sep 15 05:57:05 cloudcephosd1029 smart_failure[1223695]: Device: /dev/sdc [SAT], 1328 Offline uncorrectable sectors
363Sep 15 06:27:05 cloudcephosd1029 smart_failure[1235359]: Device: /dev/sdd [SAT], 1904 Offline uncorrectable sectors
364Sep 15 10:27:06 cloudcephosd1029 smart_failure[1325544]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
365Sep 15 11:57:05 cloudcephosd1029 smart_failure[1359132]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
366Sep 15 19:27:05 cloudcephosd1029 smart_failure[1528810]: Device: /dev/sdg [SAT], 656 Offline uncorrectable sectors
367Sep 16 00:27:05 cloudcephosd1029 smart_failure[1642179]: Device: /dev/sde [SAT], 2784 Offline uncorrectable sectors
368Sep 16 00:27:05 cloudcephosd1029 smart_failure[1642198]: Device: /dev/sdf [SAT], 960 Offline uncorrectable sectors
369Sep 16 00:57:05 cloudcephosd1029 smart_failure[1653395]: Device: /dev/sdj [SAT], 2336 Offline uncorrectable sectors
370Sep 16 05:57:05 cloudcephosd1029 smart_failure[1766484]: Device: /dev/sdc [SAT], 1328 Offline uncorrectable sectors
371Sep 16 06:27:05 cloudcephosd1029 smart_failure[1778034]: Device: /dev/sdd [SAT], 1904 Offline uncorrectable sectors
372Sep 16 10:57:05 cloudcephosd1029 smart_failure[1879608]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
373Sep 16 11:57:05 cloudcephosd1029 smart_failure[1902254]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
374Sep 16 19:27:06 cloudcephosd1029 smart_failure[2071553]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
375Sep 17 00:27:05 cloudcephosd1029 smart_failure[2184695]: Device: /dev/sde [SAT], 2792 Offline uncorrectable sectors
376Sep 17 00:27:05 cloudcephosd1029 smart_failure[2184714]: Device: /dev/sdf [SAT], 968 Offline uncorrectable sectors
377Sep 17 00:57:05 cloudcephosd1029 smart_failure[2195746]: Device: /dev/sdj [SAT], 2336 Offline uncorrectable sectors
378Sep 17 05:57:05 cloudcephosd1029 smart_failure[2308872]: Device: /dev/sdc [SAT], 1336 Offline uncorrectable sectors
379Sep 17 06:27:05 cloudcephosd1029 smart_failure[2320265]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
380Sep 17 10:57:05 cloudcephosd1029 smart_failure[2421926]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
381Sep 17 11:57:06 cloudcephosd1029 smart_failure[2444377]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
382Sep 17 19:57:05 cloudcephosd1029 smart_failure[2624512]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
383Sep 18 00:27:05 cloudcephosd1029 smart_failure[2726828]: Device: /dev/sde [SAT], 2800 Offline uncorrectable sectors
384Sep 18 00:27:05 cloudcephosd1029 smart_failure[2726847]: Device: /dev/sdf [SAT], 968 Offline uncorrectable sectors
385Sep 18 00:57:05 cloudcephosd1029 smart_failure[2738037]: Device: /dev/sdj [SAT], 2352 Offline uncorrectable sectors
386Sep 18 05:57:05 cloudcephosd1029 smart_failure[2850914]: Device: /dev/sdc [SAT], 1336 Offline uncorrectable sectors
387Sep 18 06:27:05 cloudcephosd1029 smart_failure[2862375]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
388Sep 18 10:57:05 cloudcephosd1029 smart_failure[2963945]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
389Sep 18 12:27:05 cloudcephosd1029 smart_failure[2997386]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
390Sep 18 19:57:05 cloudcephosd1029 smart_failure[3166698]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
391Sep 19 00:27:05 cloudcephosd1029 smart_failure[3268377]: Device: /dev/sde [SAT], 2824 Offline uncorrectable sectors (changed +16)
392Sep 19 00:27:05 cloudcephosd1029 smart_failure[3268396]: Device: /dev/sdf [SAT], 976 Offline uncorrectable sectors
393Sep 19 00:57:05 cloudcephosd1029 smart_failure[3279512]: Device: /dev/sdj [SAT], 2352 Offline uncorrectable sectors
394Sep 19 05:57:05 cloudcephosd1029 smart_failure[3392127]: Device: /dev/sdc [SAT], 1352 Offline uncorrectable sectors
395Sep 19 06:27:05 cloudcephosd1029 smart_failure[3403862]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
396Sep 19 10:57:05 cloudcephosd1029 smart_failure[3505775]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
397Sep 19 12:27:05 cloudcephosd1029 smart_failure[3539957]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
398Sep 19 19:57:05 cloudcephosd1029 smart_failure[3710384]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
399Sep 20 00:27:05 cloudcephosd1029 smart_failure[3813885]: Device: /dev/sde [SAT], 2832 Offline uncorrectable sectors
400Sep 20 00:27:05 cloudcephosd1029 smart_failure[3813904]: Device: /dev/sdf [SAT], 976 Offline uncorrectable sectors
401Sep 20 00:57:05 cloudcephosd1029 smart_failure[3824877]: Device: /dev/sdj [SAT], 2368 Offline uncorrectable sectors
402Sep 20 05:57:05 cloudcephosd1029 smart_failure[3938050]: Device: /dev/sdc [SAT], 1352 Offline uncorrectable sectors
403Sep 20 06:27:05 cloudcephosd1029 smart_failure[3949271]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
404Sep 20 10:57:05 cloudcephosd1029 smart_failure[4050654]: Device: /dev/sdh [SAT], 984 Offline uncorrectable sectors
405Sep 20 12:27:05 cloudcephosd1029 smart_failure[4084774]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
406Sep 20 19:57:05 cloudcephosd1029 smart_failure[60635]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
407Sep 21 00:27:05 cloudcephosd1029 smart_failure[162779]: Device: /dev/sde [SAT], 2832 Offline uncorrectable sectors
408Sep 21 00:27:05 cloudcephosd1029 smart_failure[162822]: Device: /dev/sdf [SAT], 976 Offline uncorrectable sectors
409Sep 21 00:57:05 cloudcephosd1029 smart_failure[174284]: Device: /dev/sdj [SAT], 2392 Offline uncorrectable sectors
410Sep 21 05:57:05 cloudcephosd1029 smart_failure[287122]: Device: /dev/sdc [SAT], 1368 Offline uncorrectable sectors
411Sep 21 06:27:05 cloudcephosd1029 smart_failure[298527]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
412Sep 21 10:57:05 cloudcephosd1029 smart_failure[399654]: Device: /dev/sdh [SAT], 1008 Offline uncorrectable sectors
413Sep 21 12:27:05 cloudcephosd1029 smart_failure[433418]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
414Sep 21 19:57:05 cloudcephosd1029 smart_failure[603440]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
415Sep 22 00:27:05 cloudcephosd1029 smart_failure[704938]: Device: /dev/sde [SAT], 2832 Offline uncorrectable sectors
416Sep 22 00:27:05 cloudcephosd1029 smart_failure[704980]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
417Sep 22 00:57:05 cloudcephosd1029 smart_failure[716129]: Device: /dev/sdj [SAT], 2392 Offline uncorrectable sectors
418Sep 22 05:57:06 cloudcephosd1029 smart_failure[828849]: Device: /dev/sdc [SAT], 1376 Offline uncorrectable sectors
419Sep 22 06:27:05 cloudcephosd1029 smart_failure[840207]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
420Sep 22 10:57:05 cloudcephosd1029 smart_failure[942105]: Device: /dev/sdh [SAT], 1008 Offline uncorrectable sectors
421Sep 22 12:27:05 cloudcephosd1029 smart_failure[975654]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
422Sep 22 19:57:05 cloudcephosd1029 smart_failure[1144924]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
423Sep 23 00:27:05 cloudcephosd1029 smart_failure[1246843]: Device: /dev/sde [SAT], 2832 Offline uncorrectable sectors
424Sep 23 00:27:05 cloudcephosd1029 smart_failure[1246862]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
425Sep 23 00:57:05 cloudcephosd1029 smart_failure[1257765]: Device: /dev/sdj [SAT], 2400 Offline uncorrectable sectors
426Sep 23 06:27:05 cloudcephosd1029 smart_failure[1382227]: Device: /dev/sdc [SAT], 1376 Offline uncorrectable sectors
427Sep 23 06:27:05 cloudcephosd1029 smart_failure[1382270]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
428Sep 23 10:57:05 cloudcephosd1029 smart_failure[1483654]: Device: /dev/sdh [SAT], 1032 Offline uncorrectable sectors
429Sep 23 12:27:05 cloudcephosd1029 smart_failure[1517275]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
430Sep 23 19:57:05 cloudcephosd1029 smart_failure[1685852]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
431Sep 24 00:27:05 cloudcephosd1029 smart_failure[1787916]: Device: /dev/sde [SAT], 2840 Offline uncorrectable sectors
432Sep 24 00:27:05 cloudcephosd1029 smart_failure[1787935]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
433Sep 24 00:57:05 cloudcephosd1029 smart_failure[1799177]: Device: /dev/sdj [SAT], 2416 Offline uncorrectable sectors
434Sep 24 06:27:05 cloudcephosd1029 smart_failure[1923068]: Device: /dev/sdc [SAT], 1376 Offline uncorrectable sectors
435Sep 24 06:27:05 cloudcephosd1029 smart_failure[1923087]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
436Sep 24 10:57:05 cloudcephosd1029 smart_failure[2024731]: Device: /dev/sdh [SAT], 1032 Offline uncorrectable sectors
437Sep 24 12:27:05 cloudcephosd1029 smart_failure[2058375]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
438Sep 24 19:57:05 cloudcephosd1029 smart_failure[2227309]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
439Sep 25 00:27:05 cloudcephosd1029 smart_failure[2328796]: Device: /dev/sde [SAT], 2840 Offline uncorrectable sectors
440Sep 25 00:27:05 cloudcephosd1029 smart_failure[2328815]: Device: /dev/sdf [SAT], 1016 Offline uncorrectable sectors
441Sep 25 00:57:05 cloudcephosd1029 smart_failure[2339875]: Device: /dev/sdj [SAT], 2416 Offline uncorrectable sectors
442Sep 25 06:27:05 cloudcephosd1029 smart_failure[2463666]: Device: /dev/sdc [SAT], 1384 Offline uncorrectable sectors
443Sep 25 06:27:05 cloudcephosd1029 smart_failure[2463685]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
444Sep 25 10:57:05 cloudcephosd1029 smart_failure[2564359]: Device: /dev/sdh [SAT], 1040 Offline uncorrectable sectors
445Sep 25 12:27:05 cloudcephosd1029 smart_failure[2598559]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
446Sep 25 19:57:05 cloudcephosd1029 smart_failure[2767264]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
447Sep 26 00:27:05 cloudcephosd1029 smart_failure[2868398]: Device: /dev/sde [SAT], 2848 Offline uncorrectable sectors
448Sep 26 00:27:05 cloudcephosd1029 smart_failure[2868443]: Device: /dev/sdf [SAT], 1016 Offline uncorrectable sectors
449Sep 26 00:57:05 cloudcephosd1029 smart_failure[2879744]: Device: /dev/sdj [SAT], 2424 Offline uncorrectable sectors
450Sep 26 06:27:05 cloudcephosd1029 smart_failure[3003868]: Device: /dev/sdc [SAT], 1400 Offline uncorrectable sectors
451Sep 26 06:27:05 cloudcephosd1029 smart_failure[3003916]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
452Sep 26 10:57:05 cloudcephosd1029 smart_failure[3105737]: Device: /dev/sdh [SAT], 1040 Offline uncorrectable sectors
453Sep 26 12:27:05 cloudcephosd1029 smart_failure[3139317]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
454Sep 26 19:57:05 cloudcephosd1029 smart_failure[3307704]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
455Sep 27 00:27:05 cloudcephosd1029 smart_failure[3409218]: Device: /dev/sde [SAT], 2864 Offline uncorrectable sectors
456Sep 27 00:27:05 cloudcephosd1029 smart_failure[3409263]: Device: /dev/sdf [SAT], 1024 Offline uncorrectable sectors
457Sep 27 00:57:05 cloudcephosd1029 smart_failure[3420344]: Device: /dev/sdj [SAT], 2456 Offline uncorrectable sectors
458Sep 27 06:27:05 cloudcephosd1029 smart_failure[3544840]: Device: /dev/sdc [SAT], 1400 Offline uncorrectable sectors
459Sep 27 06:27:05 cloudcephosd1029 smart_failure[3544883]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
460Sep 27 10:57:05 cloudcephosd1029 smart_failure[3645164]: Device: /dev/sdh [SAT], 1040 Offline uncorrectable sectors
461Sep 27 12:27:05 cloudcephosd1029 smart_failure[3679078]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
462Sep 27 19:57:05 cloudcephosd1029 smart_failure[3847829]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
463Sep 28 00:27:05 cloudcephosd1029 smart_failure[3949657]: Device: /dev/sde [SAT], 2872 Offline uncorrectable sectors
464Sep 28 00:27:05 cloudcephosd1029 smart_failure[3949700]: Device: /dev/sdf [SAT], 1024 Offline uncorrectable sectors
465Sep 28 00:57:05 cloudcephosd1029 smart_failure[3961065]: Device: /dev/sdj [SAT], 2464 Offline uncorrectable sectors
466Sep 28 06:27:05 cloudcephosd1029 smart_failure[4085258]: Device: /dev/sdc [SAT], 1400 Offline uncorrectable sectors
467Sep 28 06:27:05 cloudcephosd1029 smart_failure[4085306]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
468Sep 28 10:57:05 cloudcephosd1029 smart_failure[4186490]: Device: /dev/sdh [SAT], 1048 Offline uncorrectable sectors
469Sep 28 12:27:05 cloudcephosd1029 smart_failure[27182]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
470Sep 28 19:57:05 cloudcephosd1029 smart_failure[196985]: Device: /dev/sdg [SAT], 664 Offline uncorrectable sectors
471Sep 29 00:27:05 cloudcephosd1029 smart_failure[298545]: Device: /dev/sde [SAT], 2872 Offline uncorrectable sectors
472Sep 29 00:27:05 cloudcephosd1029 smart_failure[298564]: Device: /dev/sdf [SAT], 1024 Offline uncorrectable sectors
473Sep 29 00:57:05 cloudcephosd1029 smart_failure[309605]: Device: /dev/sdj [SAT], 2464 Offline uncorrectable sectors
474Sep 29 06:27:05 cloudcephosd1029 smart_failure[432779]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
475Sep 29 06:27:05 cloudcephosd1029 smart_failure[432822]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
476Sep 29 10:57:05 cloudcephosd1029 smart_failure[534165]: Device: /dev/sdh [SAT], 1048 Offline uncorrectable sectors
477Sep 29 12:27:05 cloudcephosd1029 smart_failure[567832]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
478Sep 29 19:57:05 cloudcephosd1029 smart_failure[735451]: Device: /dev/sdg [SAT], 672 Offline uncorrectable sectors
479Sep 30 00:27:05 cloudcephosd1029 smart_failure[836529]: Device: /dev/sde [SAT], 2880 Offline uncorrectable sectors
480Sep 30 00:27:05 cloudcephosd1029 smart_failure[836574]: Device: /dev/sdf [SAT], 1024 Offline uncorrectable sectors
481Sep 30 00:57:05 cloudcephosd1029 smart_failure[847742]: Device: /dev/sdj [SAT], 2472 Offline uncorrectable sectors
482Sep 30 06:27:05 cloudcephosd1029 smart_failure[971316]: Device: /dev/sdc [SAT], 1408 Offline uncorrectable sectors
483Sep 30 06:27:05 cloudcephosd1029 smart_failure[971335]: Device: /dev/sdd [SAT], 1912 Offline uncorrectable sectors
484Sep 30 10:57:05 cloudcephosd1029 smart_failure[1072014]: Device: /dev/sdh [SAT], 1048 Offline uncorrectable sectors
485Sep 30 12:27:05 cloudcephosd1029 smart_failure[1106070]: Device: /dev/sdi [SAT], 944 Offline uncorrectable sectors
486Sep 30 19:57:05 cloudcephosd1029 smart_failure[1274267]: Device: /dev/sdg [SAT], 680 Offline uncorrectable sectors
487===== NODE GROUP =====
488(1) cloudcephosd1025.eqiad.wmnet
489----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
490Oct 01 04:23:01 cloudcephosd1025 smart_failure[1591431]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
491Oct 01 09:23:01 cloudcephosd1025 smart_failure[1704445]: Device: /dev/sdh [SAT], 1904 Offline uncorrectable sectors
492Oct 01 12:23:01 cloudcephosd1025 smart_failure[1771899]: Device: /dev/sdd [SAT], 2224 Offline uncorrectable sectors
493Oct 01 13:23:01 cloudcephosd1025 smart_failure[1794383]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
494Oct 01 13:23:01 cloudcephosd1025 smart_failure[1794433]: Device: /dev/sdj [SAT], 840 Offline uncorrectable sectors
495Oct 01 15:53:01 cloudcephosd1025 smart_failure[1850634]: Device: /dev/sdf [SAT], 1520 Offline uncorrectable sectors
496Oct 01 20:53:01 cloudcephosd1025 smart_failure[1963871]: Device: /dev/sdb [SAT], 1440 Offline uncorrectable sectors
497Oct 01 22:23:01 cloudcephosd1025 smart_failure[1997548]: Device: /dev/sdi [SAT], 1592 Offline uncorrectable sectors
498Oct 02 04:23:01 cloudcephosd1025 smart_failure[2132344]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
499Oct 02 09:23:01 cloudcephosd1025 smart_failure[2244397]: Device: /dev/sdh [SAT], 1904 Offline uncorrectable sectors
500Oct 02 12:23:01 cloudcephosd1025 smart_failure[2312168]: Device: /dev/sdd [SAT], 2240 Offline uncorrectable sectors (changed +8)
501Oct 02 13:23:02 cloudcephosd1025 smart_failure[2334913]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
502Oct 02 13:23:02 cloudcephosd1025 smart_failure[2334960]: Device: /dev/sdj [SAT], 848 Offline uncorrectable sectors (changed +8)
503Oct 02 15:53:01 cloudcephosd1025 smart_failure[2390870]: Device: /dev/sdf [SAT], 1528 Offline uncorrectable sectors
504Oct 02 20:53:01 cloudcephosd1025 smart_failure[2503484]: Device: /dev/sdb [SAT], 1440 Offline uncorrectable sectors
505Oct 02 22:23:01 cloudcephosd1025 smart_failure[2536202]: Device: /dev/sdi [SAT], 1592 Offline uncorrectable sectors
506Oct 03 04:23:01 cloudcephosd1025 smart_failure[2663927]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
507Oct 03 09:23:01 cloudcephosd1025 smart_failure[2776542]: Device: /dev/sdh [SAT], 1904 Offline uncorrectable sectors
508Oct 03 12:23:01 cloudcephosd1025 smart_failure[2843667]: Device: /dev/sdd [SAT], 2248 Offline uncorrectable sectors
509Oct 03 13:23:01 cloudcephosd1025 smart_failure[2866780]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
510Oct 03 13:53:01 cloudcephosd1025 smart_failure[2877780]: Device: /dev/sdj [SAT], 848 Offline uncorrectable sectors
511Oct 03 15:53:01 cloudcephosd1025 smart_failure[2922796]: Device: /dev/sdf [SAT], 1536 Offline uncorrectable sectors
512Oct 03 20:53:01 cloudcephosd1025 smart_failure[3035559]: Device: /dev/sdb [SAT], 1440 Offline uncorrectable sectors
513Oct 03 22:23:02 cloudcephosd1025 smart_failure[3069543]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
514Oct 04 04:23:01 cloudcephosd1025 smart_failure[3204851]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
515Oct 04 09:23:01 cloudcephosd1025 smart_failure[3318347]: Device: /dev/sdh [SAT], 1912 Offline uncorrectable sectors
516Oct 04 12:23:02 cloudcephosd1025 smart_failure[3385838]: Device: /dev/sdd [SAT], 2248 Offline uncorrectable sectors
517Oct 04 13:23:02 cloudcephosd1025 smart_failure[3408428]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
518Oct 04 13:53:02 cloudcephosd1025 smart_failure[3419541]: Device: /dev/sdj [SAT], 864 Offline uncorrectable sectors
519Oct 04 15:53:02 cloudcephosd1025 smart_failure[3464781]: Device: /dev/sdf [SAT], 1560 Offline uncorrectable sectors
520Oct 04 20:53:02 cloudcephosd1025 smart_failure[3578205]: Device: /dev/sdb [SAT], 1440 Offline uncorrectable sectors
521Oct 04 22:23:02 cloudcephosd1025 smart_failure[3612343]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
522Oct 05 04:23:02 cloudcephosd1025 smart_failure[3747212]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
523Oct 05 09:23:02 cloudcephosd1025 smart_failure[3859935]: Device: /dev/sdh [SAT], 1912 Offline uncorrectable sectors
524Oct 05 12:23:02 cloudcephosd1025 smart_failure[3928230]: Device: /dev/sdd [SAT], 2248 Offline uncorrectable sectors
525Oct 05 13:23:02 cloudcephosd1025 smart_failure[3962147]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
526Oct 05 13:53:02 cloudcephosd1025 smart_failure[3973293]: Device: /dev/sdj [SAT], 864 Offline uncorrectable sectors
527Oct 05 15:53:02 cloudcephosd1025 smart_failure[4018508]: Device: /dev/sdf [SAT], 1584 Offline uncorrectable sectors
528Oct 05 20:53:02 cloudcephosd1025 smart_failure[4130925]: Device: /dev/sdb [SAT], 1448 Offline uncorrectable sectors
529Oct 05 22:23:02 cloudcephosd1025 smart_failure[4165102]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
530Oct 06 04:23:02 cloudcephosd1025 smart_failure[106710]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
531Oct 06 09:23:02 cloudcephosd1025 smart_failure[220519]: Device: /dev/sdh [SAT], 1912 Offline uncorrectable sectors
532Oct 06 12:23:02 cloudcephosd1025 smart_failure[287823]: Device: /dev/sdd [SAT], 2248 Offline uncorrectable sectors
533Oct 06 13:23:02 cloudcephosd1025 smart_failure[310104]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
534Oct 06 13:53:02 cloudcephosd1025 smart_failure[320988]: Device: /dev/sdj [SAT], 880 Offline uncorrectable sectors
535Oct 06 15:53:02 cloudcephosd1025 smart_failure[366472]: Device: /dev/sdf [SAT], 1592 Offline uncorrectable sectors
536Oct 06 20:53:02 cloudcephosd1025 smart_failure[479338]: Device: /dev/sdb [SAT], 1448 Offline uncorrectable sectors
537Oct 06 22:23:02 cloudcephosd1025 smart_failure[513085]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
538Oct 07 04:23:02 cloudcephosd1025 smart_failure[648411]: Device: /dev/sde [SAT], 1416 Offline uncorrectable sectors
539Oct 07 09:23:02 cloudcephosd1025 smart_failure[761011]: Device: /dev/sdh [SAT], 1912 Offline uncorrectable sectors
540Oct 07 12:23:02 cloudcephosd1025 smart_failure[828676]: Device: /dev/sdd [SAT], 2248 Offline uncorrectable sectors
541Oct 07 13:23:03 cloudcephosd1025 smart_failure[850999]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
542Oct 07 13:53:02 cloudcephosd1025 smart_failure[862029]: Device: /dev/sdj [SAT], 880 Offline uncorrectable sectors
543Oct 07 15:53:02 cloudcephosd1025 smart_failure[907085]: Device: /dev/sdf [SAT], 1592 Offline uncorrectable sectors
544Oct 07 20:53:02 cloudcephosd1025 smart_failure[1019682]: Device: /dev/sdb [SAT], 1448 Offline uncorrectable sectors
545Oct 07 22:23:02 cloudcephosd1025 smart_failure[1053667]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
546Oct 08 04:23:02 cloudcephosd1025 smart_failure[1189158]: Device: /dev/sde [SAT], 1424 Offline uncorrectable sectors
547Oct 08 09:23:02 cloudcephosd1025 smart_failure[1301764]: Device: /dev/sdh [SAT], 1912 Offline uncorrectable sectors
548Oct 08 12:23:02 cloudcephosd1025 smart_failure[1369194]: Device: /dev/sdd [SAT], 2256 Offline uncorrectable sectors
549Oct 08 13:23:02 cloudcephosd1025 smart_failure[1391588]: Device: /dev/sdg [SAT], 1576 Offline uncorrectable sectors
550Oct 08 13:53:02 cloudcephosd1025 smart_failure[1402665]: Device: /dev/sdj [SAT], 880 Offline uncorrectable sectors
551Oct 08 15:53:02 cloudcephosd1025 smart_failure[1447984]: Device: /dev/sdf [SAT], 1608 Offline uncorrectable sectors
552Oct 08 20:53:02 cloudcephosd1025 smart_failure[1560962]: Device: /dev/sdb [SAT], 1448 Offline uncorrectable sectors
553Oct 08 22:23:02 cloudcephosd1025 smart_failure[1594614]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
554Oct 09 04:23:02 cloudcephosd1025 smart_failure[1730123]: Device: /dev/sde [SAT], 1424 Offline uncorrectable sectors
555Oct 09 09:23:02 cloudcephosd1025 smart_failure[1842523]: Device: /dev/sdh [SAT], 1920 Offline uncorrectable sectors
556Oct 09 12:23:02 cloudcephosd1025 smart_failure[1910096]: Device: /dev/sdd [SAT], 2272 Offline uncorrectable sectors
557Oct 09 13:23:03 cloudcephosd1025 smart_failure[1932693]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
558Oct 09 13:53:02 cloudcephosd1025 smart_failure[1943804]: Device: /dev/sdj [SAT], 888 Offline uncorrectable sectors
559Oct 09 15:53:02 cloudcephosd1025 smart_failure[1988912]: Device: /dev/sdf [SAT], 1616 Offline uncorrectable sectors (changed +8)
560Oct 09 20:53:02 cloudcephosd1025 smart_failure[2101515]: Device: /dev/sdb [SAT], 1464 Offline uncorrectable sectors
561Oct 09 22:23:02 cloudcephosd1025 smart_failure[2135482]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
562Oct 10 04:23:02 cloudcephosd1025 smart_failure[2270560]: Device: /dev/sde [SAT], 1440 Offline uncorrectable sectors
563Oct 10 09:23:02 cloudcephosd1025 smart_failure[2383411]: Device: /dev/sdh [SAT], 1920 Offline uncorrectable sectors
564Oct 10 12:23:02 cloudcephosd1025 smart_failure[2451100]: Device: /dev/sdd [SAT], 2272 Offline uncorrectable sectors
565Oct 10 13:23:02 cloudcephosd1025 smart_failure[2475138]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
566Oct 10 13:53:02 cloudcephosd1025 smart_failure[2485915]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
567Oct 10 15:53:02 cloudcephosd1025 smart_failure[2530428]: Device: /dev/sdf [SAT], 1616 Offline uncorrectable sectors
568Oct 10 20:53:02 cloudcephosd1025 smart_failure[2643401]: Device: /dev/sdb [SAT], 1464 Offline uncorrectable sectors
569Oct 10 22:23:02 cloudcephosd1025 smart_failure[2677357]: Device: /dev/sdi [SAT], 1600 Offline uncorrectable sectors
570Oct 11 04:23:02 cloudcephosd1025 smart_failure[2812474]: Device: /dev/sde [SAT], 1440 Offline uncorrectable sectors
571Oct 11 09:23:02 cloudcephosd1025 smart_failure[2925458]: Device: /dev/sdh [SAT], 1920 Offline uncorrectable sectors
572Oct 11 12:23:02 cloudcephosd1025 smart_failure[2993791]: Device: /dev/sdd [SAT], 2272 Offline uncorrectable sectors
573Sep 09 08:23:01 cloudcephosd1025 smart_failure[2330694]: Device: /dev/sdh [SAT], 1824 Offline uncorrectable sectors
574Sep 09 12:23:01 cloudcephosd1025 smart_failure[2420433]: Device: /dev/sdd [SAT], 2112 Offline uncorrectable sectors (changed +8)
575Sep 09 12:53:01 cloudcephosd1025 smart_failure[2431476]: Device: /dev/sdg [SAT], 1504 Offline uncorrectable sectors
576Sep 09 13:23:01 cloudcephosd1025 smart_failure[2442933]: Device: /dev/sdj [SAT], 744 Offline uncorrectable sectors
577Sep 09 15:53:01 cloudcephosd1025 smart_failure[2498949]: Device: /dev/sdf [SAT], 1400 Offline uncorrectable sectors
578Sep 09 20:53:01 cloudcephosd1025 smart_failure[2611587]: Device: /dev/sdb [SAT], 1360 Offline uncorrectable sectors
579Sep 09 22:23:01 cloudcephosd1025 smart_failure[2645738]: Device: /dev/sdi [SAT], 1472 Offline uncorrectable sectors
580Sep 10 02:23:01 cloudcephosd1025 smart_failure[2736233]: Device: /dev/sde [SAT], 1280 Offline uncorrectable sectors
581Sep 10 08:23:01 cloudcephosd1025 smart_failure[2871579]: Device: /dev/sdh [SAT], 1824 Offline uncorrectable sectors
582Sep 10 12:23:01 cloudcephosd1025 smart_failure[2961730]: Device: /dev/sdd [SAT], 2112 Offline uncorrectable sectors
583Sep 10 12:53:01 cloudcephosd1025 smart_failure[2972797]: Device: /dev/sdg [SAT], 1504 Offline uncorrectable sectors
584Sep 10 13:23:01 cloudcephosd1025 smart_failure[2984295]: Device: /dev/sdj [SAT], 744 Offline uncorrectable sectors
585Sep 10 15:53:01 cloudcephosd1025 smart_failure[3040673]: Device: /dev/sdf [SAT], 1400 Offline uncorrectable sectors
586Sep 10 20:53:01 cloudcephosd1025 smart_failure[3153254]: Device: /dev/sdb [SAT], 1376 Offline uncorrectable sectors
587Sep 10 22:23:01 cloudcephosd1025 smart_failure[3187285]: Device: /dev/sdi [SAT], 1480 Offline uncorrectable sectors
588Sep 11 02:23:01 cloudcephosd1025 smart_failure[3277379]: Device: /dev/sde [SAT], 1280 Offline uncorrectable sectors
589Sep 11 08:23:01 cloudcephosd1025 smart_failure[3412574]: Device: /dev/sdh [SAT], 1824 Offline uncorrectable sectors
590Sep 11 12:23:01 cloudcephosd1025 smart_failure[3503531]: Device: /dev/sdd [SAT], 2136 Offline uncorrectable sectors
591Sep 11 12:53:01 cloudcephosd1025 smart_failure[3514716]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
592Sep 11 13:23:01 cloudcephosd1025 smart_failure[3525997]: Device: /dev/sdj [SAT], 760 Offline uncorrectable sectors
593Sep 11 15:53:01 cloudcephosd1025 smart_failure[3582635]: Device: /dev/sdf [SAT], 1400 Offline uncorrectable sectors
594Sep 11 20:53:01 cloudcephosd1025 smart_failure[3695921]: Device: /dev/sdb [SAT], 1400 Offline uncorrectable sectors
595Sep 11 22:23:01 cloudcephosd1025 smart_failure[3729853]: Device: /dev/sdi [SAT], 1480 Offline uncorrectable sectors
596Sep 12 02:23:01 cloudcephosd1025 smart_failure[3820624]: Device: /dev/sde [SAT], 1280 Offline uncorrectable sectors
597Sep 12 08:23:01 cloudcephosd1025 smart_failure[3956074]: Device: /dev/sdh [SAT], 1824 Offline uncorrectable sectors
598Sep 12 12:23:01 cloudcephosd1025 smart_failure[4046583]: Device: /dev/sdd [SAT], 2136 Offline uncorrectable sectors
599Sep 12 12:53:01 cloudcephosd1025 smart_failure[4057721]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
600Sep 12 13:23:01 cloudcephosd1025 smart_failure[4069535]: Device: /dev/sdj [SAT], 760 Offline uncorrectable sectors
601Sep 12 15:53:01 cloudcephosd1025 smart_failure[4125524]: Device: /dev/sdf [SAT], 1408 Offline uncorrectable sectors
602Sep 12 20:53:01 cloudcephosd1025 smart_failure[44799]: Device: /dev/sdb [SAT], 1400 Offline uncorrectable sectors
603Sep 12 22:23:01 cloudcephosd1025 smart_failure[78778]: Device: /dev/sdi [SAT], 1480 Offline uncorrectable sectors
604Sep 13 02:23:01 cloudcephosd1025 smart_failure[169491]: Device: /dev/sde [SAT], 1280 Offline uncorrectable sectors
605Sep 13 08:23:01 cloudcephosd1025 smart_failure[304934]: Device: /dev/sdh [SAT], 1832 Offline uncorrectable sectors (changed +8)
606Sep 13 12:23:01 cloudcephosd1025 smart_failure[395153]: Device: /dev/sdd [SAT], 2152 Offline uncorrectable sectors
607Sep 13 12:53:01 cloudcephosd1025 smart_failure[406319]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
608Sep 13 13:23:01 cloudcephosd1025 smart_failure[417869]: Device: /dev/sdj [SAT], 768 Offline uncorrectable sectors
609Sep 13 15:53:01 cloudcephosd1025 smart_failure[474402]: Device: /dev/sdf [SAT], 1416 Offline uncorrectable sectors
610Sep 13 20:53:01 cloudcephosd1025 smart_failure[586824]: Device: /dev/sdb [SAT], 1400 Offline uncorrectable sectors
611Sep 13 22:23:01 cloudcephosd1025 smart_failure[620868]: Device: /dev/sdi [SAT], 1480 Offline uncorrectable sectors
612Sep 14 02:23:01 cloudcephosd1025 smart_failure[711696]: Device: /dev/sde [SAT], 1288 Offline uncorrectable sectors
613Sep 14 08:23:02 cloudcephosd1025 smart_failure[847370]: Device: /dev/sdh [SAT], 1864 Offline uncorrectable sectors
614Sep 14 12:23:01 cloudcephosd1025 smart_failure[937559]: Device: /dev/sdd [SAT], 2152 Offline uncorrectable sectors
615Sep 14 12:53:01 cloudcephosd1025 smart_failure[948796]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
616Sep 14 13:23:01 cloudcephosd1025 smart_failure[960160]: Device: /dev/sdj [SAT], 768 Offline uncorrectable sectors
617Sep 14 15:53:01 cloudcephosd1025 smart_failure[1017000]: Device: /dev/sdf [SAT], 1416 Offline uncorrectable sectors
618Sep 14 20:53:01 cloudcephosd1025 smart_failure[1129802]: Device: /dev/sdb [SAT], 1400 Offline uncorrectable sectors
619Sep 14 22:23:01 cloudcephosd1025 smart_failure[1163915]: Device: /dev/sdi [SAT], 1488 Offline uncorrectable sectors
620Sep 15 02:23:01 cloudcephosd1025 smart_failure[1253817]: Device: /dev/sde [SAT], 1288 Offline uncorrectable sectors
621Sep 15 08:53:01 cloudcephosd1025 smart_failure[1400897]: Device: /dev/sdh [SAT], 1864 Offline uncorrectable sectors
622Sep 15 12:23:01 cloudcephosd1025 smart_failure[1480780]: Device: /dev/sdd [SAT], 2152 Offline uncorrectable sectors
623Sep 15 12:53:01 cloudcephosd1025 smart_failure[1491987]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
624Sep 15 13:23:01 cloudcephosd1025 smart_failure[1503689]: Device: /dev/sdj [SAT], 768 Offline uncorrectable sectors
625Sep 15 15:53:01 cloudcephosd1025 smart_failure[1559528]: Device: /dev/sdf [SAT], 1424 Offline uncorrectable sectors
626Sep 15 20:53:01 cloudcephosd1025 smart_failure[1673335]: Device: /dev/sdb [SAT], 1408 Offline uncorrectable sectors
627Sep 15 22:23:01 cloudcephosd1025 smart_failure[1707707]: Device: /dev/sdi [SAT], 1520 Offline uncorrectable sectors
628Sep 16 02:23:01 cloudcephosd1025 smart_failure[1798863]: Device: /dev/sde [SAT], 1296 Offline uncorrectable sectors
629Sep 16 08:53:02 cloudcephosd1025 smart_failure[1946509]: Device: /dev/sdh [SAT], 1880 Offline uncorrectable sectors
630Sep 16 12:23:01 cloudcephosd1025 smart_failure[2026063]: Device: /dev/sdd [SAT], 2176 Offline uncorrectable sectors
631Sep 16 12:53:01 cloudcephosd1025 smart_failure[2037224]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
632Sep 16 13:23:01 cloudcephosd1025 smart_failure[2048778]: Device: /dev/sdj [SAT], 776 Offline uncorrectable sectors
633Sep 16 15:53:01 cloudcephosd1025 smart_failure[2105037]: Device: /dev/sdf [SAT], 1432 Offline uncorrectable sectors
634Sep 16 20:53:01 cloudcephosd1025 smart_failure[2218427]: Device: /dev/sdb [SAT], 1408 Offline uncorrectable sectors
635Sep 16 22:23:01 cloudcephosd1025 smart_failure[2252810]: Device: /dev/sdi [SAT], 1520 Offline uncorrectable sectors
636Sep 17 02:23:01 cloudcephosd1025 smart_failure[2343659]: Device: /dev/sde [SAT], 1304 Offline uncorrectable sectors (changed +8)
637Sep 17 09:23:01 cloudcephosd1025 smart_failure[2502488]: Device: /dev/sdh [SAT], 1888 Offline uncorrectable sectors
638Sep 17 12:23:01 cloudcephosd1025 smart_failure[2570391]: Device: /dev/sdd [SAT], 2176 Offline uncorrectable sectors
639Sep 17 12:53:01 cloudcephosd1025 smart_failure[2581649]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
640Sep 17 13:23:01 cloudcephosd1025 smart_failure[2593033]: Device: /dev/sdj [SAT], 784 Offline uncorrectable sectors
641Sep 17 15:53:01 cloudcephosd1025 smart_failure[2649607]: Device: /dev/sdf [SAT], 1432 Offline uncorrectable sectors
642Sep 17 20:53:01 cloudcephosd1025 smart_failure[2762654]: Device: /dev/sdb [SAT], 1408 Offline uncorrectable sectors
643Sep 17 22:23:01 cloudcephosd1025 smart_failure[2796963]: Device: /dev/sdi [SAT], 1520 Offline uncorrectable sectors
644Sep 18 02:23:01 cloudcephosd1025 smart_failure[2888030]: Device: /dev/sde [SAT], 1304 Offline uncorrectable sectors
645Sep 18 09:23:01 cloudcephosd1025 smart_failure[3046676]: Device: /dev/sdh [SAT], 1888 Offline uncorrectable sectors
646Sep 18 12:23:01 cloudcephosd1025 smart_failure[3114242]: Device: /dev/sdd [SAT], 2176 Offline uncorrectable sectors
647Sep 18 12:53:01 cloudcephosd1025 smart_failure[3125883]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
648Sep 18 13:23:01 cloudcephosd1025 smart_failure[3137373]: Device: /dev/sdj [SAT], 800 Offline uncorrectable sectors (changed +8)
649Sep 18 15:53:01 cloudcephosd1025 smart_failure[3193452]: Device: /dev/sdf [SAT], 1448 Offline uncorrectable sectors
650Sep 18 20:53:01 cloudcephosd1025 smart_failure[3306871]: Device: /dev/sdb [SAT], 1408 Offline uncorrectable sectors
651Sep 18 22:23:01 cloudcephosd1025 smart_failure[3341318]: Device: /dev/sdi [SAT], 1528 Offline uncorrectable sectors
652Sep 19 02:23:01 cloudcephosd1025 smart_failure[3432656]: Device: /dev/sde [SAT], 1328 Offline uncorrectable sectors
653Sep 19 09:23:01 cloudcephosd1025 smart_failure[3592274]: Device: /dev/sdh [SAT], 1888 Offline uncorrectable sectors
654Sep 19 12:23:01 cloudcephosd1025 smart_failure[3660018]: Device: /dev/sdd [SAT], 2176 Offline uncorrectable sectors
655Sep 19 12:53:01 cloudcephosd1025 smart_failure[3671425]: Device: /dev/sdg [SAT], 1512 Offline uncorrectable sectors
656Sep 19 13:23:01 cloudcephosd1025 smart_failure[3682798]: Device: /dev/sdj [SAT], 800 Offline uncorrectable sectors
657Sep 19 15:53:01 cloudcephosd1025 smart_failure[3739328]: Device: /dev/sdf [SAT], 1448 Offline uncorrectable sectors
658Sep 19 20:53:01 cloudcephosd1025 smart_failure[3853728]: Device: /dev/sdb [SAT], 1408 Offline uncorrectable sectors
659Sep 19 22:23:01 cloudcephosd1025 smart_failure[3888267]: Device: /dev/sdi [SAT], 1536 Offline uncorrectable sectors
660Sep 20 02:23:01 cloudcephosd1025 smart_failure[3978906]: Device: /dev/sde [SAT], 1328 Offline uncorrectable sectors
661Sep 20 09:23:01 cloudcephosd1025 smart_failure[4137162]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
662Sep 20 12:23:01 cloudcephosd1025 smart_failure[10833]: Device: /dev/sdd [SAT], 2192 Offline uncorrectable sectors
663Sep 20 12:53:01 cloudcephosd1025 smart_failure[21893]: Device: /dev/sdg [SAT], 1536 Offline uncorrectable sectors
664Sep 20 13:23:01 cloudcephosd1025 smart_failure[33434]: Device: /dev/sdj [SAT], 800 Offline uncorrectable sectors
665Sep 20 15:53:01 cloudcephosd1025 smart_failure[89855]: Device: /dev/sdf [SAT], 1448 Offline uncorrectable sectors
666Sep 20 20:53:01 cloudcephosd1025 smart_failure[201987]: Device: /dev/sdb [SAT], 1408 Offline uncorrectable sectors
667Sep 20 22:23:01 cloudcephosd1025 smart_failure[236325]: Device: /dev/sdi [SAT], 1536 Offline uncorrectable sectors
668Sep 21 02:23:02 cloudcephosd1025 smart_failure[326502]: Device: /dev/sde [SAT], 1344 Offline uncorrectable sectors
669Sep 21 09:23:01 cloudcephosd1025 smart_failure[483394]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
670Sep 21 12:23:01 cloudcephosd1025 smart_failure[551285]: Device: /dev/sdd [SAT], 2192 Offline uncorrectable sectors
671Sep 21 12:53:01 cloudcephosd1025 smart_failure[562360]: Device: /dev/sdg [SAT], 1536 Offline uncorrectable sectors
672Sep 21 13:23:01 cloudcephosd1025 smart_failure[573773]: Device: /dev/sdj [SAT], 808 Offline uncorrectable sectors
673Sep 21 15:53:01 cloudcephosd1025 smart_failure[629948]: Device: /dev/sdf [SAT], 1448 Offline uncorrectable sectors
674Sep 21 20:53:01 cloudcephosd1025 smart_failure[742313]: Device: /dev/sdb [SAT], 1416 Offline uncorrectable sectors
675Sep 21 22:23:01 cloudcephosd1025 smart_failure[776434]: Device: /dev/sdi [SAT], 1536 Offline uncorrectable sectors
676Sep 22 02:53:01 cloudcephosd1025 smart_failure[877287]: Device: /dev/sde [SAT], 1352 Offline uncorrectable sectors
677Sep 22 09:23:01 cloudcephosd1025 smart_failure[1023800]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
678Sep 22 12:23:01 cloudcephosd1025 smart_failure[1090679]: Device: /dev/sdd [SAT], 2192 Offline uncorrectable sectors
679Sep 22 12:53:01 cloudcephosd1025 smart_failure[1102011]: Device: /dev/sdg [SAT], 1536 Offline uncorrectable sectors
680Sep 22 13:23:01 cloudcephosd1025 smart_failure[1113549]: Device: /dev/sdj [SAT], 808 Offline uncorrectable sectors
681Sep 22 15:53:01 cloudcephosd1025 smart_failure[1170173]: Device: /dev/sdf [SAT], 1448 Offline uncorrectable sectors
682Sep 22 20:53:01 cloudcephosd1025 smart_failure[1283260]: Device: /dev/sdb [SAT], 1416 Offline uncorrectable sectors
683Sep 22 22:23:01 cloudcephosd1025 smart_failure[1317345]: Device: /dev/sdi [SAT], 1536 Offline uncorrectable sectors
684Sep 23 02:53:02 cloudcephosd1025 smart_failure[1418740]: Device: /dev/sde [SAT], 1360 Offline uncorrectable sectors
685Sep 23 09:23:01 cloudcephosd1025 smart_failure[1565917]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
686Sep 23 12:23:01 cloudcephosd1025 smart_failure[1633288]: Device: /dev/sdd [SAT], 2192 Offline uncorrectable sectors
687Sep 23 12:53:01 cloudcephosd1025 smart_failure[1644496]: Device: /dev/sdg [SAT], 1536 Offline uncorrectable sectors
688Sep 23 13:23:01 cloudcephosd1025 smart_failure[1655643]: Device: /dev/sdj [SAT], 808 Offline uncorrectable sectors
689Sep 23 15:53:01 cloudcephosd1025 smart_failure[1711730]: Device: /dev/sdf [SAT], 1448 Offline uncorrectable sectors
690Sep 23 20:53:01 cloudcephosd1025 smart_failure[1824101]: Device: /dev/sdb [SAT], 1416 Offline uncorrectable sectors
691Sep 23 22:23:01 cloudcephosd1025 smart_failure[1858212]: Device: /dev/sdi [SAT], 1544 Offline uncorrectable sectors
692Sep 24 03:23:01 cloudcephosd1025 smart_failure[1970961]: Device: /dev/sde [SAT], 1368 Offline uncorrectable sectors
693Sep 24 09:23:01 cloudcephosd1025 smart_failure[2107189]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
694Sep 24 12:23:01 cloudcephosd1025 smart_failure[2174218]: Device: /dev/sdd [SAT], 2200 Offline uncorrectable sectors
695Sep 24 12:53:01 cloudcephosd1025 smart_failure[2185609]: Device: /dev/sdg [SAT], 1536 Offline uncorrectable sectors
696Sep 24 13:23:01 cloudcephosd1025 smart_failure[2197144]: Device: /dev/sdj [SAT], 808 Offline uncorrectable sectors
697Sep 24 15:53:01 cloudcephosd1025 smart_failure[2253325]: Device: /dev/sdf [SAT], 1456 Offline uncorrectable sectors
698Sep 24 20:53:01 cloudcephosd1025 smart_failure[2366079]: Device: /dev/sdb [SAT], 1416 Offline uncorrectable sectors
699Sep 24 22:23:01 cloudcephosd1025 smart_failure[2400288]: Device: /dev/sdi [SAT], 1552 Offline uncorrectable sectors
700Sep 25 03:23:02 cloudcephosd1025 smart_failure[2512267]: Device: /dev/sde [SAT], 1368 Offline uncorrectable sectors
701Sep 25 09:23:01 cloudcephosd1025 smart_failure[2647554]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
702Sep 25 12:23:01 cloudcephosd1025 smart_failure[2715438]: Device: /dev/sdd [SAT], 2208 Offline uncorrectable sectors
703Sep 25 12:53:01 cloudcephosd1025 smart_failure[2726880]: Device: /dev/sdg [SAT], 1536 Offline uncorrectable sectors
704Sep 25 13:23:01 cloudcephosd1025 smart_failure[2737916]: Device: /dev/sdj [SAT], 808 Offline uncorrectable sectors
705Sep 25 15:53:01 cloudcephosd1025 smart_failure[2794359]: Device: /dev/sdf [SAT], 1472 Offline uncorrectable sectors
706Sep 25 20:53:01 cloudcephosd1025 smart_failure[2907291]: Device: /dev/sdb [SAT], 1424 Offline uncorrectable sectors
707Sep 25 22:23:01 cloudcephosd1025 smart_failure[2941348]: Device: /dev/sdi [SAT], 1552 Offline uncorrectable sectors
708Sep 26 03:53:01 cloudcephosd1025 smart_failure[3065569]: Device: /dev/sde [SAT], 1384 Offline uncorrectable sectors
709Sep 26 09:23:01 cloudcephosd1025 smart_failure[3189822]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
710Sep 26 12:23:01 cloudcephosd1025 smart_failure[3256947]: Device: /dev/sdd [SAT], 2208 Offline uncorrectable sectors
711Sep 26 12:53:01 cloudcephosd1025 smart_failure[3268074]: Device: /dev/sdg [SAT], 1536 Offline uncorrectable sectors
712Sep 26 13:23:01 cloudcephosd1025 smart_failure[3279885]: Device: /dev/sdj [SAT], 816 Offline uncorrectable sectors
713Sep 26 15:53:01 cloudcephosd1025 smart_failure[3335911]: Device: /dev/sdf [SAT], 1472 Offline uncorrectable sectors
714Sep 26 20:53:01 cloudcephosd1025 smart_failure[3448158]: Device: /dev/sdb [SAT], 1432 Offline uncorrectable sectors
715Sep 26 22:23:01 cloudcephosd1025 smart_failure[3482373]: Device: /dev/sdi [SAT], 1552 Offline uncorrectable sectors
716Sep 27 03:53:01 cloudcephosd1025 smart_failure[3606574]: Device: /dev/sde [SAT], 1384 Offline uncorrectable sectors
717Sep 27 09:23:01 cloudcephosd1025 smart_failure[3729949]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
718Sep 27 12:23:01 cloudcephosd1025 smart_failure[3797970]: Device: /dev/sdd [SAT], 2216 Offline uncorrectable sectors
719Sep 27 12:53:01 cloudcephosd1025 smart_failure[3808995]: Device: /dev/sdg [SAT], 1552 Offline uncorrectable sectors
720Sep 27 13:23:01 cloudcephosd1025 smart_failure[3820476]: Device: /dev/sdj [SAT], 824 Offline uncorrectable sectors
721Sep 27 15:53:01 cloudcephosd1025 smart_failure[3877478]: Device: /dev/sdf [SAT], 1496 Offline uncorrectable sectors
722Sep 27 20:53:01 cloudcephosd1025 smart_failure[3990737]: Device: /dev/sdb [SAT], 1432 Offline uncorrectable sectors
723Sep 27 22:23:01 cloudcephosd1025 smart_failure[4024269]: Device: /dev/sdi [SAT], 1552 Offline uncorrectable sectors
724Sep 28 03:53:01 cloudcephosd1025 smart_failure[4148363]: Device: /dev/sde [SAT], 1384 Offline uncorrectable sectors
725Sep 28 09:23:01 cloudcephosd1025 smart_failure[78823]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
726Sep 28 12:23:01 cloudcephosd1025 smart_failure[146335]: Device: /dev/sdd [SAT], 2216 Offline uncorrectable sectors
727Sep 28 12:53:01 cloudcephosd1025 smart_failure[157846]: Device: /dev/sdg [SAT], 1552 Offline uncorrectable sectors
728Sep 28 13:23:01 cloudcephosd1025 smart_failure[169215]: Device: /dev/sdj [SAT], 832 Offline uncorrectable sectors
729Sep 28 15:53:01 cloudcephosd1025 smart_failure[225452]: Device: /dev/sdf [SAT], 1496 Offline uncorrectable sectors
730Sep 28 20:53:01 cloudcephosd1025 smart_failure[338480]: Device: /dev/sdb [SAT], 1440 Offline uncorrectable sectors
731Sep 28 22:23:01 cloudcephosd1025 smart_failure[372687]: Device: /dev/sdi [SAT], 1560 Offline uncorrectable sectors
732Sep 29 03:53:02 cloudcephosd1025 smart_failure[496559]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
733Sep 29 09:23:01 cloudcephosd1025 smart_failure[620641]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
734Sep 29 12:23:01 cloudcephosd1025 smart_failure[688201]: Device: /dev/sdd [SAT], 2224 Offline uncorrectable sectors
735Sep 29 12:53:01 cloudcephosd1025 smart_failure[699318]: Device: /dev/sdg [SAT], 1552 Offline uncorrectable sectors
736Sep 29 13:23:01 cloudcephosd1025 smart_failure[710762]: Device: /dev/sdj [SAT], 840 Offline uncorrectable sectors
737Sep 29 15:53:01 cloudcephosd1025 smart_failure[766777]: Device: /dev/sdf [SAT], 1504 Offline uncorrectable sectors
738Sep 29 20:53:01 cloudcephosd1025 smart_failure[879997]: Device: /dev/sdb [SAT], 1440 Offline uncorrectable sectors
739Sep 29 22:23:01 cloudcephosd1025 smart_failure[913872]: Device: /dev/sdi [SAT], 1568 Offline uncorrectable sectors
740Sep 30 04:23:01 cloudcephosd1025 smart_failure[1049210]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
741Sep 30 09:23:01 cloudcephosd1025 smart_failure[1162342]: Device: /dev/sdh [SAT], 1896 Offline uncorrectable sectors
742Sep 30 12:23:01 cloudcephosd1025 smart_failure[1229920]: Device: /dev/sdd [SAT], 2224 Offline uncorrectable sectors
743Sep 30 12:53:02 cloudcephosd1025 smart_failure[1241050]: Device: /dev/sdg [SAT], 1552 Offline uncorrectable sectors
744Sep 30 13:23:01 cloudcephosd1025 smart_failure[1252538]: Device: /dev/sdj [SAT], 840 Offline uncorrectable sectors
745Sep 30 15:53:01 cloudcephosd1025 smart_failure[1308745]: Device: /dev/sdf [SAT], 1504 Offline uncorrectable sectors
746Sep 30 20:53:01 cloudcephosd1025 smart_failure[1422032]: Device: /dev/sdb [SAT], 1440 Offline uncorrectable sectors
747Sep 30 22:23:01 cloudcephosd1025 smart_failure[1456049]: Device: /dev/sdi [SAT], 1584 Offline uncorrectable sectors
748===== NODE GROUP =====
749(1) cloudcephosd1027.eqiad.wmnet
750----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
751Oct 01 00:48:01 cloudcephosd1027 smart_failure[1407426]: Device: /dev/sdf [SAT], 2440 Offline uncorrectable sectors
752Oct 01 01:48:01 cloudcephosd1027 smart_failure[1429892]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
753Oct 01 08:18:01 cloudcephosd1027 smart_failure[1575636]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
754Oct 01 09:18:01 cloudcephosd1027 smart_failure[1597988]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
755Oct 01 09:18:01 cloudcephosd1027 smart_failure[1598007]: Device: /dev/sdi [SAT], 1056 Offline uncorrectable sectors
756Oct 01 12:18:01 cloudcephosd1027 smart_failure[1665249]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
757Oct 01 15:18:01 cloudcephosd1027 smart_failure[1732309]: Device: /dev/sdd [SAT], 1936 Offline uncorrectable sectors
758Oct 01 18:18:01 cloudcephosd1027 smart_failure[1799939]: Device: /dev/sdc [SAT], 2120 Offline uncorrectable sectors
759Oct 02 00:48:01 cloudcephosd1027 smart_failure[1945074]: Device: /dev/sdf [SAT], 2448 Offline uncorrectable sectors
760Oct 02 01:48:01 cloudcephosd1027 smart_failure[1967352]: Device: /dev/sdj [SAT], 904 Offline uncorrectable sectors
761Oct 02 08:18:01 cloudcephosd1027 smart_failure[2112840]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
762Oct 02 09:18:01 cloudcephosd1027 smart_failure[2134961]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
763Oct 02 09:18:01 cloudcephosd1027 smart_failure[2134980]: Device: /dev/sdi [SAT], 1056 Offline uncorrectable sectors
764Oct 02 12:18:01 cloudcephosd1027 smart_failure[2202462]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
765Oct 02 15:18:01 cloudcephosd1027 smart_failure[2269866]: Device: /dev/sdd [SAT], 1944 Offline uncorrectable sectors
766Oct 02 18:18:01 cloudcephosd1027 smart_failure[2337479]: Device: /dev/sdc [SAT], 2120 Offline uncorrectable sectors
767Oct 03 00:48:01 cloudcephosd1027 smart_failure[2476465]: Device: /dev/sdf [SAT], 2448 Offline uncorrectable sectors
768Oct 03 01:48:01 cloudcephosd1027 smart_failure[2496554]: Device: /dev/sdj [SAT], 904 Offline uncorrectable sectors
769Oct 03 08:18:01 cloudcephosd1027 smart_failure[2643030]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
770Oct 03 09:18:01 cloudcephosd1027 smart_failure[2665405]: Device: /dev/sdh [SAT], 680 Offline uncorrectable sectors
771Oct 03 09:18:01 cloudcephosd1027 smart_failure[2665424]: Device: /dev/sdi [SAT], 1056 Offline uncorrectable sectors
772Oct 03 12:18:01 cloudcephosd1027 smart_failure[2732540]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
773Oct 03 15:18:01 cloudcephosd1027 smart_failure[2800195]: Device: /dev/sdd [SAT], 1976 Offline uncorrectable sectors
774Oct 03 18:18:01 cloudcephosd1027 smart_failure[2867308]: Device: /dev/sdc [SAT], 2128 Offline uncorrectable sectors
775Oct 04 00:48:01 cloudcephosd1027 smart_failure[3013628]: Device: /dev/sdf [SAT], 2456 Offline uncorrectable sectors
776Oct 04 01:48:01 cloudcephosd1027 smart_failure[3036227]: Device: /dev/sdj [SAT], 904 Offline uncorrectable sectors
777Oct 04 08:18:01 cloudcephosd1027 smart_failure[3183280]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
778Oct 04 09:18:01 cloudcephosd1027 smart_failure[3205769]: Device: /dev/sdh [SAT], 680 Offline uncorrectable sectors
779Oct 04 09:18:01 cloudcephosd1027 smart_failure[3205788]: Device: /dev/sdi [SAT], 1064 Offline uncorrectable sectors
780Oct 04 12:18:01 cloudcephosd1027 smart_failure[3273448]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
781Oct 04 15:18:02 cloudcephosd1027 smart_failure[3341621]: Device: /dev/sdd [SAT], 1992 Offline uncorrectable sectors
782Oct 04 18:18:01 cloudcephosd1027 smart_failure[3409392]: Device: /dev/sdc [SAT], 2152 Offline uncorrectable sectors
783Oct 05 00:48:01 cloudcephosd1027 smart_failure[3556413]: Device: /dev/sdf [SAT], 2480 Offline uncorrectable sectors
784Oct 05 01:48:02 cloudcephosd1027 smart_failure[3578809]: Device: /dev/sdj [SAT], 904 Offline uncorrectable sectors
785Oct 05 08:18:01 cloudcephosd1027 smart_failure[3724646]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
786Oct 05 09:18:01 cloudcephosd1027 smart_failure[3747323]: Device: /dev/sdh [SAT], 680 Offline uncorrectable sectors
787Oct 05 09:18:01 cloudcephosd1027 smart_failure[3747342]: Device: /dev/sdi [SAT], 1064 Offline uncorrectable sectors
788Oct 05 12:18:01 cloudcephosd1027 smart_failure[3814997]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
789Oct 05 15:48:01 cloudcephosd1027 smart_failure[3905824]: Device: /dev/sdd [SAT], 1992 Offline uncorrectable sectors
790Oct 05 18:18:01 cloudcephosd1027 smart_failure[3961953]: Device: /dev/sdc [SAT], 2152 Offline uncorrectable sectors
791Oct 06 00:48:01 cloudcephosd1027 smart_failure[4108286]: Device: /dev/sdf [SAT], 2480 Offline uncorrectable sectors
792Oct 06 02:18:01 cloudcephosd1027 smart_failure[4142015]: Device: /dev/sdj [SAT], 904 Offline uncorrectable sectors
793Oct 06 08:18:01 cloudcephosd1027 smart_failure[83046]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
794Oct 06 09:18:01 cloudcephosd1027 smart_failure[105228]: Device: /dev/sdh [SAT], 688 Offline uncorrectable sectors
795Oct 06 09:18:01 cloudcephosd1027 smart_failure[105247]: Device: /dev/sdi [SAT], 1064 Offline uncorrectable sectors
796Oct 06 12:18:01 cloudcephosd1027 smart_failure[172800]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
797Oct 06 15:48:01 cloudcephosd1027 smart_failure[251428]: Device: /dev/sdd [SAT], 1992 Offline uncorrectable sectors
798Oct 06 18:18:01 cloudcephosd1027 smart_failure[307474]: Device: /dev/sdc [SAT], 2168 Offline uncorrectable sectors
799Oct 07 00:48:01 cloudcephosd1027 smart_failure[454729]: Device: /dev/sdf [SAT], 2496 Offline uncorrectable sectors
800Oct 07 02:18:01 cloudcephosd1027 smart_failure[488566]: Device: /dev/sdj [SAT], 912 Offline uncorrectable sectors
801Oct 07 08:18:01 cloudcephosd1027 smart_failure[624427]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
802Oct 07 09:18:01 cloudcephosd1027 smart_failure[647116]: Device: /dev/sdh [SAT], 688 Offline uncorrectable sectors
803Oct 07 09:18:01 cloudcephosd1027 smart_failure[647135]: Device: /dev/sdi [SAT], 1064 Offline uncorrectable sectors
804Oct 07 12:18:01 cloudcephosd1027 smart_failure[714900]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
805Oct 07 15:48:01 cloudcephosd1027 smart_failure[793921]: Device: /dev/sdd [SAT], 1992 Offline uncorrectable sectors
806Oct 07 18:18:01 cloudcephosd1027 smart_failure[850853]: Device: /dev/sdc [SAT], 2176 Offline uncorrectable sectors
807Oct 08 00:48:01 cloudcephosd1027 smart_failure[998025]: Device: /dev/sdf [SAT], 2496 Offline uncorrectable sectors
808Oct 08 02:18:01 cloudcephosd1027 smart_failure[1032258]: Device: /dev/sdj [SAT], 912 Offline uncorrectable sectors
809Oct 08 08:18:01 cloudcephosd1027 smart_failure[1168086]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
810Oct 08 09:18:01 cloudcephosd1027 smart_failure[1190599]: Device: /dev/sdh [SAT], 688 Offline uncorrectable sectors
811Oct 08 09:18:01 cloudcephosd1027 smart_failure[1190618]: Device: /dev/sdi [SAT], 1072 Offline uncorrectable sectors
812Oct 08 12:18:02 cloudcephosd1027 smart_failure[1258345]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
813Oct 08 15:48:01 cloudcephosd1027 smart_failure[1337519]: Device: /dev/sdd [SAT], 2008 Offline uncorrectable sectors
814Oct 08 18:18:01 cloudcephosd1027 smart_failure[1394502]: Device: /dev/sdc [SAT], 2176 Offline uncorrectable sectors
815Oct 09 00:48:01 cloudcephosd1027 smart_failure[1541620]: Device: /dev/sdf [SAT], 2496 Offline uncorrectable sectors
816Oct 09 02:18:01 cloudcephosd1027 smart_failure[1575868]: Device: /dev/sdj [SAT], 912 Offline uncorrectable sectors
817Oct 09 08:18:01 cloudcephosd1027 smart_failure[1711022]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
818Oct 09 09:18:01 cloudcephosd1027 smart_failure[1733715]: Device: /dev/sdh [SAT], 696 Offline uncorrectable sectors
819Oct 09 09:18:01 cloudcephosd1027 smart_failure[1733734]: Device: /dev/sdi [SAT], 1072 Offline uncorrectable sectors
820Oct 09 12:48:01 cloudcephosd1027 smart_failure[1813199]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
821Oct 09 15:48:01 cloudcephosd1027 smart_failure[1881028]: Device: /dev/sdd [SAT], 2008 Offline uncorrectable sectors
822Oct 09 18:18:01 cloudcephosd1027 smart_failure[1937697]: Device: /dev/sdc [SAT], 2176 Offline uncorrectable sectors
823Oct 10 00:48:01 cloudcephosd1027 smart_failure[2084995]: Device: /dev/sdf [SAT], 2504 Offline uncorrectable sectors
824Oct 10 02:18:01 cloudcephosd1027 smart_failure[2119018]: Device: /dev/sdj [SAT], 912 Offline uncorrectable sectors
825Oct 10 08:18:01 cloudcephosd1027 smart_failure[2254966]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
826Oct 10 09:18:01 cloudcephosd1027 smart_failure[2277707]: Device: /dev/sdh [SAT], 696 Offline uncorrectable sectors
827Oct 10 09:18:01 cloudcephosd1027 smart_failure[2277726]: Device: /dev/sdi [SAT], 1072 Offline uncorrectable sectors
828Oct 10 12:48:01 cloudcephosd1027 smart_failure[2356919]: Device: /dev/sdg [SAT], 976 Offline uncorrectable sectors
829Oct 10 15:48:01 cloudcephosd1027 smart_failure[2424246]: Device: /dev/sdd [SAT], 2008 Offline uncorrectable sectors
830Oct 10 18:18:01 cloudcephosd1027 smart_failure[2480871]: Device: /dev/sdc [SAT], 2184 Offline uncorrectable sectors
831Oct 11 00:48:01 cloudcephosd1027 smart_failure[2627100]: Device: /dev/sdf [SAT], 2504 Offline uncorrectable sectors
832Oct 11 02:18:01 cloudcephosd1027 smart_failure[2660979]: Device: /dev/sdj [SAT], 912 Offline uncorrectable sectors
833Oct 11 08:18:01 cloudcephosd1027 smart_failure[2796199]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
834Oct 11 09:18:01 cloudcephosd1027 smart_failure[2818983]: Device: /dev/sdh [SAT], 696 Offline uncorrectable sectors
835Oct 11 09:18:01 cloudcephosd1027 smart_failure[2819002]: Device: /dev/sdi [SAT], 1080 Offline uncorrectable sectors
836Sep 09 18:18:01 cloudcephosd1027 smart_failure[2459339]: Device: /dev/sdc [SAT], 1960 Offline uncorrectable sectors
837Sep 10 00:18:01 cloudcephosd1027 smart_failure[2594945]: Device: /dev/sdf [SAT], 2288 Offline uncorrectable sectors
838Sep 10 01:48:01 cloudcephosd1027 smart_failure[2628970]: Device: /dev/sdj [SAT], 856 Offline uncorrectable sectors
839Sep 10 07:48:01 cloudcephosd1027 smart_failure[2764651]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
840Sep 10 09:18:01 cloudcephosd1027 smart_failure[2798365]: Device: /dev/sdh [SAT], 656 Offline uncorrectable sectors
841Sep 10 09:18:01 cloudcephosd1027 smart_failure[2798384]: Device: /dev/sdi [SAT], 1000 Offline uncorrectable sectors
842Sep 10 12:18:01 cloudcephosd1027 smart_failure[2866106]: Device: /dev/sdg [SAT], 944 Offline uncorrectable sectors
843Sep 10 14:48:02 cloudcephosd1027 smart_failure[2922555]: Device: /dev/sdd [SAT], 1784 Offline uncorrectable sectors
844Sep 10 18:18:01 cloudcephosd1027 smart_failure[3001583]: Device: /dev/sdc [SAT], 1960 Offline uncorrectable sectors
845Sep 11 00:18:01 cloudcephosd1027 smart_failure[3136947]: Device: /dev/sdf [SAT], 2312 Offline uncorrectable sectors
846Sep 11 01:48:01 cloudcephosd1027 smart_failure[3170817]: Device: /dev/sdj [SAT], 856 Offline uncorrectable sectors
847Sep 11 07:48:02 cloudcephosd1027 smart_failure[3306125]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
848Sep 11 09:18:01 cloudcephosd1027 smart_failure[3339297]: Device: /dev/sdh [SAT], 656 Offline uncorrectable sectors
849Sep 11 09:18:01 cloudcephosd1027 smart_failure[3339316]: Device: /dev/sdi [SAT], 1000 Offline uncorrectable sectors
850Sep 11 12:18:01 cloudcephosd1027 smart_failure[3407012]: Device: /dev/sdg [SAT], 944 Offline uncorrectable sectors
851Sep 11 14:48:01 cloudcephosd1027 smart_failure[3462608]: Device: /dev/sdd [SAT], 1792 Offline uncorrectable sectors
852Sep 11 18:18:01 cloudcephosd1027 smart_failure[3542304]: Device: /dev/sdc [SAT], 1968 Offline uncorrectable sectors
853Sep 12 00:18:02 cloudcephosd1027 smart_failure[3677042]: Device: /dev/sdf [SAT], 2312 Offline uncorrectable sectors
854Sep 12 01:48:01 cloudcephosd1027 smart_failure[3710678]: Device: /dev/sdj [SAT], 856 Offline uncorrectable sectors
855Sep 12 08:18:01 cloudcephosd1027 smart_failure[3857342]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
856Sep 12 09:18:01 cloudcephosd1027 smart_failure[3879933]: Device: /dev/sdh [SAT], 656 Offline uncorrectable sectors
857Sep 12 09:18:01 cloudcephosd1027 smart_failure[3879952]: Device: /dev/sdi [SAT], 1032 Offline uncorrectable sectors
858Sep 12 12:18:01 cloudcephosd1027 smart_failure[3947232]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
859Sep 12 14:48:01 cloudcephosd1027 smart_failure[4004022]: Device: /dev/sdd [SAT], 1832 Offline uncorrectable sectors
860Sep 12 18:18:01 cloudcephosd1027 smart_failure[4082828]: Device: /dev/sdc [SAT], 2000 Offline uncorrectable sectors
861Sep 13 00:18:01 cloudcephosd1027 smart_failure[24656]: Device: /dev/sdf [SAT], 2320 Offline uncorrectable sectors (changed +8)
862Sep 13 01:48:01 cloudcephosd1027 smart_failure[58575]: Device: /dev/sdj [SAT], 856 Offline uncorrectable sectors
863Sep 13 08:18:01 cloudcephosd1027 smart_failure[205383]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
864Sep 13 09:18:01 cloudcephosd1027 smart_failure[227910]: Device: /dev/sdh [SAT], 656 Offline uncorrectable sectors
865Sep 13 09:18:01 cloudcephosd1027 smart_failure[227929]: Device: /dev/sdi [SAT], 1032 Offline uncorrectable sectors
866Sep 13 12:18:01 cloudcephosd1027 smart_failure[295613]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
867Sep 13 14:48:01 cloudcephosd1027 smart_failure[352211]: Device: /dev/sdd [SAT], 1832 Offline uncorrectable sectors
868Sep 13 18:18:01 cloudcephosd1027 smart_failure[430984]: Device: /dev/sdc [SAT], 2008 Offline uncorrectable sectors
869Sep 14 00:18:02 cloudcephosd1027 smart_failure[566756]: Device: /dev/sdf [SAT], 2360 Offline uncorrectable sectors
870Sep 14 01:48:01 cloudcephosd1027 smart_failure[600605]: Device: /dev/sdj [SAT], 856 Offline uncorrectable sectors
871Sep 14 08:18:01 cloudcephosd1027 smart_failure[747900]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
872Sep 14 09:18:01 cloudcephosd1027 smart_failure[770664]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
873Sep 14 09:18:01 cloudcephosd1027 smart_failure[770683]: Device: /dev/sdi [SAT], 1032 Offline uncorrectable sectors
874Sep 14 12:18:01 cloudcephosd1027 smart_failure[838348]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
875Sep 14 14:48:01 cloudcephosd1027 smart_failure[894451]: Device: /dev/sdd [SAT], 1848 Offline uncorrectable sectors
876Sep 14 18:18:01 cloudcephosd1027 smart_failure[973966]: Device: /dev/sdc [SAT], 2008 Offline uncorrectable sectors
877Sep 15 00:48:01 cloudcephosd1027 smart_failure[1120494]: Device: /dev/sdf [SAT], 2360 Offline uncorrectable sectors
878Sep 15 01:48:01 cloudcephosd1027 smart_failure[1143227]: Device: /dev/sdj [SAT], 872 Offline uncorrectable sectors
879Sep 15 08:18:01 cloudcephosd1027 smart_failure[1290368]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
880Sep 15 09:18:01 cloudcephosd1027 smart_failure[1313127]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
881Sep 15 09:18:01 cloudcephosd1027 smart_failure[1313146]: Device: /dev/sdi [SAT], 1032 Offline uncorrectable sectors
882Sep 15 12:18:01 cloudcephosd1027 smart_failure[1380859]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
883Sep 15 14:48:01 cloudcephosd1027 smart_failure[1437244]: Device: /dev/sdd [SAT], 1856 Offline uncorrectable sectors
884Sep 15 18:18:01 cloudcephosd1027 smart_failure[1516576]: Device: /dev/sdc [SAT], 2016 Offline uncorrectable sectors
885Sep 16 00:48:01 cloudcephosd1027 smart_failure[1664028]: Device: /dev/sdf [SAT], 2360 Offline uncorrectable sectors
886Sep 16 01:48:01 cloudcephosd1027 smart_failure[1686674]: Device: /dev/sdj [SAT], 872 Offline uncorrectable sectors
887Sep 16 08:18:01 cloudcephosd1027 smart_failure[1834409]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
888Sep 16 09:18:01 cloudcephosd1027 smart_failure[1857033]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
889Sep 16 09:18:01 cloudcephosd1027 smart_failure[1857052]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
890Sep 16 12:18:01 cloudcephosd1027 smart_failure[1924894]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
891Sep 16 14:48:01 cloudcephosd1027 smart_failure[1981282]: Device: /dev/sdd [SAT], 1872 Offline uncorrectable sectors
892Sep 16 18:18:02 cloudcephosd1027 smart_failure[2060523]: Device: /dev/sdc [SAT], 2016 Offline uncorrectable sectors
893Sep 17 00:48:01 cloudcephosd1027 smart_failure[2207437]: Device: /dev/sdf [SAT], 2360 Offline uncorrectable sectors
894Sep 17 01:48:01 cloudcephosd1027 smart_failure[2230152]: Device: /dev/sdj [SAT], 872 Offline uncorrectable sectors
895Sep 17 08:18:01 cloudcephosd1027 smart_failure[2377517]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
896Sep 17 09:18:01 cloudcephosd1027 smart_failure[2399970]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
897Sep 17 09:18:01 cloudcephosd1027 smart_failure[2399989]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
898Sep 17 12:18:01 cloudcephosd1027 smart_failure[2467690]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
899Sep 17 14:48:01 cloudcephosd1027 smart_failure[2524037]: Device: /dev/sdd [SAT], 1880 Offline uncorrectable sectors
900Sep 17 18:18:01 cloudcephosd1027 smart_failure[2603134]: Device: /dev/sdc [SAT], 2016 Offline uncorrectable sectors
901Sep 18 00:48:01 cloudcephosd1027 smart_failure[2750432]: Device: /dev/sdf [SAT], 2360 Offline uncorrectable sectors
902Sep 18 01:48:01 cloudcephosd1027 smart_failure[2773062]: Device: /dev/sdj [SAT], 872 Offline uncorrectable sectors
903Sep 18 08:18:01 cloudcephosd1027 smart_failure[2919987]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
904Sep 18 09:18:01 cloudcephosd1027 smart_failure[2942850]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
905Sep 18 09:18:01 cloudcephosd1027 smart_failure[2942869]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
906Sep 18 12:18:01 cloudcephosd1027 smart_failure[3010690]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
907Sep 18 14:48:01 cloudcephosd1027 smart_failure[3067133]: Device: /dev/sdd [SAT], 1888 Offline uncorrectable sectors
908Sep 18 18:18:01 cloudcephosd1027 smart_failure[3146154]: Device: /dev/sdc [SAT], 2016 Offline uncorrectable sectors
909Sep 19 00:48:01 cloudcephosd1027 smart_failure[3293819]: Device: /dev/sdf [SAT], 2368 Offline uncorrectable sectors
910Sep 19 01:48:01 cloudcephosd1027 smart_failure[3316545]: Device: /dev/sdj [SAT], 872 Offline uncorrectable sectors
911Sep 19 08:18:01 cloudcephosd1027 smart_failure[3464350]: Device: /dev/sde [SAT], 816 Offline uncorrectable sectors
912Sep 19 09:18:01 cloudcephosd1027 smart_failure[3487175]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
913Sep 19 09:18:01 cloudcephosd1027 smart_failure[3487194]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
914Sep 19 12:18:01 cloudcephosd1027 smart_failure[3555128]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
915Sep 19 14:48:01 cloudcephosd1027 smart_failure[3611956]: Device: /dev/sdd [SAT], 1888 Offline uncorrectable sectors
916Sep 19 18:18:01 cloudcephosd1027 smart_failure[3692031]: Device: /dev/sdc [SAT], 2032 Offline uncorrectable sectors
917Sep 20 00:48:01 cloudcephosd1027 smart_failure[3840873]: Device: /dev/sdf [SAT], 2384 Offline uncorrectable sectors
918Sep 20 01:48:01 cloudcephosd1027 smart_failure[3863570]: Device: /dev/sdj [SAT], 872 Offline uncorrectable sectors
919Sep 20 08:18:01 cloudcephosd1027 smart_failure[4011443]: Device: /dev/sde [SAT], 824 Offline uncorrectable sectors
920Sep 20 09:18:01 cloudcephosd1027 smart_failure[4034071]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
921Sep 20 09:18:01 cloudcephosd1027 smart_failure[4034090]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
922Sep 20 12:18:01 cloudcephosd1027 smart_failure[4102182]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
923Sep 20 14:48:01 cloudcephosd1027 smart_failure[4159068]: Device: /dev/sdd [SAT], 1888 Offline uncorrectable sectors
924Sep 20 18:18:01 cloudcephosd1027 smart_failure[44348]: Device: /dev/sdc [SAT], 2056 Offline uncorrectable sectors
925Sep 21 00:48:01 cloudcephosd1027 smart_failure[191847]: Device: /dev/sdf [SAT], 2384 Offline uncorrectable sectors
926Sep 21 01:48:01 cloudcephosd1027 smart_failure[214638]: Device: /dev/sdj [SAT], 872 Offline uncorrectable sectors
927Sep 21 08:18:01 cloudcephosd1027 smart_failure[361552]: Device: /dev/sde [SAT], 832 Offline uncorrectable sectors
928Sep 21 09:18:01 cloudcephosd1027 smart_failure[384309]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
929Sep 21 09:18:01 cloudcephosd1027 smart_failure[384333]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
930Sep 21 12:18:01 cloudcephosd1027 smart_failure[451993]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
931Sep 21 14:48:01 cloudcephosd1027 smart_failure[508845]: Device: /dev/sdd [SAT], 1904 Offline uncorrectable sectors
932Sep 21 18:18:01 cloudcephosd1027 smart_failure[588521]: Device: /dev/sdc [SAT], 2064 Offline uncorrectable sectors
933Sep 22 00:48:01 cloudcephosd1027 smart_failure[735549]: Device: /dev/sdf [SAT], 2408 Offline uncorrectable sectors
934Sep 22 01:48:01 cloudcephosd1027 smart_failure[757990]: Device: /dev/sdj [SAT], 880 Offline uncorrectable sectors
935Sep 22 08:18:01 cloudcephosd1027 smart_failure[905631]: Device: /dev/sde [SAT], 832 Offline uncorrectable sectors
936Sep 22 09:18:01 cloudcephosd1027 smart_failure[928544]: Device: /dev/sdh [SAT], 664 Offline uncorrectable sectors
937Sep 22 09:18:01 cloudcephosd1027 smart_failure[928563]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
938Sep 22 12:18:01 cloudcephosd1027 smart_failure[995830]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
939Sep 22 14:48:01 cloudcephosd1027 smart_failure[1052134]: Device: /dev/sdd [SAT], 1904 Offline uncorrectable sectors
940Sep 22 18:18:01 cloudcephosd1027 smart_failure[1131384]: Device: /dev/sdc [SAT], 2064 Offline uncorrectable sectors
941Sep 23 00:48:01 cloudcephosd1027 smart_failure[1278120]: Device: /dev/sdf [SAT], 2408 Offline uncorrectable sectors
942Sep 23 01:48:01 cloudcephosd1027 smart_failure[1300613]: Device: /dev/sdj [SAT], 888 Offline uncorrectable sectors
943Sep 23 08:18:01 cloudcephosd1027 smart_failure[1447754]: Device: /dev/sde [SAT], 832 Offline uncorrectable sectors
944Sep 23 09:18:01 cloudcephosd1027 smart_failure[1470307]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
945Sep 23 09:18:01 cloudcephosd1027 smart_failure[1470326]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
946Sep 23 12:18:01 cloudcephosd1027 smart_failure[1537516]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
947Sep 23 14:48:02 cloudcephosd1027 smart_failure[1593971]: Device: /dev/sdd [SAT], 1904 Offline uncorrectable sectors
948Sep 23 18:18:01 cloudcephosd1027 smart_failure[1672879]: Device: /dev/sdc [SAT], 2088 Offline uncorrectable sectors
949Sep 24 00:48:01 cloudcephosd1027 smart_failure[1819642]: Device: /dev/sdf [SAT], 2408 Offline uncorrectable sectors
950Sep 24 01:48:01 cloudcephosd1027 smart_failure[1842237]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
951Sep 24 08:18:01 cloudcephosd1027 smart_failure[1989228]: Device: /dev/sde [SAT], 832 Offline uncorrectable sectors
952Sep 24 09:18:01 cloudcephosd1027 smart_failure[2011794]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
953Sep 24 09:18:01 cloudcephosd1027 smart_failure[2011813]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
954Sep 24 12:18:01 cloudcephosd1027 smart_failure[2078919]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
955Sep 24 15:18:01 cloudcephosd1027 smart_failure[2146663]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
956Sep 24 18:18:01 cloudcephosd1027 smart_failure[2214197]: Device: /dev/sdc [SAT], 2088 Offline uncorrectable sectors
957Sep 25 00:48:01 cloudcephosd1027 smart_failure[2360345]: Device: /dev/sdf [SAT], 2408 Offline uncorrectable sectors
958Sep 25 01:48:01 cloudcephosd1027 smart_failure[2382782]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
959Sep 25 08:18:01 cloudcephosd1027 smart_failure[2529324]: Device: /dev/sde [SAT], 832 Offline uncorrectable sectors
960Sep 25 09:18:01 cloudcephosd1027 smart_failure[2552168]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
961Sep 25 09:18:01 cloudcephosd1027 smart_failure[2552187]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
962Sep 25 12:18:01 cloudcephosd1027 smart_failure[2619853]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
963Sep 25 15:18:01 cloudcephosd1027 smart_failure[2688116]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
964Sep 25 18:18:01 cloudcephosd1027 smart_failure[2755463]: Device: /dev/sdc [SAT], 2104 Offline uncorrectable sectors
965Sep 26 00:48:01 cloudcephosd1027 smart_failure[2901584]: Device: /dev/sdf [SAT], 2408 Offline uncorrectable sectors
966Sep 26 01:48:01 cloudcephosd1027 smart_failure[2924159]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
967Sep 26 08:18:01 cloudcephosd1027 smart_failure[3070117]: Device: /dev/sde [SAT], 832 Offline uncorrectable sectors
968Sep 26 09:18:01 cloudcephosd1027 smart_failure[3092699]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
969Sep 26 09:18:01 cloudcephosd1027 smart_failure[3092718]: Device: /dev/sdi [SAT], 1048 Offline uncorrectable sectors
970Sep 26 12:18:01 cloudcephosd1027 smart_failure[3160652]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
971Sep 26 15:18:01 cloudcephosd1027 smart_failure[3228096]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
972Sep 26 18:18:01 cloudcephosd1027 smart_failure[3295370]: Device: /dev/sdc [SAT], 2104 Offline uncorrectable sectors
973Sep 27 00:48:01 cloudcephosd1027 smart_failure[3441243]: Device: /dev/sdf [SAT], 2416 Offline uncorrectable sectors
974Sep 27 01:48:01 cloudcephosd1027 smart_failure[3463783]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
975Sep 27 08:18:01 cloudcephosd1027 smart_failure[3610113]: Device: /dev/sde [SAT], 832 Offline uncorrectable sectors
976Sep 27 09:18:01 cloudcephosd1027 smart_failure[3632515]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
977Sep 27 09:18:01 cloudcephosd1027 smart_failure[3632534]: Device: /dev/sdi [SAT], 1048 Offline uncorrectable sectors
978Sep 27 12:18:01 cloudcephosd1027 smart_failure[3700013]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
979Sep 27 15:18:01 cloudcephosd1027 smart_failure[3768553]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
980Sep 27 18:18:01 cloudcephosd1027 smart_failure[3836321]: Device: /dev/sdc [SAT], 2104 Offline uncorrectable sectors
981Sep 28 00:48:01 cloudcephosd1027 smart_failure[3982503]: Device: /dev/sdf [SAT], 2424 Offline uncorrectable sectors
982Sep 28 01:48:01 cloudcephosd1027 smart_failure[4005070]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
983Sep 28 08:18:01 cloudcephosd1027 smart_failure[4151903]: Device: /dev/sde [SAT], 856 Offline uncorrectable sectors
984Sep 28 09:18:01 cloudcephosd1027 smart_failure[4174125]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
985Sep 28 09:18:01 cloudcephosd1027 smart_failure[4174144]: Device: /dev/sdi [SAT], 1048 Offline uncorrectable sectors
986Sep 28 12:18:01 cloudcephosd1027 smart_failure[47614]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
987Sep 28 15:18:01 cloudcephosd1027 smart_failure[115715]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
988Sep 28 18:18:01 cloudcephosd1027 smart_failure[183294]: Device: /dev/sdc [SAT], 2112 Offline uncorrectable sectors
989Sep 29 00:48:01 cloudcephosd1027 smart_failure[330094]: Device: /dev/sdf [SAT], 2432 Offline uncorrectable sectors
990Sep 29 01:48:01 cloudcephosd1027 smart_failure[352460]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
991Sep 29 08:18:01 cloudcephosd1027 smart_failure[498095]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
992Sep 29 09:18:01 cloudcephosd1027 smart_failure[520900]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
993Sep 29 09:18:01 cloudcephosd1027 smart_failure[520919]: Device: /dev/sdi [SAT], 1048 Offline uncorrectable sectors
994Sep 29 12:18:01 cloudcephosd1027 smart_failure[588811]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
995Sep 29 15:18:01 cloudcephosd1027 smart_failure[655813]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
996Sep 29 18:18:01 cloudcephosd1027 smart_failure[723201]: Device: /dev/sdc [SAT], 2112 Offline uncorrectable sectors
997Sep 30 00:48:01 cloudcephosd1027 smart_failure[869050]: Device: /dev/sdf [SAT], 2432 Offline uncorrectable sectors
998Sep 30 01:48:01 cloudcephosd1027 smart_failure[891323]: Device: /dev/sdj [SAT], 896 Offline uncorrectable sectors
999Sep 30 08:18:01 cloudcephosd1027 smart_failure[1037422]: Device: /dev/sde [SAT], 864 Offline uncorrectable sectors
1000Sep 30 09:18:01 cloudcephosd1027 smart_failure[1059941]: Device: /dev/sdh [SAT], 672 Offline uncorrectable sectors
1001Sep 30 09:18:01 cloudcephosd1027 smart_failure[1059960]: Device: /dev/sdi [SAT], 1056 Offline uncorrectable sectors
1002Sep 30 12:18:01 cloudcephosd1027 smart_failure[1127080]: Device: /dev/sdg [SAT], 968 Offline uncorrectable sectors
1003Sep 30 15:18:01 cloudcephosd1027 smart_failure[1194281]: Device: /dev/sdd [SAT], 1920 Offline uncorrectable sectors
1004Sep 30 18:18:01 cloudcephosd1027 smart_failure[1261533]: Device: /dev/sdc [SAT], 2112 Offline uncorrectable sectors
1005===== NODE GROUP =====
1006(1) cloudcephosd1032.eqiad.wmnet
1007----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
1008Oct 01 01:12:04 cloudcephosd1032 smart_failure[1233833]: Device: /dev/sdc [SAT], 1840 Offline uncorrectable sectors
1009Oct 01 04:12:04 cloudcephosd1032 smart_failure[1301467]: Device: /dev/sde [SAT], 2408 Offline uncorrectable sectors
1010Oct 01 05:42:04 cloudcephosd1032 smart_failure[1335330]: Device: /dev/sdh [SAT], 2240 Offline uncorrectable sectors
1011Oct 01 09:12:05 cloudcephosd1032 smart_failure[1414046]: Device: /dev/sdf [SAT], 1624 Offline uncorrectable sectors
1012Oct 01 12:42:04 cloudcephosd1032 smart_failure[1492773]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1013Oct 01 19:42:04 cloudcephosd1032 smart_failure[1650404]: Device: /dev/sdd [SAT], 2632 Offline uncorrectable sectors
1014Oct 01 19:42:04 cloudcephosd1032 smart_failure[1650454]: Device: /dev/sdi [SAT], 1432 Offline uncorrectable sectors
1015Oct 01 22:12:04 cloudcephosd1032 smart_failure[1706312]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
1016Oct 02 01:12:04 cloudcephosd1032 smart_failure[1773526]: Device: /dev/sdc [SAT], 1848 Offline uncorrectable sectors
1017Oct 02 04:12:04 cloudcephosd1032 smart_failure[1841058]: Device: /dev/sde [SAT], 2416 Offline uncorrectable sectors
1018Oct 02 05:42:04 cloudcephosd1032 smart_failure[1874484]: Device: /dev/sdh [SAT], 2240 Offline uncorrectable sectors
1019Oct 02 09:12:04 cloudcephosd1032 smart_failure[1953108]: Device: /dev/sdf [SAT], 1624 Offline uncorrectable sectors
1020Oct 02 12:42:04 cloudcephosd1032 smart_failure[2031668]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1021Oct 02 19:42:04 cloudcephosd1032 smart_failure[2189713]: Device: /dev/sdd [SAT], 2640 Offline uncorrectable sectors
1022Oct 02 19:42:04 cloudcephosd1032 smart_failure[2189732]: Device: /dev/sdi [SAT], 1432 Offline uncorrectable sectors
1023Oct 02 22:12:04 cloudcephosd1032 smart_failure[2244850]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
1024Oct 03 01:12:04 cloudcephosd1032 smart_failure[2304989]: Device: /dev/sdc [SAT], 1848 Offline uncorrectable sectors
1025Oct 03 04:12:04 cloudcephosd1032 smart_failure[2371893]: Device: /dev/sde [SAT], 2424 Offline uncorrectable sectors
1026Oct 03 05:42:04 cloudcephosd1032 smart_failure[2405626]: Device: /dev/sdh [SAT], 2256 Offline uncorrectable sectors
1027Oct 03 09:12:04 cloudcephosd1032 smart_failure[2484303]: Device: /dev/sdf [SAT], 1632 Offline uncorrectable sectors
1028Oct 03 12:42:04 cloudcephosd1032 smart_failure[2563013]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1029Oct 03 19:42:04 cloudcephosd1032 smart_failure[2720945]: Device: /dev/sdd [SAT], 2640 Offline uncorrectable sectors
1030Oct 03 19:42:04 cloudcephosd1032 smart_failure[2720994]: Device: /dev/sdi [SAT], 1448 Offline uncorrectable sectors
1031Oct 03 22:12:04 cloudcephosd1032 smart_failure[2777841]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
1032Oct 04 01:12:04 cloudcephosd1032 smart_failure[2845714]: Device: /dev/sdc [SAT], 1864 Offline uncorrectable sectors
1033Oct 04 04:12:04 cloudcephosd1032 smart_failure[2913260]: Device: /dev/sde [SAT], 2424 Offline uncorrectable sectors
1034Oct 04 05:42:04 cloudcephosd1032 smart_failure[2947008]: Device: /dev/sdh [SAT], 2264 Offline uncorrectable sectors
1035Oct 04 09:42:00 cloudcephosd1032 smart_failure[3038704]: Device: /dev/sdf [SAT], 1632 Offline uncorrectable sectors
1036Oct 04 13:12:00 cloudcephosd1032 smart_failure[3118385]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1037Oct 04 20:12:00 cloudcephosd1032 smart_failure[3277831]: Device: /dev/sdd [SAT], 2640 Offline uncorrectable sectors
1038Oct 04 20:12:00 cloudcephosd1032 smart_failure[3277911]: Device: /dev/sdi [SAT], 1448 Offline uncorrectable sectors
1039Oct 04 22:42:00 cloudcephosd1032 smart_failure[3334002]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
1040Oct 05 01:42:01 cloudcephosd1032 smart_failure[3402110]: Device: /dev/sdc [SAT], 1864 Offline uncorrectable sectors
1041Oct 05 04:42:00 cloudcephosd1032 smart_failure[3469157]: Device: /dev/sde [SAT], 2424 Offline uncorrectable sectors
1042Oct 05 06:12:00 cloudcephosd1032 smart_failure[3502614]: Device: /dev/sdh [SAT], 2272 Offline uncorrectable sectors
1043Oct 05 09:42:00 cloudcephosd1032 smart_failure[3581675]: Device: /dev/sdf [SAT], 1632 Offline uncorrectable sectors
1044Oct 05 13:12:00 cloudcephosd1032 smart_failure[3672206]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1045Oct 05 20:12:00 cloudcephosd1032 smart_failure[3830405]: Device: /dev/sdd [SAT], 2656 Offline uncorrectable sectors
1046Oct 05 20:12:00 cloudcephosd1032 smart_failure[3830424]: Device: /dev/sdi [SAT], 1448 Offline uncorrectable sectors
1047Oct 05 22:42:00 cloudcephosd1032 smart_failure[3887012]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
1048Oct 06 02:12:00 cloudcephosd1032 smart_failure[3966366]: Device: /dev/sdc [SAT], 1896 Offline uncorrectable sectors
1049Oct 06 04:42:00 cloudcephosd1032 smart_failure[4022837]: Device: /dev/sde [SAT], 2424 Offline uncorrectable sectors
1050Oct 06 06:12:00 cloudcephosd1032 smart_failure[4056761]: Device: /dev/sdh [SAT], 2272 Offline uncorrectable sectors
1051Oct 06 09:42:01 cloudcephosd1032 smart_failure[4135996]: Device: /dev/sdf [SAT], 1632 Offline uncorrectable sectors
1052Oct 06 13:12:00 cloudcephosd1032 smart_failure[21473]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1053Oct 06 20:12:00 cloudcephosd1032 smart_failure[179769]: Device: /dev/sdd [SAT], 2656 Offline uncorrectable sectors
1054Oct 06 20:12:00 cloudcephosd1032 smart_failure[179819]: Device: /dev/sdi [SAT], 1448 Offline uncorrectable sectors
1055Oct 06 22:42:00 cloudcephosd1032 smart_failure[235997]: Device: /dev/sdg [SAT], 1336 Offline uncorrectable sectors
1056Oct 07 02:12:00 cloudcephosd1032 smart_failure[314767]: Device: /dev/sdc [SAT], 1896 Offline uncorrectable sectors
1057Oct 07 04:42:00 cloudcephosd1032 smart_failure[370479]: Device: /dev/sde [SAT], 2432 Offline uncorrectable sectors
1058Oct 07 06:12:00 cloudcephosd1032 smart_failure[404122]: Device: /dev/sdh [SAT], 2272 Offline uncorrectable sectors
1059Oct 07 10:12:00 cloudcephosd1032 smart_failure[494019]: Device: /dev/sdf [SAT], 1632 Offline uncorrectable sectors
1060Oct 07 13:12:00 cloudcephosd1032 smart_failure[561026]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1061Oct 07 20:12:00 cloudcephosd1032 smart_failure[717844]: Device: /dev/sdd [SAT], 2656 Offline uncorrectable sectors
1062Oct 07 20:12:00 cloudcephosd1032 smart_failure[717897]: Device: /dev/sdi [SAT], 1448 Offline uncorrectable sectors
1063Oct 07 22:42:00 cloudcephosd1032 smart_failure[773814]: Device: /dev/sdg [SAT], 1352 Offline uncorrectable sectors
1064Oct 08 02:12:00 cloudcephosd1032 smart_failure[852755]: Device: /dev/sdc [SAT], 1904 Offline uncorrectable sectors
1065Oct 08 04:42:00 cloudcephosd1032 smart_failure[908581]: Device: /dev/sde [SAT], 2464 Offline uncorrectable sectors
1066Oct 08 06:12:00 cloudcephosd1032 smart_failure[942420]: Device: /dev/sdh [SAT], 2272 Offline uncorrectable sectors
1067Oct 08 10:12:01 cloudcephosd1032 smart_failure[1031956]: Device: /dev/sdf [SAT], 1632 Offline uncorrectable sectors
1068Oct 08 13:12:00 cloudcephosd1032 smart_failure[1098844]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1069Oct 08 20:12:00 cloudcephosd1032 smart_failure[1256129]: Device: /dev/sdd [SAT], 2656 Offline uncorrectable sectors
1070Oct 08 20:12:00 cloudcephosd1032 smart_failure[1256166]: Device: /dev/sdi [SAT], 1456 Offline uncorrectable sectors
1071Oct 08 22:42:00 cloudcephosd1032 smart_failure[1311623]: Device: /dev/sdg [SAT], 1352 Offline uncorrectable sectors
1072Oct 09 02:12:00 cloudcephosd1032 smart_failure[1390576]: Device: /dev/sdc [SAT], 1912 Offline uncorrectable sectors
1073Oct 09 04:42:00 cloudcephosd1032 smart_failure[1446366]: Device: /dev/sde [SAT], 2472 Offline uncorrectable sectors
1074Oct 09 06:12:00 cloudcephosd1032 smart_failure[1479758]: Device: /dev/sdh [SAT], 2272 Offline uncorrectable sectors
1075Oct 09 10:42:00 cloudcephosd1032 smart_failure[1580722]: Device: /dev/sdf [SAT], 1640 Offline uncorrectable sectors
1076Oct 09 13:12:00 cloudcephosd1032 smart_failure[1636712]: Device: /dev/sdj [SAT], 1424 Offline uncorrectable sectors
1077Oct 09 20:12:00 cloudcephosd1032 smart_failure[1793473]: Device: /dev/sdd [SAT], 2664 Offline uncorrectable sectors
1078Oct 09 20:12:00 cloudcephosd1032 smart_failure[1793523]: Device: /dev/sdi [SAT], 1456 Offline uncorrectable sectors
1079Oct 09 22:42:00 cloudcephosd1032 smart_failure[1849387]: Device: /dev/sdg [SAT], 1352 Offline uncorrectable sectors
1080Oct 10 02:12:00 cloudcephosd1032 smart_failure[1927970]: Device: /dev/sdc [SAT], 1912 Offline uncorrectable sectors
1081Oct 10 04:42:00 cloudcephosd1032 smart_failure[1983806]: Device: /dev/sde [SAT], 2480 Offline uncorrectable sectors
1082Oct 10 06:12:00 cloudcephosd1032 smart_failure[2017499]: Device: /dev/sdh [SAT], 2272 Offline uncorrectable sectors
1083Oct 10 10:42:00 cloudcephosd1032 smart_failure[2118722]: Device: /dev/sdf [SAT], 1648 Offline uncorrectable sectors
1084Oct 10 13:12:00 cloudcephosd1032 smart_failure[2174742]: Device: /dev/sdj [SAT], 1424 Offline uncorrectable sectors
1085Oct 10 20:12:00 cloudcephosd1032 smart_failure[2332639]: Device: /dev/sdd [SAT], 2664 Offline uncorrectable sectors
1086Oct 10 20:12:00 cloudcephosd1032 smart_failure[2332689]: Device: /dev/sdi [SAT], 1456 Offline uncorrectable sectors
1087Oct 10 22:42:00 cloudcephosd1032 smart_failure[2389420]: Device: /dev/sdg [SAT], 1368 Offline uncorrectable sectors
1088Oct 11 02:12:00 cloudcephosd1032 smart_failure[2469117]: Device: /dev/sdc [SAT], 1912 Offline uncorrectable sectors
1089Oct 11 04:42:00 cloudcephosd1032 smart_failure[2525505]: Device: /dev/sde [SAT], 2512 Offline uncorrectable sectors
1090Oct 11 06:12:00 cloudcephosd1032 smart_failure[2559578]: Device: /dev/sdh [SAT], 2288 Offline uncorrectable sectors
1091Oct 11 10:42:00 cloudcephosd1032 smart_failure[2661159]: Device: /dev/sdf [SAT], 1648 Offline uncorrectable sectors
1092Sep 14 19:42:04 cloudcephosd1032 smart_failure[844690]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1093Sep 14 19:42:04 cloudcephosd1032 smart_failure[844709]: Device: /dev/sdi [SAT], 1256 Offline uncorrectable sectors (changed +8)
1094Sep 14 20:42:04 cloudcephosd1032 smart_failure[867205]: Device: /dev/sdg [SAT], 1248 Offline uncorrectable sectors
1095Sep 15 00:12:04 cloudcephosd1032 smart_failure[946216]: Device: /dev/sdc [SAT], 1712 Offline uncorrectable sectors
1096Sep 15 04:12:04 cloudcephosd1032 smart_failure[1036514]: Device: /dev/sde [SAT], 2280 Offline uncorrectable sectors
1097Sep 15 05:42:04 cloudcephosd1032 smart_failure[1070671]: Device: /dev/sdh [SAT], 2136 Offline uncorrectable sectors
1098Sep 15 08:42:04 cloudcephosd1032 smart_failure[1137959]: Device: /dev/sdf [SAT], 1520 Offline uncorrectable sectors
1099Sep 15 12:42:04 cloudcephosd1032 smart_failure[1228307]: Device: /dev/sdj [SAT], 1368 Offline uncorrectable sectors
1100Sep 15 19:42:04 cloudcephosd1032 smart_failure[1386205]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1101Sep 15 19:42:04 cloudcephosd1032 smart_failure[1386255]: Device: /dev/sdi [SAT], 1264 Offline uncorrectable sectors
1102Sep 15 20:42:05 cloudcephosd1032 smart_failure[1408880]: Device: /dev/sdg [SAT], 1248 Offline uncorrectable sectors
1103Sep 16 00:12:04 cloudcephosd1032 smart_failure[1488259]: Device: /dev/sdc [SAT], 1712 Offline uncorrectable sectors
1104Sep 16 04:12:04 cloudcephosd1032 smart_failure[1579253]: Device: /dev/sde [SAT], 2280 Offline uncorrectable sectors
1105Sep 16 05:42:04 cloudcephosd1032 smart_failure[1612802]: Device: /dev/sdh [SAT], 2144 Offline uncorrectable sectors
1106Sep 16 08:42:04 cloudcephosd1032 smart_failure[1680484]: Device: /dev/sdf [SAT], 1528 Offline uncorrectable sectors
1107Sep 16 12:42:04 cloudcephosd1032 smart_failure[1770908]: Device: /dev/sdj [SAT], 1368 Offline uncorrectable sectors
1108Sep 16 19:42:04 cloudcephosd1032 smart_failure[1928652]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1109Sep 16 19:42:04 cloudcephosd1032 smart_failure[1928703]: Device: /dev/sdi [SAT], 1280 Offline uncorrectable sectors
1110Sep 16 21:12:04 cloudcephosd1032 smart_failure[1962951]: Device: /dev/sdg [SAT], 1248 Offline uncorrectable sectors
1111Sep 17 00:12:04 cloudcephosd1032 smart_failure[2030829]: Device: /dev/sdc [SAT], 1712 Offline uncorrectable sectors
1112Sep 17 04:12:05 cloudcephosd1032 smart_failure[2121236]: Device: /dev/sde [SAT], 2288 Offline uncorrectable sectors
1113Sep 17 05:42:04 cloudcephosd1032 smart_failure[2154836]: Device: /dev/sdh [SAT], 2152 Offline uncorrectable sectors (changed +8)
1114Sep 17 08:42:04 cloudcephosd1032 smart_failure[2222769]: Device: /dev/sdf [SAT], 1528 Offline uncorrectable sectors
1115Sep 17 12:42:04 cloudcephosd1032 smart_failure[2312970]: Device: /dev/sdj [SAT], 1368 Offline uncorrectable sectors
1116Sep 17 19:42:04 cloudcephosd1032 smart_failure[2470596]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1117Sep 17 19:42:04 cloudcephosd1032 smart_failure[2470615]: Device: /dev/sdi [SAT], 1288 Offline uncorrectable sectors
1118Sep 17 21:12:05 cloudcephosd1032 smart_failure[2504461]: Device: /dev/sdg [SAT], 1256 Offline uncorrectable sectors
1119Sep 18 00:12:04 cloudcephosd1032 smart_failure[2572497]: Device: /dev/sdc [SAT], 1712 Offline uncorrectable sectors
1120Sep 18 04:12:04 cloudcephosd1032 smart_failure[2663268]: Device: /dev/sde [SAT], 2288 Offline uncorrectable sectors
1121Sep 18 05:42:04 cloudcephosd1032 smart_failure[2697015]: Device: /dev/sdh [SAT], 2152 Offline uncorrectable sectors
1122Sep 18 08:42:04 cloudcephosd1032 smart_failure[2764094]: Device: /dev/sdf [SAT], 1528 Offline uncorrectable sectors
1123Sep 18 12:42:04 cloudcephosd1032 smart_failure[2854592]: Device: /dev/sdj [SAT], 1368 Offline uncorrectable sectors
1124Sep 18 19:42:04 cloudcephosd1032 smart_failure[3012241]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1125Sep 18 19:42:04 cloudcephosd1032 smart_failure[3012290]: Device: /dev/sdi [SAT], 1296 Offline uncorrectable sectors
1126Sep 18 21:42:04 cloudcephosd1032 smart_failure[3057729]: Device: /dev/sdg [SAT], 1264 Offline uncorrectable sectors
1127Sep 19 00:12:04 cloudcephosd1032 smart_failure[3114767]: Device: /dev/sdc [SAT], 1720 Offline uncorrectable sectors
1128Sep 19 04:12:04 cloudcephosd1032 smart_failure[3205895]: Device: /dev/sde [SAT], 2288 Offline uncorrectable sectors
1129Sep 19 05:42:04 cloudcephosd1032 smart_failure[3239702]: Device: /dev/sdh [SAT], 2152 Offline uncorrectable sectors
1130Sep 19 08:42:04 cloudcephosd1032 smart_failure[3307936]: Device: /dev/sdf [SAT], 1528 Offline uncorrectable sectors
1131Sep 19 12:42:04 cloudcephosd1032 smart_failure[3397768]: Device: /dev/sdj [SAT], 1368 Offline uncorrectable sectors
1132Sep 19 19:42:04 cloudcephosd1032 smart_failure[3556750]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1133Sep 19 19:42:04 cloudcephosd1032 smart_failure[3556798]: Device: /dev/sdi [SAT], 1336 Offline uncorrectable sectors
1134Sep 19 21:42:05 cloudcephosd1032 smart_failure[3602611]: Device: /dev/sdg [SAT], 1264 Offline uncorrectable sectors
1135Sep 20 00:12:04 cloudcephosd1032 smart_failure[3659915]: Device: /dev/sdc [SAT], 1728 Offline uncorrectable sectors
1136Sep 20 04:12:04 cloudcephosd1032 smart_failure[3749784]: Device: /dev/sde [SAT], 2288 Offline uncorrectable sectors
1137Sep 20 05:42:04 cloudcephosd1032 smart_failure[3783557]: Device: /dev/sdh [SAT], 2152 Offline uncorrectable sectors
1138Sep 20 08:42:04 cloudcephosd1032 smart_failure[3851079]: Device: /dev/sdf [SAT], 1528 Offline uncorrectable sectors
1139Sep 20 12:42:04 cloudcephosd1032 smart_failure[3940758]: Device: /dev/sdj [SAT], 1368 Offline uncorrectable sectors
1140Sep 20 19:42:04 cloudcephosd1032 smart_failure[4097554]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1141Sep 20 19:42:04 cloudcephosd1032 smart_failure[4097604]: Device: /dev/sdi [SAT], 1352 Offline uncorrectable sectors
1142Sep 20 22:12:04 cloudcephosd1032 smart_failure[4153928]: Device: /dev/sdg [SAT], 1272 Offline uncorrectable sectors
1143Sep 21 00:12:04 cloudcephosd1032 smart_failure[4899]: Device: /dev/sdc [SAT], 1728 Offline uncorrectable sectors
1144Sep 21 04:12:04 cloudcephosd1032 smart_failure[94659]: Device: /dev/sde [SAT], 2320 Offline uncorrectable sectors
1145Sep 21 05:42:04 cloudcephosd1032 smart_failure[127895]: Device: /dev/sdh [SAT], 2160 Offline uncorrectable sectors
1146Sep 21 08:42:04 cloudcephosd1032 smart_failure[194822]: Device: /dev/sdf [SAT], 1536 Offline uncorrectable sectors
1147Sep 21 12:42:04 cloudcephosd1032 smart_failure[285358]: Device: /dev/sdj [SAT], 1368 Offline uncorrectable sectors
1148Sep 21 19:42:04 cloudcephosd1032 smart_failure[442823]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1149Sep 21 19:42:04 cloudcephosd1032 smart_failure[442872]: Device: /dev/sdi [SAT], 1352 Offline uncorrectable sectors
1150Sep 21 22:12:05 cloudcephosd1032 smart_failure[498681]: Device: /dev/sdg [SAT], 1272 Offline uncorrectable sectors
1151Sep 22 00:12:04 cloudcephosd1032 smart_failure[543333]: Device: /dev/sdc [SAT], 1728 Offline uncorrectable sectors
1152Sep 22 04:12:04 cloudcephosd1032 smart_failure[632699]: Device: /dev/sde [SAT], 2336 Offline uncorrectable sectors
1153Sep 22 05:42:04 cloudcephosd1032 smart_failure[665967]: Device: /dev/sdh [SAT], 2160 Offline uncorrectable sectors
1154Sep 22 08:42:04 cloudcephosd1032 smart_failure[733436]: Device: /dev/sdf [SAT], 1552 Offline uncorrectable sectors
1155Sep 22 12:42:04 cloudcephosd1032 smart_failure[822538]: Device: /dev/sdj [SAT], 1376 Offline uncorrectable sectors
1156Sep 22 19:42:04 cloudcephosd1032 smart_failure[979623]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1157Sep 22 19:42:04 cloudcephosd1032 smart_failure[979672]: Device: /dev/sdi [SAT], 1368 Offline uncorrectable sectors
1158Sep 22 22:12:04 cloudcephosd1032 smart_failure[1036144]: Device: /dev/sdg [SAT], 1272 Offline uncorrectable sectors
1159Sep 23 00:12:04 cloudcephosd1032 smart_failure[1080983]: Device: /dev/sdc [SAT], 1728 Offline uncorrectable sectors
1160Sep 23 04:12:04 cloudcephosd1032 smart_failure[1171419]: Device: /dev/sde [SAT], 2336 Offline uncorrectable sectors
1161Sep 23 05:42:04 cloudcephosd1032 smart_failure[1205037]: Device: /dev/sdh [SAT], 2160 Offline uncorrectable sectors
1162Sep 23 08:42:04 cloudcephosd1032 smart_failure[1272525]: Device: /dev/sdf [SAT], 1560 Offline uncorrectable sectors
1163Sep 23 12:42:04 cloudcephosd1032 smart_failure[1362435]: Device: /dev/sdj [SAT], 1392 Offline uncorrectable sectors
1164Sep 23 19:42:04 cloudcephosd1032 smart_failure[1520454]: Device: /dev/sdd [SAT], 2592 Offline uncorrectable sectors
1165Sep 23 19:42:04 cloudcephosd1032 smart_failure[1520503]: Device: /dev/sdi [SAT], 1368 Offline uncorrectable sectors
1166Sep 23 22:12:04 cloudcephosd1032 smart_failure[1577052]: Device: /dev/sdg [SAT], 1272 Offline uncorrectable sectors
1167Sep 24 00:12:05 cloudcephosd1032 smart_failure[1622115]: Device: /dev/sdc [SAT], 1736 Offline uncorrectable sectors
1168Sep 24 04:12:04 cloudcephosd1032 smart_failure[1712071]: Device: /dev/sde [SAT], 2360 Offline uncorrectable sectors
1169Sep 24 05:42:04 cloudcephosd1032 smart_failure[1745802]: Device: /dev/sdh [SAT], 2168 Offline uncorrectable sectors
1170Sep 24 08:42:04 cloudcephosd1032 smart_failure[1813701]: Device: /dev/sdf [SAT], 1576 Offline uncorrectable sectors
1171Sep 24 12:42:04 cloudcephosd1032 smart_failure[1903188]: Device: /dev/sdj [SAT], 1392 Offline uncorrectable sectors
1172Sep 24 19:42:04 cloudcephosd1032 smart_failure[2060080]: Device: /dev/sdd [SAT], 2600 Offline uncorrectable sectors
1173Sep 24 19:42:04 cloudcephosd1032 smart_failure[2060134]: Device: /dev/sdi [SAT], 1392 Offline uncorrectable sectors
1174Sep 24 22:12:04 cloudcephosd1032 smart_failure[2116760]: Device: /dev/sdg [SAT], 1280 Offline uncorrectable sectors
1175Sep 25 00:42:04 cloudcephosd1032 smart_failure[2173376]: Device: /dev/sdc [SAT], 1744 Offline uncorrectable sectors
1176Sep 25 04:12:04 cloudcephosd1032 smart_failure[2252264]: Device: /dev/sde [SAT], 2384 Offline uncorrectable sectors
1177Sep 25 05:42:04 cloudcephosd1032 smart_failure[2285864]: Device: /dev/sdh [SAT], 2184 Offline uncorrectable sectors
1178Sep 25 08:42:04 cloudcephosd1032 smart_failure[2353430]: Device: /dev/sdf [SAT], 1600 Offline uncorrectable sectors
1179Sep 25 12:42:04 cloudcephosd1032 smart_failure[2443251]: Device: /dev/sdj [SAT], 1400 Offline uncorrectable sectors
1180Sep 25 19:42:04 cloudcephosd1032 smart_failure[2600604]: Device: /dev/sdd [SAT], 2600 Offline uncorrectable sectors
1181Sep 25 19:42:04 cloudcephosd1032 smart_failure[2600623]: Device: /dev/sdi [SAT], 1400 Offline uncorrectable sectors
1182Sep 25 22:12:04 cloudcephosd1032 smart_failure[2656972]: Device: /dev/sdg [SAT], 1280 Offline uncorrectable sectors
1183Sep 26 00:42:04 cloudcephosd1032 smart_failure[2713235]: Device: /dev/sdc [SAT], 1752 Offline uncorrectable sectors
1184Sep 26 04:12:04 cloudcephosd1032 smart_failure[2792148]: Device: /dev/sde [SAT], 2400 Offline uncorrectable sectors
1185Sep 26 05:42:04 cloudcephosd1032 smart_failure[2825497]: Device: /dev/sdh [SAT], 2192 Offline uncorrectable sectors
1186Sep 26 08:42:04 cloudcephosd1032 smart_failure[2893091]: Device: /dev/sdf [SAT], 1600 Offline uncorrectable sectors
1187Sep 26 12:42:04 cloudcephosd1032 smart_failure[2983126]: Device: /dev/sdj [SAT], 1408 Offline uncorrectable sectors
1188Sep 26 19:42:04 cloudcephosd1032 smart_failure[3140441]: Device: /dev/sdd [SAT], 2600 Offline uncorrectable sectors
1189Sep 26 19:42:04 cloudcephosd1032 smart_failure[3140460]: Device: /dev/sdi [SAT], 1400 Offline uncorrectable sectors
1190Sep 26 22:12:04 cloudcephosd1032 smart_failure[3196936]: Device: /dev/sdg [SAT], 1296 Offline uncorrectable sectors
1191Sep 27 00:42:04 cloudcephosd1032 smart_failure[3253404]: Device: /dev/sdc [SAT], 1776 Offline uncorrectable sectors
1192Sep 27 04:12:04 cloudcephosd1032 smart_failure[3332276]: Device: /dev/sde [SAT], 2400 Offline uncorrectable sectors
1193Sep 27 05:42:04 cloudcephosd1032 smart_failure[3366030]: Device: /dev/sdh [SAT], 2208 Offline uncorrectable sectors
1194Sep 27 08:42:05 cloudcephosd1032 smart_failure[3433173]: Device: /dev/sdf [SAT], 1608 Offline uncorrectable sectors
1195Sep 27 12:42:04 cloudcephosd1032 smart_failure[3523493]: Device: /dev/sdj [SAT], 1408 Offline uncorrectable sectors
1196Sep 27 19:42:04 cloudcephosd1032 smart_failure[3681055]: Device: /dev/sdd [SAT], 2616 Offline uncorrectable sectors
1197Sep 27 19:42:04 cloudcephosd1032 smart_failure[3681104]: Device: /dev/sdi [SAT], 1416 Offline uncorrectable sectors
1198Sep 27 22:12:04 cloudcephosd1032 smart_failure[3737941]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
1199Sep 28 00:42:05 cloudcephosd1032 smart_failure[3794507]: Device: /dev/sdc [SAT], 1776 Offline uncorrectable sectors
1200Sep 28 04:12:04 cloudcephosd1032 smart_failure[3873478]: Device: /dev/sde [SAT], 2408 Offline uncorrectable sectors
1201Sep 28 05:42:04 cloudcephosd1032 smart_failure[3907307]: Device: /dev/sdh [SAT], 2224 Offline uncorrectable sectors
1202Sep 28 09:12:04 cloudcephosd1032 smart_failure[3986437]: Device: /dev/sdf [SAT], 1608 Offline uncorrectable sectors
1203Sep 28 12:42:04 cloudcephosd1032 smart_failure[4064786]: Device: /dev/sdj [SAT], 1408 Offline uncorrectable sectors
1204Sep 28 19:42:04 cloudcephosd1032 smart_failure[29530]: Device: /dev/sdd [SAT], 2616 Offline uncorrectable sectors
1205Sep 28 19:42:04 cloudcephosd1032 smart_failure[29549]: Device: /dev/sdi [SAT], 1416 Offline uncorrectable sectors
1206Sep 28 22:12:04 cloudcephosd1032 smart_failure[85745]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
1207Sep 29 01:12:04 cloudcephosd1032 smart_failure[153250]: Device: /dev/sdc [SAT], 1792 Offline uncorrectable sectors
1208Sep 29 04:12:04 cloudcephosd1032 smart_failure[220731]: Device: /dev/sde [SAT], 2408 Offline uncorrectable sectors
1209Sep 29 05:42:04 cloudcephosd1032 smart_failure[254157]: Device: /dev/sdh [SAT], 2224 Offline uncorrectable sectors
1210Sep 29 09:12:04 cloudcephosd1032 smart_failure[333099]: Device: /dev/sdf [SAT], 1624 Offline uncorrectable sectors
1211Sep 29 12:42:04 cloudcephosd1032 smart_failure[411879]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1212Sep 29 19:42:04 cloudcephosd1032 smart_failure[569430]: Device: /dev/sdd [SAT], 2624 Offline uncorrectable sectors
1213Sep 29 19:42:04 cloudcephosd1032 smart_failure[569480]: Device: /dev/sdi [SAT], 1416 Offline uncorrectable sectors
1214Sep 29 22:12:04 cloudcephosd1032 smart_failure[625884]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
1215Sep 30 01:12:04 cloudcephosd1032 smart_failure[693119]: Device: /dev/sdc [SAT], 1800 Offline uncorrectable sectors
1216Sep 30 04:12:04 cloudcephosd1032 smart_failure[760736]: Device: /dev/sde [SAT], 2408 Offline uncorrectable sectors
1217Sep 30 05:42:04 cloudcephosd1032 smart_failure[794321]: Device: /dev/sdh [SAT], 2240 Offline uncorrectable sectors
1218Sep 30 09:12:04 cloudcephosd1032 smart_failure[873529]: Device: /dev/sdf [SAT], 1624 Offline uncorrectable sectors
1219Sep 30 12:42:04 cloudcephosd1032 smart_failure[952078]: Device: /dev/sdj [SAT], 1416 Offline uncorrectable sectors
1220Sep 30 19:42:04 cloudcephosd1032 smart_failure[1109789]: Device: /dev/sdd [SAT], 2624 Offline uncorrectable sectors
1221Sep 30 19:42:04 cloudcephosd1032 smart_failure[1109838]: Device: /dev/sdi [SAT], 1424 Offline uncorrectable sectors
1222Sep 30 22:12:04 cloudcephosd1032 smart_failure[1166252]: Device: /dev/sdg [SAT], 1320 Offline uncorrectable sectors
1223===== NODE GROUP =====
1224(1) cloudcephosd1033.eqiad.wmnet
1225----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
1226Oct 01 01:25:02 cloudcephosd1033 smart_failure[814502]: Device: /dev/sdg [SAT], 880 Offline uncorrectable sectors
1227Oct 01 01:55:02 cloudcephosd1033 smart_failure[825672]: Device: /dev/sdd [SAT], 1272 Offline uncorrectable sectors
1228Oct 01 01:55:02 cloudcephosd1033 smart_failure[825725]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
1229Oct 01 06:25:02 cloudcephosd1033 smart_failure[927106]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1230Oct 01 07:25:02 cloudcephosd1033 smart_failure[949698]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1231Oct 01 07:55:02 cloudcephosd1033 smart_failure[960920]: Device: /dev/sdj [SAT], 1944 Offline uncorrectable sectors
1232Oct 01 09:25:02 cloudcephosd1033 smart_failure[994870]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
1233Oct 01 16:25:02 cloudcephosd1033 smart_failure[1152270]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1234Oct 02 01:25:02 cloudcephosd1033 smart_failure[1353822]: Device: /dev/sdg [SAT], 896 Offline uncorrectable sectors
1235Oct 02 01:55:03 cloudcephosd1033 smart_failure[1365502]: Device: /dev/sdd [SAT], 1272 Offline uncorrectable sectors
1236Oct 02 01:55:03 cloudcephosd1033 smart_failure[1365552]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
1237Oct 02 06:25:02 cloudcephosd1033 smart_failure[1466876]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1238Oct 02 07:25:02 cloudcephosd1033 smart_failure[1489459]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1239Oct 02 07:55:02 cloudcephosd1033 smart_failure[1500578]: Device: /dev/sdj [SAT], 1952 Offline uncorrectable sectors
1240Oct 02 09:25:02 cloudcephosd1033 smart_failure[1533891]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors (changed +8)
1241Oct 02 16:25:02 cloudcephosd1033 smart_failure[1691106]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1242Oct 03 01:25:02 cloudcephosd1033 smart_failure[1884652]: Device: /dev/sdg [SAT], 904 Offline uncorrectable sectors
1243Oct 03 02:25:02 cloudcephosd1033 smart_failure[1907235]: Device: /dev/sdd [SAT], 1272 Offline uncorrectable sectors
1244Oct 03 02:25:02 cloudcephosd1033 smart_failure[1907279]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
1245Oct 03 06:25:02 cloudcephosd1033 smart_failure[1997297]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1246Oct 03 07:25:02 cloudcephosd1033 smart_failure[2020077]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1247Oct 03 07:55:02 cloudcephosd1033 smart_failure[2031006]: Device: /dev/sdj [SAT], 1952 Offline uncorrectable sectors
1248Oct 03 09:25:02 cloudcephosd1033 smart_failure[2064091]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
1249Oct 03 16:25:02 cloudcephosd1033 smart_failure[2220836]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1250Oct 04 01:25:02 cloudcephosd1033 smart_failure[2422931]: Device: /dev/sdg [SAT], 904 Offline uncorrectable sectors
1251Oct 04 02:25:03 cloudcephosd1033 smart_failure[2445547]: Device: /dev/sdd [SAT], 1272 Offline uncorrectable sectors
1252Oct 04 02:25:03 cloudcephosd1033 smart_failure[2445596]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1253Oct 04 06:25:02 cloudcephosd1033 smart_failure[2535360]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1254Oct 04 07:25:02 cloudcephosd1033 smart_failure[2558904]: Device: /dev/sdi [SAT], 848 Offline uncorrectable sectors
1255Oct 04 07:55:02 cloudcephosd1033 smart_failure[2569857]: Device: /dev/sdj [SAT], 1960 Offline uncorrectable sectors
1256Oct 04 09:25:02 cloudcephosd1033 smart_failure[2603814]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
1257Oct 04 16:25:02 cloudcephosd1033 smart_failure[2761945]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1258Oct 05 01:55:01 cloudcephosd1033 smart_failure[2975550]: Device: /dev/sdg [SAT], 904 Offline uncorrectable sectors
1259Oct 05 02:55:01 cloudcephosd1033 smart_failure[2997860]: Device: /dev/sdd [SAT], 1272 Offline uncorrectable sectors
1260Oct 05 02:55:01 cloudcephosd1033 smart_failure[2997909]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1261Oct 05 06:55:01 cloudcephosd1033 smart_failure[3088061]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1262Oct 05 07:55:01 cloudcephosd1033 smart_failure[3110400]: Device: /dev/sdi [SAT], 856 Offline uncorrectable sectors
1263Oct 05 08:25:01 cloudcephosd1033 smart_failure[3121876]: Device: /dev/sdj [SAT], 1968 Offline uncorrectable sectors
1264Oct 05 09:55:01 cloudcephosd1033 smart_failure[3155525]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
1265Oct 05 16:55:01 cloudcephosd1033 smart_failure[3325461]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1266Oct 06 01:55:01 cloudcephosd1033 smart_failure[3527566]: Device: /dev/sdg [SAT], 904 Offline uncorrectable sectors
1267Oct 06 02:55:01 cloudcephosd1033 smart_failure[3550252]: Device: /dev/sdd [SAT], 1280 Offline uncorrectable sectors
1268Oct 06 02:55:01 cloudcephosd1033 smart_failure[3550271]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1269Oct 06 06:55:01 cloudcephosd1033 smart_failure[3640917]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1270Oct 06 07:55:01 cloudcephosd1033 smart_failure[3663871]: Device: /dev/sdi [SAT], 856 Offline uncorrectable sectors
1271Oct 06 08:25:01 cloudcephosd1033 smart_failure[3675312]: Device: /dev/sdj [SAT], 1976 Offline uncorrectable sectors
1272Oct 06 09:55:01 cloudcephosd1033 smart_failure[3708949]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
1273Oct 06 16:55:01 cloudcephosd1033 smart_failure[3865656]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1274Oct 07 01:55:01 cloudcephosd1033 smart_failure[4067652]: Device: /dev/sdg [SAT], 904 Offline uncorrectable sectors
1275Oct 07 02:55:01 cloudcephosd1033 smart_failure[4089988]: Device: /dev/sdd [SAT], 1280 Offline uncorrectable sectors
1276Oct 07 02:55:01 cloudcephosd1033 smart_failure[4090038]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1277Oct 07 06:55:01 cloudcephosd1033 smart_failure[4179888]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1278Oct 07 07:55:01 cloudcephosd1033 smart_failure[8663]: Device: /dev/sdi [SAT], 864 Offline uncorrectable sectors
1279Oct 07 08:25:01 cloudcephosd1033 smart_failure[20047]: Device: /dev/sdj [SAT], 1976 Offline uncorrectable sectors
1280Oct 07 09:55:01 cloudcephosd1033 smart_failure[53683]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1281Oct 07 16:55:01 cloudcephosd1033 smart_failure[210845]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1282Oct 08 01:55:01 cloudcephosd1033 smart_failure[413319]: Device: /dev/sdg [SAT], 920 Offline uncorrectable sectors
1283Oct 08 02:55:01 cloudcephosd1033 smart_failure[436049]: Device: /dev/sdd [SAT], 1280 Offline uncorrectable sectors
1284Oct 08 02:55:01 cloudcephosd1033 smart_failure[436108]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1285Oct 08 06:55:01 cloudcephosd1033 smart_failure[526002]: Device: /dev/sdb [SAT], 632 Offline uncorrectable sectors
1286Oct 08 07:55:01 cloudcephosd1033 smart_failure[548460]: Device: /dev/sdi [SAT], 864 Offline uncorrectable sectors
1287Oct 08 08:25:01 cloudcephosd1033 smart_failure[559780]: Device: /dev/sdj [SAT], 1984 Offline uncorrectable sectors
1288Oct 08 09:55:01 cloudcephosd1033 smart_failure[593143]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1289Oct 08 16:55:01 cloudcephosd1033 smart_failure[750503]: Device: /dev/sdf [SAT], 1016 Offline uncorrectable sectors
1290Oct 09 01:55:01 cloudcephosd1033 smart_failure[953058]: Device: /dev/sdg [SAT], 920 Offline uncorrectable sectors
1291Oct 09 02:55:01 cloudcephosd1033 smart_failure[975661]: Device: /dev/sdd [SAT], 1280 Offline uncorrectable sectors
1292Oct 09 02:55:01 cloudcephosd1033 smart_failure[975710]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1293Oct 09 06:55:01 cloudcephosd1033 smart_failure[1065243]: Device: /dev/sdb [SAT], 640 Offline uncorrectable sectors
1294Oct 09 07:55:01 cloudcephosd1033 smart_failure[1087308]: Device: /dev/sdi [SAT], 864 Offline uncorrectable sectors
1295Oct 09 08:25:01 cloudcephosd1033 smart_failure[1098863]: Device: /dev/sdj [SAT], 1984 Offline uncorrectable sectors
1296Oct 09 09:55:01 cloudcephosd1033 smart_failure[1132309]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1297Oct 09 16:55:01 cloudcephosd1033 smart_failure[1289167]: Device: /dev/sdf [SAT], 1016 Offline uncorrectable sectors
1298Oct 10 01:55:01 cloudcephosd1033 smart_failure[1492573]: Device: /dev/sdg [SAT], 920 Offline uncorrectable sectors
1299Oct 10 02:55:01 cloudcephosd1033 smart_failure[1514898]: Device: /dev/sdd [SAT], 1296 Offline uncorrectable sectors
1300Oct 10 02:55:01 cloudcephosd1033 smart_failure[1514947]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1301Oct 10 06:55:01 cloudcephosd1033 smart_failure[1604766]: Device: /dev/sdb [SAT], 640 Offline uncorrectable sectors
1302Oct 10 07:55:01 cloudcephosd1033 smart_failure[1627398]: Device: /dev/sdi [SAT], 864 Offline uncorrectable sectors
1303Oct 10 08:25:01 cloudcephosd1033 smart_failure[1639086]: Device: /dev/sdj [SAT], 1992 Offline uncorrectable sectors
1304Oct 10 09:55:01 cloudcephosd1033 smart_failure[1672544]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1305Oct 10 16:55:01 cloudcephosd1033 smart_failure[1829843]: Device: /dev/sdf [SAT], 1024 Offline uncorrectable sectors
1306Oct 11 01:55:01 cloudcephosd1033 smart_failure[2032877]: Device: /dev/sdg [SAT], 928 Offline uncorrectable sectors
1307Oct 11 02:55:01 cloudcephosd1033 smart_failure[2055153]: Device: /dev/sdd [SAT], 1304 Offline uncorrectable sectors
1308Oct 11 02:55:01 cloudcephosd1033 smart_failure[2055204]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
1309Oct 11 06:55:01 cloudcephosd1033 smart_failure[2145455]: Device: /dev/sdb [SAT], 656 Offline uncorrectable sectors
1310Oct 11 07:55:01 cloudcephosd1033 smart_failure[2167965]: Device: /dev/sdi [SAT], 864 Offline uncorrectable sectors
1311Oct 11 08:25:01 cloudcephosd1033 smart_failure[2179284]: Device: /dev/sdj [SAT], 2000 Offline uncorrectable sectors
1312Oct 11 09:55:01 cloudcephosd1033 smart_failure[2213336]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1313Sep 14 15:55:02 cloudcephosd1033 smart_failure[323250]: Device: /dev/sdf [SAT], 992 Offline uncorrectable sectors
1314Sep 15 00:25:02 cloudcephosd1033 smart_failure[515104]: Device: /dev/sdg [SAT], 840 Offline uncorrectable sectors
1315Sep 15 01:55:02 cloudcephosd1033 smart_failure[548913]: Device: /dev/sdd [SAT], 1176 Offline uncorrectable sectors
1316Sep 15 01:55:02 cloudcephosd1033 smart_failure[548962]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
1317Sep 15 06:25:02 cloudcephosd1033 smart_failure[651273]: Device: /dev/sdb [SAT], 560 Offline uncorrectable sectors
1318Sep 15 07:25:02 cloudcephosd1033 smart_failure[673713]: Device: /dev/sdi [SAT], 816 Offline uncorrectable sectors
1319Sep 15 07:25:02 cloudcephosd1033 smart_failure[673761]: Device: /dev/sdj [SAT], 1712 Offline uncorrectable sectors
1320Sep 15 08:55:02 cloudcephosd1033 smart_failure[707287]: Device: /dev/sde [SAT], 896 Offline uncorrectable sectors
1321Sep 15 15:55:02 cloudcephosd1033 smart_failure[865992]: Device: /dev/sdf [SAT], 992 Offline uncorrectable sectors
1322Sep 16 00:25:02 cloudcephosd1033 smart_failure[1058928]: Device: /dev/sdg [SAT], 840 Offline uncorrectable sectors
1323Sep 16 01:55:02 cloudcephosd1033 smart_failure[1092631]: Device: /dev/sdd [SAT], 1184 Offline uncorrectable sectors
1324Sep 16 01:55:02 cloudcephosd1033 smart_failure[1092681]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
1325Sep 16 06:25:02 cloudcephosd1033 smart_failure[1195011]: Device: /dev/sdb [SAT], 560 Offline uncorrectable sectors
1326Sep 16 07:25:03 cloudcephosd1033 smart_failure[1217646]: Device: /dev/sdi [SAT], 816 Offline uncorrectable sectors
1327Sep 16 07:25:03 cloudcephosd1033 smart_failure[1217688]: Device: /dev/sdj [SAT], 1712 Offline uncorrectable sectors
1328Sep 16 08:55:02 cloudcephosd1033 smart_failure[1251216]: Device: /dev/sde [SAT], 904 Offline uncorrectable sectors
1329Sep 16 15:55:02 cloudcephosd1033 smart_failure[1410025]: Device: /dev/sdf [SAT], 992 Offline uncorrectable sectors
1330Sep 17 00:25:03 cloudcephosd1033 smart_failure[1602858]: Device: /dev/sdg [SAT], 840 Offline uncorrectable sectors
1331Sep 17 01:55:02 cloudcephosd1033 smart_failure[1636707]: Device: /dev/sdd [SAT], 1184 Offline uncorrectable sectors
1332Sep 17 01:55:02 cloudcephosd1033 smart_failure[1636756]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
1333Sep 17 06:25:02 cloudcephosd1033 smart_failure[1738885]: Device: /dev/sdb [SAT], 560 Offline uncorrectable sectors
1334Sep 17 07:25:02 cloudcephosd1033 smart_failure[1761486]: Device: /dev/sdi [SAT], 816 Offline uncorrectable sectors
1335Sep 17 07:55:02 cloudcephosd1033 smart_failure[1772813]: Device: /dev/sdj [SAT], 1736 Offline uncorrectable sectors
1336Sep 17 08:55:02 cloudcephosd1033 smart_failure[1795380]: Device: /dev/sde [SAT], 912 Offline uncorrectable sectors
1337Sep 17 15:55:02 cloudcephosd1033 smart_failure[1953505]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors (changed +8)
1338Sep 18 00:55:02 cloudcephosd1033 smart_failure[2157372]: Device: /dev/sdg [SAT], 840 Offline uncorrectable sectors
1339Sep 18 01:55:02 cloudcephosd1033 smart_failure[2179870]: Device: /dev/sdd [SAT], 1184 Offline uncorrectable sectors
1340Sep 18 01:55:02 cloudcephosd1033 smart_failure[2179919]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
1341Sep 18 06:25:02 cloudcephosd1033 smart_failure[2281714]: Device: /dev/sdb [SAT], 560 Offline uncorrectable sectors
1342Sep 18 07:25:02 cloudcephosd1033 smart_failure[2304543]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1343Sep 18 07:55:02 cloudcephosd1033 smart_failure[2315733]: Device: /dev/sdj [SAT], 1736 Offline uncorrectable sectors
1344Sep 18 08:55:03 cloudcephosd1033 smart_failure[2337979]: Device: /dev/sde [SAT], 928 Offline uncorrectable sectors
1345Sep 18 15:55:02 cloudcephosd1033 smart_failure[2496080]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1346Sep 19 00:55:02 cloudcephosd1033 smart_failure[2700761]: Device: /dev/sdg [SAT], 848 Offline uncorrectable sectors
1347Sep 19 01:55:02 cloudcephosd1033 smart_failure[2723450]: Device: /dev/sdd [SAT], 1192 Offline uncorrectable sectors
1348Sep 19 01:55:03 cloudcephosd1033 smart_failure[2723499]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
1349Sep 19 06:25:02 cloudcephosd1033 smart_failure[2826033]: Device: /dev/sdb [SAT], 568 Offline uncorrectable sectors
1350Sep 19 07:25:02 cloudcephosd1033 smart_failure[2848898]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1351Sep 19 07:55:02 cloudcephosd1033 smart_failure[2860173]: Device: /dev/sdj [SAT], 1744 Offline uncorrectable sectors
1352Sep 19 09:25:02 cloudcephosd1033 smart_failure[2894097]: Device: /dev/sde [SAT], 928 Offline uncorrectable sectors
1353Sep 19 15:55:02 cloudcephosd1033 smart_failure[3041263]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1354Sep 20 00:55:02 cloudcephosd1033 smart_failure[3246875]: Device: /dev/sdg [SAT], 848 Offline uncorrectable sectors
1355Sep 20 01:55:02 cloudcephosd1033 smart_failure[3269265]: Device: /dev/sdd [SAT], 1200 Offline uncorrectable sectors
1356Sep 20 01:55:02 cloudcephosd1033 smart_failure[3269314]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
1357Sep 20 06:25:02 cloudcephosd1033 smart_failure[3370605]: Device: /dev/sdb [SAT], 568 Offline uncorrectable sectors
1358Sep 20 07:25:02 cloudcephosd1033 smart_failure[3393254]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1359Sep 20 07:55:02 cloudcephosd1033 smart_failure[3404451]: Device: /dev/sdj [SAT], 1744 Offline uncorrectable sectors
1360Sep 20 09:25:02 cloudcephosd1033 smart_failure[3438451]: Device: /dev/sde [SAT], 944 Offline uncorrectable sectors
1361Sep 20 15:55:02 cloudcephosd1033 smart_failure[3585074]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1362Sep 21 00:55:02 cloudcephosd1033 smart_failure[3787202]: Device: /dev/sdg [SAT], 848 Offline uncorrectable sectors
1363Sep 21 01:55:02 cloudcephosd1033 smart_failure[3810023]: Device: /dev/sdd [SAT], 1200 Offline uncorrectable sectors
1364Sep 21 01:55:02 cloudcephosd1033 smart_failure[3810042]: Device: /dev/sdh [SAT], 1384 Offline uncorrectable sectors
1365Sep 21 06:25:02 cloudcephosd1033 smart_failure[3910607]: Device: /dev/sdb [SAT], 616 Offline uncorrectable sectors
1366Sep 21 07:25:02 cloudcephosd1033 smart_failure[3933134]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1367Sep 21 07:55:02 cloudcephosd1033 smart_failure[3943984]: Device: /dev/sdj [SAT], 1752 Offline uncorrectable sectors
1368Sep 21 09:25:02 cloudcephosd1033 smart_failure[3977881]: Device: /dev/sde [SAT], 944 Offline uncorrectable sectors
1369Sep 21 15:55:02 cloudcephosd1033 smart_failure[4124739]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1370Sep 22 00:55:02 cloudcephosd1033 smart_failure[133164]: Device: /dev/sdg [SAT], 848 Offline uncorrectable sectors
1371Sep 22 01:55:02 cloudcephosd1033 smart_failure[155447]: Device: /dev/sdd [SAT], 1208 Offline uncorrectable sectors
1372Sep 22 01:55:02 cloudcephosd1033 smart_failure[155496]: Device: /dev/sdh [SAT], 1384 Offline uncorrectable sectors
1373Sep 22 06:25:02 cloudcephosd1033 smart_failure[256561]: Device: /dev/sdb [SAT], 616 Offline uncorrectable sectors
1374Sep 22 07:25:02 cloudcephosd1033 smart_failure[279095]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1375Sep 22 07:55:02 cloudcephosd1033 smart_failure[290133]: Device: /dev/sdj [SAT], 1776 Offline uncorrectable sectors
1376Sep 22 09:25:02 cloudcephosd1033 smart_failure[324219]: Device: /dev/sde [SAT], 944 Offline uncorrectable sectors
1377Sep 22 15:55:02 cloudcephosd1033 smart_failure[469877]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1378Sep 23 00:55:02 cloudcephosd1033 smart_failure[673043]: Device: /dev/sdg [SAT], 856 Offline uncorrectable sectors
1379Sep 23 01:55:02 cloudcephosd1033 smart_failure[695606]: Device: /dev/sdd [SAT], 1208 Offline uncorrectable sectors
1380Sep 23 01:55:02 cloudcephosd1033 smart_failure[695625]: Device: /dev/sdh [SAT], 1384 Offline uncorrectable sectors
1381Sep 23 06:25:02 cloudcephosd1033 smart_failure[796684]: Device: /dev/sdb [SAT], 616 Offline uncorrectable sectors
1382Sep 23 07:25:02 cloudcephosd1033 smart_failure[819443]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1383Sep 23 07:55:02 cloudcephosd1033 smart_failure[830876]: Device: /dev/sdj [SAT], 1776 Offline uncorrectable sectors
1384Sep 23 09:25:02 cloudcephosd1033 smart_failure[864463]: Device: /dev/sde [SAT], 944 Offline uncorrectable sectors
1385Sep 23 15:55:02 cloudcephosd1033 smart_failure[1010339]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1386Sep 24 00:55:02 cloudcephosd1033 smart_failure[1212090]: Device: /dev/sdg [SAT], 864 Offline uncorrectable sectors
1387Sep 24 01:55:02 cloudcephosd1033 smart_failure[1235114]: Device: /dev/sdd [SAT], 1208 Offline uncorrectable sectors
1388Sep 24 01:55:02 cloudcephosd1033 smart_failure[1235133]: Device: /dev/sdh [SAT], 1384 Offline uncorrectable sectors
1389Sep 24 06:25:02 cloudcephosd1033 smart_failure[1335975]: Device: /dev/sdb [SAT], 616 Offline uncorrectable sectors
1390Sep 24 07:25:02 cloudcephosd1033 smart_failure[1358711]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1391Sep 24 07:55:02 cloudcephosd1033 smart_failure[1370715]: Device: /dev/sdj [SAT], 1776 Offline uncorrectable sectors
1392Sep 24 09:25:02 cloudcephosd1033 smart_failure[1404175]: Device: /dev/sde [SAT], 952 Offline uncorrectable sectors
1393Sep 24 15:55:02 cloudcephosd1033 smart_failure[1549854]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1394Sep 25 00:55:02 cloudcephosd1033 smart_failure[1752421]: Device: /dev/sdg [SAT], 864 Offline uncorrectable sectors
1395Sep 25 01:55:02 cloudcephosd1033 smart_failure[1774935]: Device: /dev/sdd [SAT], 1208 Offline uncorrectable sectors
1396Sep 25 01:55:02 cloudcephosd1033 smart_failure[1774954]: Device: /dev/sdh [SAT], 1392 Offline uncorrectable sectors
1397Sep 25 06:25:02 cloudcephosd1033 smart_failure[1875873]: Device: /dev/sdb [SAT], 616 Offline uncorrectable sectors
1398Sep 25 07:25:02 cloudcephosd1033 smart_failure[1898508]: Device: /dev/sdi [SAT], 824 Offline uncorrectable sectors
1399Sep 25 07:55:02 cloudcephosd1033 smart_failure[1909730]: Device: /dev/sdj [SAT], 1808 Offline uncorrectable sectors
1400Sep 25 09:25:02 cloudcephosd1033 smart_failure[1943546]: Device: /dev/sde [SAT], 952 Offline uncorrectable sectors
1401Sep 25 15:55:02 cloudcephosd1033 smart_failure[2089145]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1402Sep 26 00:55:02 cloudcephosd1033 smart_failure[2292066]: Device: /dev/sdg [SAT], 872 Offline uncorrectable sectors
1403Sep 26 01:55:02 cloudcephosd1033 smart_failure[2314794]: Device: /dev/sdd [SAT], 1216 Offline uncorrectable sectors
1404Sep 26 01:55:02 cloudcephosd1033 smart_failure[2314842]: Device: /dev/sdh [SAT], 1392 Offline uncorrectable sectors
1405Sep 26 06:25:02 cloudcephosd1033 smart_failure[2416678]: Device: /dev/sdb [SAT], 616 Offline uncorrectable sectors
1406Sep 26 07:25:02 cloudcephosd1033 smart_failure[2439331]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1407Sep 26 07:55:02 cloudcephosd1033 smart_failure[2450146]: Device: /dev/sdj [SAT], 1816 Offline uncorrectable sectors
1408Sep 26 09:25:02 cloudcephosd1033 smart_failure[2484288]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
1409Sep 26 15:55:02 cloudcephosd1033 smart_failure[2631805]: Device: /dev/sdf [SAT], 1000 Offline uncorrectable sectors
1410Sep 27 00:55:02 cloudcephosd1033 smart_failure[2835273]: Device: /dev/sdg [SAT], 880 Offline uncorrectable sectors
1411Sep 27 01:55:02 cloudcephosd1033 smart_failure[2857617]: Device: /dev/sdd [SAT], 1240 Offline uncorrectable sectors
1412Sep 27 01:55:02 cloudcephosd1033 smart_failure[2857670]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors (changed +8)
1413Sep 27 06:25:02 cloudcephosd1033 smart_failure[2959230]: Device: /dev/sdb [SAT], 624 Offline uncorrectable sectors
1414Sep 27 07:25:02 cloudcephosd1033 smart_failure[2981025]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1415Sep 27 07:55:02 cloudcephosd1033 smart_failure[2992267]: Device: /dev/sdj [SAT], 1840 Offline uncorrectable sectors
1416Sep 27 09:25:02 cloudcephosd1033 smart_failure[3025957]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
1417Sep 27 15:55:02 cloudcephosd1033 smart_failure[3172401]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1418Sep 28 00:55:02 cloudcephosd1033 smart_failure[3375359]: Device: /dev/sdg [SAT], 880 Offline uncorrectable sectors
1419Sep 28 01:55:02 cloudcephosd1033 smart_failure[3398050]: Device: /dev/sdd [SAT], 1240 Offline uncorrectable sectors
1420Sep 28 01:55:02 cloudcephosd1033 smart_failure[3398100]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
1421Sep 28 06:25:02 cloudcephosd1033 smart_failure[3499355]: Device: /dev/sdb [SAT], 624 Offline uncorrectable sectors
1422Sep 28 07:25:02 cloudcephosd1033 smart_failure[3521859]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1423Sep 28 07:55:02 cloudcephosd1033 smart_failure[3532809]: Device: /dev/sdj [SAT], 1848 Offline uncorrectable sectors
1424Sep 28 09:25:02 cloudcephosd1033 smart_failure[3566492]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
1425Sep 28 15:55:03 cloudcephosd1033 smart_failure[3712859]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1426Sep 29 00:55:02 cloudcephosd1033 smart_failure[3915773]: Device: /dev/sdg [SAT], 880 Offline uncorrectable sectors
1427Sep 29 01:55:02 cloudcephosd1033 smart_failure[3938195]: Device: /dev/sdd [SAT], 1264 Offline uncorrectable sectors
1428Sep 29 01:55:02 cloudcephosd1033 smart_failure[3938244]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
1429Sep 29 06:25:02 cloudcephosd1033 smart_failure[4039502]: Device: /dev/sdb [SAT], 624 Offline uncorrectable sectors
1430Sep 29 07:25:02 cloudcephosd1033 smart_failure[4062043]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1431Sep 29 07:55:03 cloudcephosd1033 smart_failure[4073222]: Device: /dev/sdj [SAT], 1848 Offline uncorrectable sectors
1432Sep 29 09:25:02 cloudcephosd1033 smart_failure[4107169]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
1433Sep 29 16:25:02 cloudcephosd1033 smart_failure[70229]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1434Sep 30 00:55:03 cloudcephosd1033 smart_failure[261832]: Device: /dev/sdg [SAT], 880 Offline uncorrectable sectors
1435Sep 30 01:55:02 cloudcephosd1033 smart_failure[284042]: Device: /dev/sdd [SAT], 1264 Offline uncorrectable sectors
1436Sep 30 01:55:02 cloudcephosd1033 smart_failure[284093]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
1437Sep 30 06:25:02 cloudcephosd1033 smart_failure[385655]: Device: /dev/sdb [SAT], 624 Offline uncorrectable sectors
1438Sep 30 07:25:02 cloudcephosd1033 smart_failure[408439]: Device: /dev/sdi [SAT], 840 Offline uncorrectable sectors
1439Sep 30 07:55:02 cloudcephosd1033 smart_failure[419651]: Device: /dev/sdj [SAT], 1912 Offline uncorrectable sectors
1440Sep 30 09:25:02 cloudcephosd1033 smart_failure[453691]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
1441Sep 30 16:25:03 cloudcephosd1033 smart_failure[611519]: Device: /dev/sdf [SAT], 1008 Offline uncorrectable sectors
1442===== NODE GROUP =====
1443(1) cloudcephosd1026.eqiad.wmnet
1444----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
1445Oct 01 00:02:04 cloudcephosd1026 smart_failure[1394667]: Device: /dev/sdj [SAT], 1080 Offline uncorrectable sectors
1446Oct 01 12:02:04 cloudcephosd1026 smart_failure[1666741]: Device: /dev/sdi [SAT], 2000 Offline uncorrectable sectors
1447Oct 01 17:02:04 cloudcephosd1026 smart_failure[1780005]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1448Oct 01 18:32:04 cloudcephosd1026 smart_failure[1814575]: Device: /dev/sdd [SAT], 1368 Offline uncorrectable sectors
1449Oct 01 20:32:04 cloudcephosd1026 smart_failure[1859923]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1450Oct 01 22:02:04 cloudcephosd1026 smart_failure[1893688]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1451Oct 01 22:02:04 cloudcephosd1026 smart_failure[1893707]: Device: /dev/sdg [SAT], 2160 Offline uncorrectable sectors
1452Oct 01 23:02:04 cloudcephosd1026 smart_failure[1916317]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
1453Oct 02 00:02:04 cloudcephosd1026 smart_failure[1938917]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1454Oct 02 12:02:04 cloudcephosd1026 smart_failure[2209338]: Device: /dev/sdi [SAT], 2016 Offline uncorrectable sectors
1455Oct 02 17:02:04 cloudcephosd1026 smart_failure[2321953]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1456Oct 02 18:32:04 cloudcephosd1026 smart_failure[2355870]: Device: /dev/sdd [SAT], 1368 Offline uncorrectable sectors
1457Oct 02 20:32:04 cloudcephosd1026 smart_failure[2400856]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1458Oct 02 22:02:04 cloudcephosd1026 smart_failure[2433286]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1459Oct 02 22:02:04 cloudcephosd1026 smart_failure[2433305]: Device: /dev/sdg [SAT], 2168 Offline uncorrectable sectors
1460Oct 02 23:02:04 cloudcephosd1026 smart_failure[2453196]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
1461Oct 03 00:02:04 cloudcephosd1026 smart_failure[2473415]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1462Oct 03 12:02:04 cloudcephosd1026 smart_failure[2740858]: Device: /dev/sdi [SAT], 2016 Offline uncorrectable sectors
1463Oct 03 17:02:04 cloudcephosd1026 smart_failure[2853709]: Device: /dev/sdh [SAT], 888 Offline uncorrectable sectors
1464Oct 03 18:32:04 cloudcephosd1026 smart_failure[2887607]: Device: /dev/sdd [SAT], 1368 Offline uncorrectable sectors
1465Oct 03 20:32:04 cloudcephosd1026 smart_failure[2933005]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1466Oct 03 22:02:04 cloudcephosd1026 smart_failure[2966956]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1467Oct 03 22:02:04 cloudcephosd1026 smart_failure[2966975]: Device: /dev/sdg [SAT], 2168 Offline uncorrectable sectors
1468Oct 03 23:02:04 cloudcephosd1026 smart_failure[2989473]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
1469Oct 04 00:02:04 cloudcephosd1026 smart_failure[3012050]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1470Oct 04 12:02:04 cloudcephosd1026 smart_failure[3284036]: Device: /dev/sdi [SAT], 2024 Offline uncorrectable sectors
1471Oct 04 17:02:04 cloudcephosd1026 smart_failure[3398347]: Device: /dev/sdh [SAT], 888 Offline uncorrectable sectors
1472Oct 04 18:32:04 cloudcephosd1026 smart_failure[3432518]: Device: /dev/sdd [SAT], 1368 Offline uncorrectable sectors
1473Oct 04 20:32:04 cloudcephosd1026 smart_failure[3478086]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1474Oct 04 22:02:04 cloudcephosd1026 smart_failure[3511580]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1475Oct 04 22:02:04 cloudcephosd1026 smart_failure[3511599]: Device: /dev/sdg [SAT], 2168 Offline uncorrectable sectors
1476Oct 04 23:02:04 cloudcephosd1026 smart_failure[3534144]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
1477Oct 05 00:02:04 cloudcephosd1026 smart_failure[3556831]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1478Oct 05 12:02:04 cloudcephosd1026 smart_failure[3827077]: Device: /dev/sdi [SAT], 2032 Offline uncorrectable sectors
1479Oct 05 17:02:04 cloudcephosd1026 smart_failure[3951813]: Device: /dev/sdh [SAT], 888 Offline uncorrectable sectors
1480Oct 05 18:32:04 cloudcephosd1026 smart_failure[3985861]: Device: /dev/sdd [SAT], 1376 Offline uncorrectable sectors
1481Oct 05 20:32:04 cloudcephosd1026 smart_failure[4030844]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1482Oct 05 22:02:04 cloudcephosd1026 smart_failure[4064936]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1483Oct 05 22:02:04 cloudcephosd1026 smart_failure[4064955]: Device: /dev/sdg [SAT], 2176 Offline uncorrectable sectors
1484Oct 05 23:02:04 cloudcephosd1026 smart_failure[4087552]: Device: /dev/sdc [SAT], 1280 Offline uncorrectable sectors
1485Oct 06 00:02:04 cloudcephosd1026 smart_failure[4110389]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1486Oct 06 12:02:04 cloudcephosd1026 smart_failure[186208]: Device: /dev/sdi [SAT], 2048 Offline uncorrectable sectors (changed +16)
1487Oct 06 17:02:04 cloudcephosd1026 smart_failure[299205]: Device: /dev/sdh [SAT], 888 Offline uncorrectable sectors
1488Oct 06 18:32:04 cloudcephosd1026 smart_failure[333070]: Device: /dev/sdd [SAT], 1432 Offline uncorrectable sectors
1489Oct 06 20:32:04 cloudcephosd1026 smart_failure[378347]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1490Oct 06 22:02:04 cloudcephosd1026 smart_failure[411988]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1491Oct 06 22:02:04 cloudcephosd1026 smart_failure[412007]: Device: /dev/sdg [SAT], 2176 Offline uncorrectable sectors
1492Oct 06 23:02:05 cloudcephosd1026 smart_failure[434165]: Device: /dev/sdc [SAT], 1280 Offline uncorrectable sectors
1493Oct 07 00:02:04 cloudcephosd1026 smart_failure[456793]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1494Oct 07 12:02:04 cloudcephosd1026 smart_failure[727768]: Device: /dev/sdi [SAT], 2056 Offline uncorrectable sectors
1495Oct 07 17:02:04 cloudcephosd1026 smart_failure[840874]: Device: /dev/sdh [SAT], 896 Offline uncorrectable sectors
1496Oct 07 18:32:04 cloudcephosd1026 smart_failure[875037]: Device: /dev/sdd [SAT], 1432 Offline uncorrectable sectors
1497Oct 07 20:32:04 cloudcephosd1026 smart_failure[920104]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1498Oct 07 22:02:04 cloudcephosd1026 smart_failure[953919]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1499Oct 07 22:02:04 cloudcephosd1026 smart_failure[953938]: Device: /dev/sdg [SAT], 2184 Offline uncorrectable sectors
1500Oct 07 23:02:04 cloudcephosd1026 smart_failure[976477]: Device: /dev/sdc [SAT], 1280 Offline uncorrectable sectors
1501Oct 08 00:02:04 cloudcephosd1026 smart_failure[999297]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1502Oct 08 12:02:04 cloudcephosd1026 smart_failure[1270632]: Device: /dev/sdi [SAT], 2056 Offline uncorrectable sectors
1503Oct 08 17:02:04 cloudcephosd1026 smart_failure[1383705]: Device: /dev/sdh [SAT], 896 Offline uncorrectable sectors
1504Oct 08 18:32:04 cloudcephosd1026 smart_failure[1417927]: Device: /dev/sdd [SAT], 1432 Offline uncorrectable sectors
1505Oct 08 20:32:04 cloudcephosd1026 smart_failure[1463203]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1506Oct 08 22:02:04 cloudcephosd1026 smart_failure[1496760]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1507Oct 08 22:02:04 cloudcephosd1026 smart_failure[1496779]: Device: /dev/sdg [SAT], 2192 Offline uncorrectable sectors
1508Oct 08 23:02:04 cloudcephosd1026 smart_failure[1519289]: Device: /dev/sdc [SAT], 1280 Offline uncorrectable sectors
1509Oct 09 00:02:05 cloudcephosd1026 smart_failure[1542201]: Device: /dev/sdj [SAT], 1088 Offline uncorrectable sectors
1510Oct 09 12:02:04 cloudcephosd1026 smart_failure[1813779]: Device: /dev/sdi [SAT], 2056 Offline uncorrectable sectors
1511Oct 09 17:02:05 cloudcephosd1026 smart_failure[1926453]: Device: /dev/sdh [SAT], 896 Offline uncorrectable sectors
1512Oct 09 18:32:04 cloudcephosd1026 smart_failure[1960675]: Device: /dev/sdd [SAT], 1432 Offline uncorrectable sectors
1513Oct 09 20:32:04 cloudcephosd1026 smart_failure[2006090]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1514Oct 09 22:02:04 cloudcephosd1026 smart_failure[2039929]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1515Oct 09 22:02:04 cloudcephosd1026 smart_failure[2039948]: Device: /dev/sdg [SAT], 2192 Offline uncorrectable sectors
1516Oct 09 23:02:04 cloudcephosd1026 smart_failure[2062454]: Device: /dev/sdc [SAT], 1280 Offline uncorrectable sectors
1517Oct 10 00:32:04 cloudcephosd1026 smart_failure[2096683]: Device: /dev/sdj [SAT], 1096 Offline uncorrectable sectors
1518Oct 10 12:02:04 cloudcephosd1026 smart_failure[2356724]: Device: /dev/sdi [SAT], 2064 Offline uncorrectable sectors
1519Oct 10 17:32:04 cloudcephosd1026 smart_failure[2481443]: Device: /dev/sdh [SAT], 896 Offline uncorrectable sectors
1520Oct 10 18:32:04 cloudcephosd1026 smart_failure[2504120]: Device: /dev/sdd [SAT], 1432 Offline uncorrectable sectors
1521Oct 10 20:32:04 cloudcephosd1026 smart_failure[2549717]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1522Oct 10 22:02:04 cloudcephosd1026 smart_failure[2583758]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1523Oct 10 22:02:04 cloudcephosd1026 smart_failure[2583777]: Device: /dev/sdg [SAT], 2192 Offline uncorrectable sectors
1524Oct 10 23:02:04 cloudcephosd1026 smart_failure[2606546]: Device: /dev/sdc [SAT], 1280 Offline uncorrectable sectors
1525Oct 11 00:32:04 cloudcephosd1026 smart_failure[2640622]: Device: /dev/sdj [SAT], 1104 Offline uncorrectable sectors
1526Oct 11 12:02:04 cloudcephosd1026 smart_failure[2900519]: Device: /dev/sdi [SAT], 2064 Offline uncorrectable sectors
1527Sep 08 12:02:04 cloudcephosd1026 smart_failure[1753688]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
1528Sep 08 17:02:04 cloudcephosd1026 smart_failure[1866905]: Device: /dev/sdh [SAT], 840 Offline uncorrectable sectors
1529Sep 08 18:32:04 cloudcephosd1026 smart_failure[1901153]: Device: /dev/sdd [SAT], 1328 Offline uncorrectable sectors
1530Sep 08 19:32:04 cloudcephosd1026 smart_failure[1923909]: Device: /dev/sdf [SAT], 656 Offline uncorrectable sectors
1531Sep 08 21:32:04 cloudcephosd1026 smart_failure[1969449]: Device: /dev/sde [SAT], 640 Offline uncorrectable sectors
1532Sep 08 21:32:04 cloudcephosd1026 smart_failure[1969468]: Device: /dev/sdg [SAT], 2032 Offline uncorrectable sectors
1533Sep 08 23:02:04 cloudcephosd1026 smart_failure[2003412]: Device: /dev/sdc [SAT], 1176 Offline uncorrectable sectors
1534Sep 09 00:02:04 cloudcephosd1026 smart_failure[2026238]: Device: /dev/sdj [SAT], 992 Offline uncorrectable sectors
1535Sep 09 12:02:04 cloudcephosd1026 smart_failure[2297805]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
1536Sep 09 17:02:04 cloudcephosd1026 smart_failure[2410245]: Device: /dev/sdh [SAT], 840 Offline uncorrectable sectors
1537Sep 09 18:32:04 cloudcephosd1026 smart_failure[2444342]: Device: /dev/sdd [SAT], 1328 Offline uncorrectable sectors
1538Sep 09 19:32:04 cloudcephosd1026 smart_failure[2467044]: Device: /dev/sdf [SAT], 656 Offline uncorrectable sectors
1539Sep 09 21:32:04 cloudcephosd1026 smart_failure[2511887]: Device: /dev/sde [SAT], 640 Offline uncorrectable sectors
1540Sep 09 21:32:04 cloudcephosd1026 smart_failure[2511906]: Device: /dev/sdg [SAT], 2032 Offline uncorrectable sectors
1541Sep 09 23:02:04 cloudcephosd1026 smart_failure[2545914]: Device: /dev/sdc [SAT], 1184 Offline uncorrectable sectors
1542Sep 10 00:02:04 cloudcephosd1026 smart_failure[2568756]: Device: /dev/sdj [SAT], 992 Offline uncorrectable sectors
1543Sep 10 12:02:04 cloudcephosd1026 smart_failure[2840891]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
1544Sep 10 17:02:04 cloudcephosd1026 smart_failure[2954183]: Device: /dev/sdh [SAT], 840 Offline uncorrectable sectors
1545Sep 10 18:32:04 cloudcephosd1026 smart_failure[2988265]: Device: /dev/sdd [SAT], 1328 Offline uncorrectable sectors
1546Sep 10 19:32:04 cloudcephosd1026 smart_failure[3010693]: Device: /dev/sdf [SAT], 656 Offline uncorrectable sectors
1547Sep 10 21:32:04 cloudcephosd1026 smart_failure[3055934]: Device: /dev/sde [SAT], 656 Offline uncorrectable sectors
1548Sep 10 21:32:04 cloudcephosd1026 smart_failure[3055953]: Device: /dev/sdg [SAT], 2040 Offline uncorrectable sectors
1549Sep 10 23:02:04 cloudcephosd1026 smart_failure[3089950]: Device: /dev/sdc [SAT], 1184 Offline uncorrectable sectors
1550Sep 11 00:02:04 cloudcephosd1026 smart_failure[3112777]: Device: /dev/sdj [SAT], 992 Offline uncorrectable sectors
1551Sep 11 12:02:04 cloudcephosd1026 smart_failure[3385414]: Device: /dev/sdi [SAT], 1800 Offline uncorrectable sectors
1552Sep 11 17:02:04 cloudcephosd1026 smart_failure[3498209]: Device: /dev/sdh [SAT], 848 Offline uncorrectable sectors
1553Sep 11 18:32:04 cloudcephosd1026 smart_failure[3532410]: Device: /dev/sdd [SAT], 1336 Offline uncorrectable sectors
1554Sep 11 19:32:04 cloudcephosd1026 smart_failure[3555301]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1555Sep 11 21:32:04 cloudcephosd1026 smart_failure[3601044]: Device: /dev/sde [SAT], 656 Offline uncorrectable sectors
1556Sep 11 21:32:04 cloudcephosd1026 smart_failure[3601063]: Device: /dev/sdg [SAT], 2048 Offline uncorrectable sectors
1557Sep 11 23:02:04 cloudcephosd1026 smart_failure[3635143]: Device: /dev/sdc [SAT], 1192 Offline uncorrectable sectors
1558Sep 12 00:02:04 cloudcephosd1026 smart_failure[3658086]: Device: /dev/sdj [SAT], 992 Offline uncorrectable sectors
1559Sep 12 12:02:04 cloudcephosd1026 smart_failure[3929875]: Device: /dev/sdi [SAT], 1832 Offline uncorrectable sectors
1560Sep 12 17:02:04 cloudcephosd1026 smart_failure[4042508]: Device: /dev/sdh [SAT], 848 Offline uncorrectable sectors
1561Sep 12 18:32:04 cloudcephosd1026 smart_failure[4076610]: Device: /dev/sdd [SAT], 1336 Offline uncorrectable sectors
1562Sep 12 19:32:04 cloudcephosd1026 smart_failure[4099200]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1563Sep 12 21:32:04 cloudcephosd1026 smart_failure[4144399]: Device: /dev/sde [SAT], 656 Offline uncorrectable sectors
1564Sep 12 21:32:04 cloudcephosd1026 smart_failure[4144418]: Device: /dev/sdg [SAT], 2048 Offline uncorrectable sectors
1565Sep 12 23:02:04 cloudcephosd1026 smart_failure[4178371]: Device: /dev/sdc [SAT], 1192 Offline uncorrectable sectors
1566Sep 13 00:02:04 cloudcephosd1026 smart_failure[7372]: Device: /dev/sdj [SAT], 992 Offline uncorrectable sectors
1567Sep 13 12:02:04 cloudcephosd1026 smart_failure[278784]: Device: /dev/sdi [SAT], 1848 Offline uncorrectable sectors
1568Sep 13 17:02:04 cloudcephosd1026 smart_failure[391768]: Device: /dev/sdh [SAT], 848 Offline uncorrectable sectors
1569Sep 13 18:32:04 cloudcephosd1026 smart_failure[425783]: Device: /dev/sdd [SAT], 1336 Offline uncorrectable sectors
1570Sep 13 19:32:04 cloudcephosd1026 smart_failure[448512]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1571Sep 13 21:32:04 cloudcephosd1026 smart_failure[494119]: Device: /dev/sde [SAT], 656 Offline uncorrectable sectors
1572Sep 13 21:32:04 cloudcephosd1026 smart_failure[494138]: Device: /dev/sdg [SAT], 2048 Offline uncorrectable sectors
1573Sep 13 23:02:04 cloudcephosd1026 smart_failure[527613]: Device: /dev/sdc [SAT], 1200 Offline uncorrectable sectors
1574Sep 14 00:02:04 cloudcephosd1026 smart_failure[550294]: Device: /dev/sdj [SAT], 1000 Offline uncorrectable sectors
1575Sep 14 12:02:04 cloudcephosd1026 smart_failure[821083]: Device: /dev/sdi [SAT], 1872 Offline uncorrectable sectors
1576Sep 14 17:02:04 cloudcephosd1026 smart_failure[933566]: Device: /dev/sdh [SAT], 848 Offline uncorrectable sectors
1577Sep 14 18:32:04 cloudcephosd1026 smart_failure[967652]: Device: /dev/sdd [SAT], 1336 Offline uncorrectable sectors
1578Sep 14 19:32:04 cloudcephosd1026 smart_failure[990614]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1579Sep 14 21:32:04 cloudcephosd1026 smart_failure[1036191]: Device: /dev/sde [SAT], 656 Offline uncorrectable sectors
1580Sep 14 21:32:04 cloudcephosd1026 smart_failure[1036210]: Device: /dev/sdg [SAT], 2048 Offline uncorrectable sectors
1581Sep 14 23:02:04 cloudcephosd1026 smart_failure[1069694]: Device: /dev/sdc [SAT], 1200 Offline uncorrectable sectors
1582Sep 15 00:02:04 cloudcephosd1026 smart_failure[1092061]: Device: /dev/sdj [SAT], 1000 Offline uncorrectable sectors
1583Sep 15 12:02:04 cloudcephosd1026 smart_failure[1364230]: Device: /dev/sdi [SAT], 1872 Offline uncorrectable sectors
1584Sep 15 17:02:04 cloudcephosd1026 smart_failure[1477248]: Device: /dev/sdh [SAT], 848 Offline uncorrectable sectors
1585Sep 15 18:32:04 cloudcephosd1026 smart_failure[1511667]: Device: /dev/sdd [SAT], 1336 Offline uncorrectable sectors
1586Sep 15 19:32:04 cloudcephosd1026 smart_failure[1534373]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1587Sep 15 21:32:04 cloudcephosd1026 smart_failure[1579995]: Device: /dev/sde [SAT], 656 Offline uncorrectable sectors
1588Sep 15 21:32:04 cloudcephosd1026 smart_failure[1580014]: Device: /dev/sdg [SAT], 2056 Offline uncorrectable sectors
1589Sep 15 23:02:04 cloudcephosd1026 smart_failure[1614127]: Device: /dev/sdc [SAT], 1200 Offline uncorrectable sectors
1590Sep 16 00:02:04 cloudcephosd1026 smart_failure[1637048]: Device: /dev/sdj [SAT], 1008 Offline uncorrectable sectors
1591Sep 16 12:02:04 cloudcephosd1026 smart_failure[1910555]: Device: /dev/sdi [SAT], 1872 Offline uncorrectable sectors
1592Sep 16 17:02:04 cloudcephosd1026 smart_failure[2023880]: Device: /dev/sdh [SAT], 848 Offline uncorrectable sectors
1593Sep 16 18:32:04 cloudcephosd1026 smart_failure[2058187]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1594Sep 16 19:32:04 cloudcephosd1026 smart_failure[2080754]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1595Sep 16 21:32:04 cloudcephosd1026 smart_failure[2126347]: Device: /dev/sde [SAT], 664 Offline uncorrectable sectors
1596Sep 16 21:32:04 cloudcephosd1026 smart_failure[2126366]: Device: /dev/sdg [SAT], 2064 Offline uncorrectable sectors
1597Sep 16 23:02:04 cloudcephosd1026 smart_failure[2160419]: Device: /dev/sdc [SAT], 1200 Offline uncorrectable sectors
1598Sep 17 00:02:04 cloudcephosd1026 smart_failure[2183283]: Device: /dev/sdj [SAT], 1008 Offline uncorrectable sectors
1599Sep 17 12:02:04 cloudcephosd1026 smart_failure[2455735]: Device: /dev/sdi [SAT], 1912 Offline uncorrectable sectors
1600Sep 17 17:02:04 cloudcephosd1026 smart_failure[2569082]: Device: /dev/sdh [SAT], 856 Offline uncorrectable sectors
1601Sep 17 18:32:04 cloudcephosd1026 smart_failure[2603094]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1602Sep 17 19:32:04 cloudcephosd1026 smart_failure[2625890]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1603Sep 17 21:32:04 cloudcephosd1026 smart_failure[2671404]: Device: /dev/sde [SAT], 664 Offline uncorrectable sectors
1604Sep 17 21:32:04 cloudcephosd1026 smart_failure[2671423]: Device: /dev/sdg [SAT], 2064 Offline uncorrectable sectors
1605Sep 17 23:02:04 cloudcephosd1026 smart_failure[2705488]: Device: /dev/sdc [SAT], 1216 Offline uncorrectable sectors
1606Sep 18 00:02:04 cloudcephosd1026 smart_failure[2728375]: Device: /dev/sdj [SAT], 1008 Offline uncorrectable sectors
1607Sep 18 12:02:04 cloudcephosd1026 smart_failure[3000582]: Device: /dev/sdi [SAT], 1912 Offline uncorrectable sectors
1608Sep 18 17:02:04 cloudcephosd1026 smart_failure[3113015]: Device: /dev/sdh [SAT], 856 Offline uncorrectable sectors
1609Sep 18 18:32:04 cloudcephosd1026 smart_failure[3146934]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1610Sep 18 19:32:04 cloudcephosd1026 smart_failure[3169618]: Device: /dev/sdf [SAT], 664 Offline uncorrectable sectors
1611Sep 18 21:32:04 cloudcephosd1026 smart_failure[3214767]: Device: /dev/sde [SAT], 672 Offline uncorrectable sectors
1612Sep 18 21:32:04 cloudcephosd1026 smart_failure[3214786]: Device: /dev/sdg [SAT], 2064 Offline uncorrectable sectors
1613Sep 18 23:02:04 cloudcephosd1026 smart_failure[3248551]: Device: /dev/sdc [SAT], 1216 Offline uncorrectable sectors
1614Sep 19 00:02:04 cloudcephosd1026 smart_failure[3271329]: Device: /dev/sdj [SAT], 1008 Offline uncorrectable sectors
1615Sep 19 12:02:04 cloudcephosd1026 smart_failure[3543275]: Device: /dev/sdi [SAT], 1912 Offline uncorrectable sectors
1616Sep 19 17:02:04 cloudcephosd1026 smart_failure[3656719]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1617Sep 19 18:32:04 cloudcephosd1026 smart_failure[3691186]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1618Sep 19 19:32:04 cloudcephosd1026 smart_failure[3714156]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1619Sep 19 21:32:04 cloudcephosd1026 smart_failure[3760011]: Device: /dev/sde [SAT], 672 Offline uncorrectable sectors
1620Sep 19 21:32:04 cloudcephosd1026 smart_failure[3760030]: Device: /dev/sdg [SAT], 2064 Offline uncorrectable sectors
1621Sep 19 23:02:04 cloudcephosd1026 smart_failure[3794487]: Device: /dev/sdc [SAT], 1224 Offline uncorrectable sectors (changed +8)
1622Sep 20 00:02:04 cloudcephosd1026 smart_failure[3816958]: Device: /dev/sdj [SAT], 1008 Offline uncorrectable sectors
1623Sep 20 12:02:04 cloudcephosd1026 smart_failure[4088273]: Device: /dev/sdi [SAT], 1912 Offline uncorrectable sectors
1624Sep 20 17:02:04 cloudcephosd1026 smart_failure[7259]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1625Sep 20 18:32:04 cloudcephosd1026 smart_failure[41262]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1626Sep 20 19:32:04 cloudcephosd1026 smart_failure[63748]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1627Sep 20 21:32:04 cloudcephosd1026 smart_failure[109084]: Device: /dev/sde [SAT], 672 Offline uncorrectable sectors
1628Sep 20 21:32:04 cloudcephosd1026 smart_failure[109103]: Device: /dev/sdg [SAT], 2072 Offline uncorrectable sectors
1629Sep 20 23:02:04 cloudcephosd1026 smart_failure[143051]: Device: /dev/sdc [SAT], 1224 Offline uncorrectable sectors
1630Sep 21 00:02:04 cloudcephosd1026 smart_failure[165710]: Device: /dev/sdj [SAT], 1008 Offline uncorrectable sectors
1631Sep 21 12:02:04 cloudcephosd1026 smart_failure[436621]: Device: /dev/sdi [SAT], 1920 Offline uncorrectable sectors
1632Sep 21 17:02:04 cloudcephosd1026 smart_failure[549605]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1633Sep 21 18:32:04 cloudcephosd1026 smart_failure[583461]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1634Sep 21 19:32:04 cloudcephosd1026 smart_failure[606211]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1635Sep 21 21:32:04 cloudcephosd1026 smart_failure[651302]: Device: /dev/sde [SAT], 672 Offline uncorrectable sectors
1636Sep 21 21:32:04 cloudcephosd1026 smart_failure[651321]: Device: /dev/sdg [SAT], 2072 Offline uncorrectable sectors
1637Sep 21 23:02:04 cloudcephosd1026 smart_failure[685198]: Device: /dev/sdc [SAT], 1232 Offline uncorrectable sectors
1638Sep 22 00:02:04 cloudcephosd1026 smart_failure[707712]: Device: /dev/sdj [SAT], 1016 Offline uncorrectable sectors
1639Sep 22 12:02:04 cloudcephosd1026 smart_failure[978502]: Device: /dev/sdi [SAT], 1920 Offline uncorrectable sectors
1640Sep 22 17:02:04 cloudcephosd1026 smart_failure[1091192]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1641Sep 22 18:32:04 cloudcephosd1026 smart_failure[1125204]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1642Sep 22 19:32:05 cloudcephosd1026 smart_failure[1147445]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1643Sep 22 21:32:04 cloudcephosd1026 smart_failure[1192430]: Device: /dev/sde [SAT], 672 Offline uncorrectable sectors
1644Sep 22 21:32:04 cloudcephosd1026 smart_failure[1192449]: Device: /dev/sdg [SAT], 2072 Offline uncorrectable sectors
1645Sep 22 23:02:04 cloudcephosd1026 smart_failure[1225879]: Device: /dev/sdc [SAT], 1232 Offline uncorrectable sectors
1646Sep 23 00:02:04 cloudcephosd1026 smart_failure[1248231]: Device: /dev/sdj [SAT], 1016 Offline uncorrectable sectors
1647Sep 23 12:02:04 cloudcephosd1026 smart_failure[1518780]: Device: /dev/sdi [SAT], 1920 Offline uncorrectable sectors
1648Sep 23 17:02:04 cloudcephosd1026 smart_failure[1631481]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1649Sep 23 18:32:04 cloudcephosd1026 smart_failure[1665701]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1650Sep 23 20:02:04 cloudcephosd1026 smart_failure[1699333]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1651Sep 23 21:32:04 cloudcephosd1026 smart_failure[1733197]: Device: /dev/sde [SAT], 680 Offline uncorrectable sectors
1652Sep 23 21:32:04 cloudcephosd1026 smart_failure[1733216]: Device: /dev/sdg [SAT], 2112 Offline uncorrectable sectors
1653Sep 23 23:02:04 cloudcephosd1026 smart_failure[1766549]: Device: /dev/sdc [SAT], 1232 Offline uncorrectable sectors
1654Sep 24 00:02:04 cloudcephosd1026 smart_failure[1789062]: Device: /dev/sdj [SAT], 1048 Offline uncorrectable sectors
1655Sep 24 12:02:04 cloudcephosd1026 smart_failure[2059678]: Device: /dev/sdi [SAT], 1928 Offline uncorrectable sectors
1656Sep 24 17:02:04 cloudcephosd1026 smart_failure[2172095]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1657Sep 24 18:32:04 cloudcephosd1026 smart_failure[2206135]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1658Sep 24 20:02:05 cloudcephosd1026 smart_failure[2239502]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1659Sep 24 21:32:04 cloudcephosd1026 smart_failure[2273403]: Device: /dev/sde [SAT], 680 Offline uncorrectable sectors
1660Sep 24 21:32:04 cloudcephosd1026 smart_failure[2273422]: Device: /dev/sdg [SAT], 2120 Offline uncorrectable sectors
1661Sep 24 23:02:04 cloudcephosd1026 smart_failure[2307342]: Device: /dev/sdc [SAT], 1232 Offline uncorrectable sectors
1662Sep 25 00:02:04 cloudcephosd1026 smart_failure[2330340]: Device: /dev/sdj [SAT], 1064 Offline uncorrectable sectors
1663Sep 25 12:02:05 cloudcephosd1026 smart_failure[2600423]: Device: /dev/sdi [SAT], 1936 Offline uncorrectable sectors
1664Sep 25 17:02:04 cloudcephosd1026 smart_failure[2713350]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1665Sep 25 18:32:04 cloudcephosd1026 smart_failure[2747654]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1666Sep 25 20:32:04 cloudcephosd1026 smart_failure[2793040]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1667Sep 25 21:32:04 cloudcephosd1026 smart_failure[2815867]: Device: /dev/sde [SAT], 680 Offline uncorrectable sectors
1668Sep 25 21:32:04 cloudcephosd1026 smart_failure[2815886]: Device: /dev/sdg [SAT], 2128 Offline uncorrectable sectors
1669Sep 25 23:02:04 cloudcephosd1026 smart_failure[2849737]: Device: /dev/sdc [SAT], 1240 Offline uncorrectable sectors
1670Sep 26 00:02:04 cloudcephosd1026 smart_failure[2872600]: Device: /dev/sdj [SAT], 1072 Offline uncorrectable sectors
1671Sep 26 12:02:04 cloudcephosd1026 smart_failure[3145377]: Device: /dev/sdi [SAT], 1944 Offline uncorrectable sectors
1672Sep 26 17:02:04 cloudcephosd1026 smart_failure[3258350]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1673Sep 26 18:32:04 cloudcephosd1026 smart_failure[3292473]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1674Sep 26 20:32:05 cloudcephosd1026 smart_failure[3337817]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1675Sep 26 21:32:04 cloudcephosd1026 smart_failure[3360821]: Device: /dev/sde [SAT], 688 Offline uncorrectable sectors
1676Sep 26 21:32:04 cloudcephosd1026 smart_failure[3360840]: Device: /dev/sdg [SAT], 2144 Offline uncorrectable sectors
1677Sep 26 23:02:04 cloudcephosd1026 smart_failure[3394849]: Device: /dev/sdc [SAT], 1248 Offline uncorrectable sectors
1678Sep 27 00:02:04 cloudcephosd1026 smart_failure[3417580]: Device: /dev/sdj [SAT], 1080 Offline uncorrectable sectors
1679Sep 27 12:02:04 cloudcephosd1026 smart_failure[3687790]: Device: /dev/sdi [SAT], 1960 Offline uncorrectable sectors (changed +16)
1680Sep 27 17:02:04 cloudcephosd1026 smart_failure[3800417]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1681Sep 27 18:32:04 cloudcephosd1026 smart_failure[3834489]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1682Sep 27 20:32:04 cloudcephosd1026 smart_failure[3879287]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1683Sep 27 21:32:04 cloudcephosd1026 smart_failure[3901693]: Device: /dev/sde [SAT], 688 Offline uncorrectable sectors
1684Sep 27 21:32:04 cloudcephosd1026 smart_failure[3901712]: Device: /dev/sdg [SAT], 2152 Offline uncorrectable sectors
1685Sep 27 23:02:04 cloudcephosd1026 smart_failure[3935629]: Device: /dev/sdc [SAT], 1256 Offline uncorrectable sectors
1686Sep 28 00:02:04 cloudcephosd1026 smart_failure[3958290]: Device: /dev/sdj [SAT], 1080 Offline uncorrectable sectors
1687Sep 28 12:02:04 cloudcephosd1026 smart_failure[35838]: Device: /dev/sdi [SAT], 1968 Offline uncorrectable sectors
1688Sep 28 17:02:04 cloudcephosd1026 smart_failure[147723]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1689Sep 28 18:32:04 cloudcephosd1026 smart_failure[181863]: Device: /dev/sdd [SAT], 1344 Offline uncorrectable sectors
1690Sep 28 20:32:04 cloudcephosd1026 smart_failure[226866]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1691Sep 28 21:32:05 cloudcephosd1026 smart_failure[249269]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1692Sep 28 21:32:05 cloudcephosd1026 smart_failure[249288]: Device: /dev/sdg [SAT], 2160 Offline uncorrectable sectors
1693Sep 28 23:02:04 cloudcephosd1026 smart_failure[283232]: Device: /dev/sdc [SAT], 1264 Offline uncorrectable sectors
1694Sep 29 00:02:04 cloudcephosd1026 smart_failure[305975]: Device: /dev/sdj [SAT], 1080 Offline uncorrectable sectors
1695Sep 29 12:02:04 cloudcephosd1026 smart_failure[577359]: Device: /dev/sdi [SAT], 1976 Offline uncorrectable sectors
1696Sep 29 17:02:04 cloudcephosd1026 smart_failure[690453]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1697Sep 29 18:32:04 cloudcephosd1026 smart_failure[724601]: Device: /dev/sdd [SAT], 1368 Offline uncorrectable sectors
1698Sep 29 20:32:04 cloudcephosd1026 smart_failure[769913]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1699Sep 29 22:02:04 cloudcephosd1026 smart_failure[803999]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1700Sep 29 22:02:04 cloudcephosd1026 smart_failure[804018]: Device: /dev/sdg [SAT], 2160 Offline uncorrectable sectors
1701Sep 29 23:02:04 cloudcephosd1026 smart_failure[826593]: Device: /dev/sdc [SAT], 1264 Offline uncorrectable sectors
1702Sep 30 00:02:04 cloudcephosd1026 smart_failure[849259]: Device: /dev/sdj [SAT], 1080 Offline uncorrectable sectors
1703Sep 30 12:02:04 cloudcephosd1026 smart_failure[1121896]: Device: /dev/sdi [SAT], 1984 Offline uncorrectable sectors
1704Sep 30 17:02:04 cloudcephosd1026 smart_failure[1235424]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
1705Sep 30 18:32:04 cloudcephosd1026 smart_failure[1269472]: Device: /dev/sdd [SAT], 1368 Offline uncorrectable sectors
1706Sep 30 20:32:04 cloudcephosd1026 smart_failure[1315083]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1707Sep 30 22:02:04 cloudcephosd1026 smart_failure[1349052]: Device: /dev/sde [SAT], 704 Offline uncorrectable sectors
1708Sep 30 22:02:04 cloudcephosd1026 smart_failure[1349071]: Device: /dev/sdg [SAT], 2160 Offline uncorrectable sectors
1709Sep 30 23:02:04 cloudcephosd1026 smart_failure[1371828]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
1710===== NODE GROUP =====
1711(1) cloudcephosd1034.eqiad.wmnet
1712----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
1713Oct 01 00:57:02 cloudcephosd1034 smart_failure[1437351]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1714Oct 01 01:27:03 cloudcephosd1034 smart_failure[1448698]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1715Oct 01 08:57:02 cloudcephosd1034 smart_failure[1619401]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1716Oct 01 10:57:02 cloudcephosd1034 smart_failure[1664812]: Device: /dev/sde [SAT], 1024 Offline uncorrectable sectors
1717Oct 01 11:57:02 cloudcephosd1034 smart_failure[1687596]: Device: /dev/sdf [SAT], 704 Offline uncorrectable sectors
1718Oct 01 11:57:02 cloudcephosd1034 smart_failure[1687615]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1719Oct 01 16:57:02 cloudcephosd1034 smart_failure[1800886]: Device: /dev/sdc [SAT], 2640 Offline uncorrectable sectors
1720Oct 02 00:57:02 cloudcephosd1034 smart_failure[1981745]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1721Oct 02 01:57:02 cloudcephosd1034 smart_failure[2004347]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1722Oct 02 08:57:02 cloudcephosd1034 smart_failure[2163338]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1723Oct 02 10:57:02 cloudcephosd1034 smart_failure[2208102]: Device: /dev/sde [SAT], 1024 Offline uncorrectable sectors
1724Oct 02 11:57:02 cloudcephosd1034 smart_failure[2230942]: Device: /dev/sdf [SAT], 704 Offline uncorrectable sectors
1725Oct 02 11:57:02 cloudcephosd1034 smart_failure[2230961]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1726Oct 02 16:57:02 cloudcephosd1034 smart_failure[2343752]: Device: /dev/sdc [SAT], 2656 Offline uncorrectable sectors
1727Oct 03 00:57:02 cloudcephosd1034 smart_failure[2517317]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1728Oct 03 01:57:02 cloudcephosd1034 smart_failure[2538830]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1729Oct 03 08:57:02 cloudcephosd1034 smart_failure[2697458]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1730Oct 03 10:57:02 cloudcephosd1034 smart_failure[2742954]: Device: /dev/sde [SAT], 1024 Offline uncorrectable sectors
1731Oct 03 11:57:02 cloudcephosd1034 smart_failure[2765321]: Device: /dev/sdf [SAT], 712 Offline uncorrectable sectors
1732Oct 03 11:57:02 cloudcephosd1034 smart_failure[2765340]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1733Oct 03 16:57:02 cloudcephosd1034 smart_failure[2878246]: Device: /dev/sdc [SAT], 2664 Offline uncorrectable sectors
1734Oct 04 00:57:02 cloudcephosd1034 smart_failure[3059164]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1735Oct 04 01:57:02 cloudcephosd1034 smart_failure[3081956]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1736Oct 04 08:57:02 cloudcephosd1034 smart_failure[3241181]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1737Oct 04 10:57:02 cloudcephosd1034 smart_failure[3286259]: Device: /dev/sde [SAT], 1024 Offline uncorrectable sectors
1738Oct 04 11:57:02 cloudcephosd1034 smart_failure[3309058]: Device: /dev/sdf [SAT], 712 Offline uncorrectable sectors
1739Oct 04 11:57:02 cloudcephosd1034 smart_failure[3309106]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1740Oct 04 16:57:02 cloudcephosd1034 smart_failure[3423609]: Device: /dev/sdc [SAT], 2664 Offline uncorrectable sectors
1741Oct 05 00:57:02 cloudcephosd1034 smart_failure[3604013]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1742Oct 05 01:57:02 cloudcephosd1034 smart_failure[3626619]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1743Oct 05 08:57:02 cloudcephosd1034 smart_failure[3784215]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1744Oct 05 10:57:02 cloudcephosd1034 smart_failure[3829658]: Device: /dev/sde [SAT], 1024 Offline uncorrectable sectors
1745Oct 05 11:57:02 cloudcephosd1034 smart_failure[3852216]: Device: /dev/sdf [SAT], 712 Offline uncorrectable sectors
1746Oct 05 11:57:02 cloudcephosd1034 smart_failure[3852235]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1747Oct 05 16:57:02 cloudcephosd1034 smart_failure[3976565]: Device: /dev/sdc [SAT], 2672 Offline uncorrectable sectors
1748Oct 06 00:57:02 cloudcephosd1034 smart_failure[4158342]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1749Oct 06 01:57:02 cloudcephosd1034 smart_failure[4180737]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
1750Oct 06 08:57:02 cloudcephosd1034 smart_failure[144364]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1751Oct 06 10:57:02 cloudcephosd1034 smart_failure[189190]: Device: /dev/sde [SAT], 1024 Offline uncorrectable sectors
1752Oct 06 11:57:02 cloudcephosd1034 smart_failure[211762]: Device: /dev/sdf [SAT], 712 Offline uncorrectable sectors
1753Oct 06 11:57:02 cloudcephosd1034 smart_failure[211781]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1754Oct 06 16:57:02 cloudcephosd1034 smart_failure[324591]: Device: /dev/sdc [SAT], 2688 Offline uncorrectable sectors
1755Oct 07 00:57:02 cloudcephosd1034 smart_failure[505819]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1756Oct 07 01:57:02 cloudcephosd1034 smart_failure[528340]: Device: /dev/sdj [SAT], 488 Offline uncorrectable sectors
1757Oct 07 08:57:02 cloudcephosd1034 smart_failure[686447]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1758Oct 07 10:57:02 cloudcephosd1034 smart_failure[731787]: Device: /dev/sde [SAT], 1024 Offline uncorrectable sectors
1759Oct 07 11:57:02 cloudcephosd1034 smart_failure[754465]: Device: /dev/sdf [SAT], 712 Offline uncorrectable sectors
1760Oct 07 11:57:02 cloudcephosd1034 smart_failure[754484]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1761Oct 07 16:57:02 cloudcephosd1034 smart_failure[867585]: Device: /dev/sdc [SAT], 2696 Offline uncorrectable sectors
1762Oct 08 00:57:02 cloudcephosd1034 smart_failure[1048518]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1763Oct 08 01:57:02 cloudcephosd1034 smart_failure[1071474]: Device: /dev/sdj [SAT], 488 Offline uncorrectable sectors
1764Oct 08 08:57:02 cloudcephosd1034 smart_failure[1229981]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1765Oct 08 10:57:02 cloudcephosd1034 smart_failure[1275265]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
1766Oct 08 11:57:02 cloudcephosd1034 smart_failure[1297599]: Device: /dev/sdf [SAT], 720 Offline uncorrectable sectors
1767Oct 08 11:57:02 cloudcephosd1034 smart_failure[1297618]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1768Oct 08 16:57:02 cloudcephosd1034 smart_failure[1410693]: Device: /dev/sdc [SAT], 2728 Offline uncorrectable sectors
1769Oct 09 00:57:02 cloudcephosd1034 smart_failure[1591946]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1770Oct 09 01:57:02 cloudcephosd1034 smart_failure[1614769]: Device: /dev/sdj [SAT], 488 Offline uncorrectable sectors
1771Oct 09 08:57:02 cloudcephosd1034 smart_failure[1772588]: Device: /dev/sdd [SAT], 320 Offline uncorrectable sectors
1772Oct 09 10:57:02 cloudcephosd1034 smart_failure[1818191]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
1773Oct 09 11:57:02 cloudcephosd1034 smart_failure[1840908]: Device: /dev/sdf [SAT], 720 Offline uncorrectable sectors
1774Oct 09 11:57:02 cloudcephosd1034 smart_failure[1840927]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1775Oct 09 16:57:02 cloudcephosd1034 smart_failure[1953794]: Device: /dev/sdc [SAT], 2736 Offline uncorrectable sectors
1776Oct 10 00:57:02 cloudcephosd1034 smart_failure[2135262]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1777Oct 10 01:57:02 cloudcephosd1034 smart_failure[2157775]: Device: /dev/sdj [SAT], 488 Offline uncorrectable sectors
1778Oct 10 08:57:02 cloudcephosd1034 smart_failure[2316379]: Device: /dev/sdd [SAT], 320 Offline uncorrectable sectors
1779Oct 10 10:57:02 cloudcephosd1034 smart_failure[2361817]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
1780Oct 10 11:57:03 cloudcephosd1034 smart_failure[2384378]: Device: /dev/sdf [SAT], 720 Offline uncorrectable sectors
1781Oct 10 11:57:03 cloudcephosd1034 smart_failure[2384397]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1782Oct 10 16:57:02 cloudcephosd1034 smart_failure[2498855]: Device: /dev/sdc [SAT], 2744 Offline uncorrectable sectors
1783Oct 11 00:57:02 cloudcephosd1034 smart_failure[2680188]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1784Oct 11 01:57:02 cloudcephosd1034 smart_failure[2702640]: Device: /dev/sdj [SAT], 488 Offline uncorrectable sectors
1785Oct 11 08:57:02 cloudcephosd1034 smart_failure[2860686]: Device: /dev/sdd [SAT], 320 Offline uncorrectable sectors
1786Oct 11 10:57:02 cloudcephosd1034 smart_failure[2906055]: Device: /dev/sde [SAT], 1040 Offline uncorrectable sectors
1787Oct 11 12:27:02 cloudcephosd1034 smart_failure[2940153]: Device: /dev/sdf [SAT], 720 Offline uncorrectable sectors
1788Oct 11 12:27:02 cloudcephosd1034 smart_failure[2940172]: Device: /dev/sdi [SAT], 520 Offline uncorrectable sectors
1789Sep 14 10:57:02 cloudcephosd1034 smart_failure[814509]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
1790Sep 14 11:27:02 cloudcephosd1034 smart_failure[825923]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1791Sep 14 11:27:02 cloudcephosd1034 smart_failure[825942]: Device: /dev/sdi [SAT], 488 Offline uncorrectable sectors
1792Sep 14 16:27:03 cloudcephosd1034 smart_failure[938747]: Device: /dev/sdc [SAT], 2424 Offline uncorrectable sectors
1793Sep 15 00:57:02 cloudcephosd1034 smart_failure[1130657]: Device: /dev/sdg [SAT], 1096 Offline uncorrectable sectors
1794Sep 15 00:57:02 cloudcephosd1034 smart_failure[1130676]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1795Sep 15 08:57:02 cloudcephosd1034 smart_failure[1312030]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1796Sep 15 10:57:02 cloudcephosd1034 smart_failure[1357654]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
1797Sep 15 11:27:02 cloudcephosd1034 smart_failure[1369182]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1798Sep 15 11:27:02 cloudcephosd1034 smart_failure[1369201]: Device: /dev/sdi [SAT], 488 Offline uncorrectable sectors
1799Sep 15 16:27:02 cloudcephosd1034 smart_failure[1482858]: Device: /dev/sdc [SAT], 2424 Offline uncorrectable sectors
1800Sep 16 00:57:02 cloudcephosd1034 smart_failure[1677199]: Device: /dev/sdg [SAT], 1096 Offline uncorrectable sectors
1801Sep 16 00:57:02 cloudcephosd1034 smart_failure[1677218]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1802Sep 16 08:57:02 cloudcephosd1034 smart_failure[1859117]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1803Sep 16 10:57:02 cloudcephosd1034 smart_failure[1904785]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
1804Sep 16 11:27:02 cloudcephosd1034 smart_failure[1916269]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1805Sep 16 11:27:02 cloudcephosd1034 smart_failure[1916288]: Device: /dev/sdi [SAT], 488 Offline uncorrectable sectors
1806Sep 16 16:27:02 cloudcephosd1034 smart_failure[2029508]: Device: /dev/sdc [SAT], 2432 Offline uncorrectable sectors
1807Sep 17 00:57:02 cloudcephosd1034 smart_failure[2222877]: Device: /dev/sdg [SAT], 1096 Offline uncorrectable sectors
1808Sep 17 00:57:02 cloudcephosd1034 smart_failure[2222897]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1809Sep 17 08:57:02 cloudcephosd1034 smart_failure[2404987]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1810Sep 17 10:57:02 cloudcephosd1034 smart_failure[2450350]: Device: /dev/sde [SAT], 984 Offline uncorrectable sectors
1811Sep 17 11:27:02 cloudcephosd1034 smart_failure[2461487]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1812Sep 17 11:27:02 cloudcephosd1034 smart_failure[2461539]: Device: /dev/sdi [SAT], 488 Offline uncorrectable sectors
1813Sep 17 16:27:02 cloudcephosd1034 smart_failure[2575071]: Device: /dev/sdc [SAT], 2448 Offline uncorrectable sectors
1814Sep 18 00:57:02 cloudcephosd1034 smart_failure[2768525]: Device: /dev/sdg [SAT], 1096 Offline uncorrectable sectors
1815Sep 18 00:57:02 cloudcephosd1034 smart_failure[2768544]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1816Sep 18 08:57:02 cloudcephosd1034 smart_failure[2949739]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1817Sep 18 10:57:02 cloudcephosd1034 smart_failure[2994415]: Device: /dev/sde [SAT], 984 Offline uncorrectable sectors
1818Sep 18 11:27:02 cloudcephosd1034 smart_failure[3005889]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1819Sep 18 11:27:02 cloudcephosd1034 smart_failure[3005908]: Device: /dev/sdi [SAT], 488 Offline uncorrectable sectors
1820Sep 18 16:27:02 cloudcephosd1034 smart_failure[3118473]: Device: /dev/sdc [SAT], 2456 Offline uncorrectable sectors
1821Sep 19 00:57:02 cloudcephosd1034 smart_failure[3310813]: Device: /dev/sdg [SAT], 1096 Offline uncorrectable sectors
1822Sep 19 00:57:02 cloudcephosd1034 smart_failure[3310863]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1823Sep 19 08:57:02 cloudcephosd1034 smart_failure[3492430]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1824Sep 19 10:57:02 cloudcephosd1034 smart_failure[3537574]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1825Sep 19 11:27:03 cloudcephosd1034 smart_failure[3549035]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1826Sep 19 11:27:03 cloudcephosd1034 smart_failure[3549054]: Device: /dev/sdi [SAT], 488 Offline uncorrectable sectors
1827Sep 19 16:27:02 cloudcephosd1034 smart_failure[3662581]: Device: /dev/sdc [SAT], 2464 Offline uncorrectable sectors
1828Sep 20 00:57:02 cloudcephosd1034 smart_failure[3857022]: Device: /dev/sdg [SAT], 1104 Offline uncorrectable sectors
1829Sep 20 00:57:02 cloudcephosd1034 smart_failure[3857041]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1830Sep 20 08:57:02 cloudcephosd1034 smart_failure[4037992]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1831Sep 20 10:57:02 cloudcephosd1034 smart_failure[4082848]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1832Sep 20 11:57:02 cloudcephosd1034 smart_failure[4105585]: Device: /dev/sdf [SAT], 672 Offline uncorrectable sectors
1833Sep 20 11:57:02 cloudcephosd1034 smart_failure[4105604]: Device: /dev/sdi [SAT], 488 Offline uncorrectable sectors
1834Sep 20 16:27:02 cloudcephosd1034 smart_failure[13563]: Device: /dev/sdc [SAT], 2464 Offline uncorrectable sectors
1835Sep 21 00:57:02 cloudcephosd1034 smart_failure[205728]: Device: /dev/sdg [SAT], 1104 Offline uncorrectable sectors
1836Sep 21 00:57:02 cloudcephosd1034 smart_failure[205777]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1837Sep 21 08:57:02 cloudcephosd1034 smart_failure[386198]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1838Sep 21 10:57:02 cloudcephosd1034 smart_failure[431671]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1839Sep 21 11:57:02 cloudcephosd1034 smart_failure[454376]: Device: /dev/sdf [SAT], 680 Offline uncorrectable sectors
1840Sep 21 11:57:02 cloudcephosd1034 smart_failure[454395]: Device: /dev/sdi [SAT], 496 Offline uncorrectable sectors
1841Sep 21 16:27:02 cloudcephosd1034 smart_failure[556608]: Device: /dev/sdc [SAT], 2480 Offline uncorrectable sectors (changed +8)
1842Sep 22 00:57:02 cloudcephosd1034 smart_failure[748416]: Device: /dev/sdg [SAT], 1104 Offline uncorrectable sectors
1843Sep 22 00:57:02 cloudcephosd1034 smart_failure[748437]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1844Sep 22 08:57:02 cloudcephosd1034 smart_failure[929441]: Device: /dev/sdd [SAT], 304 Offline uncorrectable sectors
1845Sep 22 10:57:02 cloudcephosd1034 smart_failure[974354]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1846Sep 22 11:57:02 cloudcephosd1034 smart_failure[996919]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1847Sep 22 11:57:02 cloudcephosd1034 smart_failure[996968]: Device: /dev/sdi [SAT], 496 Offline uncorrectable sectors
1848Sep 22 16:27:02 cloudcephosd1034 smart_failure[1098581]: Device: /dev/sdc [SAT], 2496 Offline uncorrectable sectors
1849Sep 23 00:57:02 cloudcephosd1034 smart_failure[1288895]: Device: /dev/sdg [SAT], 1104 Offline uncorrectable sectors
1850Sep 23 00:57:02 cloudcephosd1034 smart_failure[1288944]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1851Sep 23 08:57:02 cloudcephosd1034 smart_failure[1469200]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1852Sep 23 10:57:02 cloudcephosd1034 smart_failure[1514551]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1853Sep 23 11:57:02 cloudcephosd1034 smart_failure[1537175]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1854Sep 23 11:57:02 cloudcephosd1034 smart_failure[1537194]: Device: /dev/sdi [SAT], 504 Offline uncorrectable sectors
1855Sep 23 16:27:02 cloudcephosd1034 smart_failure[1638980]: Device: /dev/sdc [SAT], 2520 Offline uncorrectable sectors
1856Sep 24 00:57:02 cloudcephosd1034 smart_failure[1830601]: Device: /dev/sdg [SAT], 1104 Offline uncorrectable sectors
1857Sep 24 00:57:02 cloudcephosd1034 smart_failure[1830620]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1858Sep 24 08:57:02 cloudcephosd1034 smart_failure[2011186]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1859Sep 24 10:57:02 cloudcephosd1034 smart_failure[2056050]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1860Sep 24 11:57:02 cloudcephosd1034 smart_failure[2078362]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1861Sep 24 11:57:02 cloudcephosd1034 smart_failure[2078417]: Device: /dev/sdi [SAT], 504 Offline uncorrectable sectors
1862Sep 24 16:27:02 cloudcephosd1034 smart_failure[2179508]: Device: /dev/sdc [SAT], 2552 Offline uncorrectable sectors
1863Sep 25 00:57:02 cloudcephosd1034 smart_failure[2370769]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1864Sep 25 00:57:02 cloudcephosd1034 smart_failure[2370817]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
1865Sep 25 08:57:02 cloudcephosd1034 smart_failure[2550952]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1866Sep 25 10:57:02 cloudcephosd1034 smart_failure[2596609]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
1867Sep 25 11:57:02 cloudcephosd1034 smart_failure[2619250]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1868Sep 25 11:57:02 cloudcephosd1034 smart_failure[2619269]: Device: /dev/sdi [SAT], 504 Offline uncorrectable sectors
1869Sep 25 16:27:02 cloudcephosd1034 smart_failure[2720500]: Device: /dev/sdc [SAT], 2560 Offline uncorrectable sectors
1870Sep 26 00:57:02 cloudcephosd1034 smart_failure[2914023]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1871Sep 26 00:57:02 cloudcephosd1034 smart_failure[2914042]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1872Sep 26 08:57:02 cloudcephosd1034 smart_failure[3094735]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1873Sep 26 10:57:02 cloudcephosd1034 smart_failure[3140238]: Device: /dev/sde [SAT], 1000 Offline uncorrectable sectors
1874Sep 26 11:57:02 cloudcephosd1034 smart_failure[3162879]: Device: /dev/sdf [SAT], 688 Offline uncorrectable sectors
1875Sep 26 11:57:02 cloudcephosd1034 smart_failure[3162928]: Device: /dev/sdi [SAT], 504 Offline uncorrectable sectors
1876Sep 26 16:27:02 cloudcephosd1034 smart_failure[3265104]: Device: /dev/sdc [SAT], 2560 Offline uncorrectable sectors
1877Sep 27 00:57:02 cloudcephosd1034 smart_failure[3457758]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1878Sep 27 00:57:02 cloudcephosd1034 smart_failure[3457807]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1879Sep 27 08:57:02 cloudcephosd1034 smart_failure[3637627]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1880Sep 27 10:57:02 cloudcephosd1034 smart_failure[3682565]: Device: /dev/sde [SAT], 1008 Offline uncorrectable sectors
1881Sep 27 11:57:02 cloudcephosd1034 smart_failure[3705570]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1882Sep 27 11:57:02 cloudcephosd1034 smart_failure[3705589]: Device: /dev/sdi [SAT], 504 Offline uncorrectable sectors
1883Sep 27 16:27:03 cloudcephosd1034 smart_failure[3806971]: Device: /dev/sdc [SAT], 2584 Offline uncorrectable sectors
1884Sep 28 00:57:02 cloudcephosd1034 smart_failure[3999874]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1885Sep 28 00:57:02 cloudcephosd1034 smart_failure[3999893]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1886Sep 28 08:57:03 cloudcephosd1034 smart_failure[4181115]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1887Sep 28 10:57:02 cloudcephosd1034 smart_failure[32080]: Device: /dev/sde [SAT], 1008 Offline uncorrectable sectors
1888Sep 28 11:57:02 cloudcephosd1034 smart_failure[54981]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1889Sep 28 11:57:02 cloudcephosd1034 smart_failure[55034]: Device: /dev/sdi [SAT], 512 Offline uncorrectable sectors
1890Sep 28 16:57:02 cloudcephosd1034 smart_failure[167331]: Device: /dev/sdc [SAT], 2616 Offline uncorrectable sectors
1891Sep 29 00:57:03 cloudcephosd1034 smart_failure[348374]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1892Sep 29 00:57:03 cloudcephosd1034 smart_failure[348393]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1893Sep 29 08:57:02 cloudcephosd1034 smart_failure[529356]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1894Sep 29 10:57:02 cloudcephosd1034 smart_failure[574466]: Device: /dev/sde [SAT], 1016 Offline uncorrectable sectors
1895Sep 29 11:57:02 cloudcephosd1034 smart_failure[596929]: Device: /dev/sdf [SAT], 696 Offline uncorrectable sectors
1896Sep 29 11:57:02 cloudcephosd1034 smart_failure[596948]: Device: /dev/sdi [SAT], 512 Offline uncorrectable sectors
1897Sep 29 16:57:02 cloudcephosd1034 smart_failure[710163]: Device: /dev/sdc [SAT], 2616 Offline uncorrectable sectors
1898Sep 30 00:57:02 cloudcephosd1034 smart_failure[891940]: Device: /dev/sdg [SAT], 1120 Offline uncorrectable sectors
1899Sep 30 01:27:02 cloudcephosd1034 smart_failure[903192]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
1900Sep 30 08:57:02 cloudcephosd1034 smart_failure[1073826]: Device: /dev/sdd [SAT], 312 Offline uncorrectable sectors
1901Sep 30 10:57:02 cloudcephosd1034 smart_failure[1118986]: Device: /dev/sde [SAT], 1016 Offline uncorrectable sectors
1902Sep 30 11:57:02 cloudcephosd1034 smart_failure[1141756]: Device: /dev/sdf [SAT], 704 Offline uncorrectable sectors
1903Sep 30 11:57:02 cloudcephosd1034 smart_failure[1141805]: Device: /dev/sdi [SAT], 512 Offline uncorrectable sectors
1904Sep 30 16:57:02 cloudcephosd1034 smart_failure[1255456]: Device: /dev/sdc [SAT], 2624 Offline uncorrectable sectors
1905===== NODE GROUP =====
1906(1) cloudcephosd1031.eqiad.wmnet
1907----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
1908Oct 01 01:16:04 cloudcephosd1031 smart_failure[1157501]: Device: /dev/sdd [SAT], 1520 Offline uncorrectable sectors
1909Oct 01 02:16:04 cloudcephosd1031 smart_failure[1180117]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
1910Oct 01 02:46:04 cloudcephosd1031 smart_failure[1191117]: Device: /dev/sdj [SAT], 2432 Offline uncorrectable sectors
1911Oct 01 10:16:04 cloudcephosd1031 smart_failure[1360906]: Device: /dev/sdf [SAT], 1872 Offline uncorrectable sectors
1912Oct 01 12:46:04 cloudcephosd1031 smart_failure[1417386]: Device: /dev/sdg [SAT], 1312 Offline uncorrectable sectors
1913Oct 01 16:46:04 cloudcephosd1031 smart_failure[1508121]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
1914Oct 01 17:16:05 cloudcephosd1031 smart_failure[1519660]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
1915Oct 01 17:46:04 cloudcephosd1031 smart_failure[1530755]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
1916Oct 02 01:16:04 cloudcephosd1031 smart_failure[1699751]: Device: /dev/sdd [SAT], 1520 Offline uncorrectable sectors
1917Oct 02 02:16:04 cloudcephosd1031 smart_failure[1722211]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
1918Oct 02 02:46:04 cloudcephosd1031 smart_failure[1733475]: Device: /dev/sdj [SAT], 2448 Offline uncorrectable sectors
1919Oct 02 10:16:04 cloudcephosd1031 smart_failure[1902268]: Device: /dev/sdf [SAT], 1872 Offline uncorrectable sectors
1920Oct 02 12:46:04 cloudcephosd1031 smart_failure[1958563]: Device: /dev/sdg [SAT], 1312 Offline uncorrectable sectors
1921Oct 02 16:46:04 cloudcephosd1031 smart_failure[2048606]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
1922Oct 02 17:16:04 cloudcephosd1031 smart_failure[2059764]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
1923Oct 02 17:46:04 cloudcephosd1031 smart_failure[2071167]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
1924Oct 03 01:16:04 cloudcephosd1031 smart_failure[2231429]: Device: /dev/sdd [SAT], 1536 Offline uncorrectable sectors
1925Oct 03 02:16:04 cloudcephosd1031 smart_failure[2254029]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
1926Oct 03 02:46:04 cloudcephosd1031 smart_failure[2265459]: Device: /dev/sdj [SAT], 2448 Offline uncorrectable sectors
1927Oct 03 10:16:04 cloudcephosd1031 smart_failure[2434643]: Device: /dev/sdf [SAT], 1880 Offline uncorrectable sectors
1928Oct 03 12:46:04 cloudcephosd1031 smart_failure[2490404]: Device: /dev/sdg [SAT], 1312 Offline uncorrectable sectors
1929Oct 03 16:46:04 cloudcephosd1031 smart_failure[2580257]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
1930Oct 03 17:16:05 cloudcephosd1031 smart_failure[2591549]: Device: /dev/sdi [SAT], 1800 Offline uncorrectable sectors
1931Oct 03 17:46:05 cloudcephosd1031 smart_failure[2602699]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
1932Oct 04 01:16:04 cloudcephosd1031 smart_failure[2772311]: Device: /dev/sdd [SAT], 1536 Offline uncorrectable sectors
1933Oct 04 02:16:04 cloudcephosd1031 smart_failure[2794850]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
1934Oct 04 02:46:04 cloudcephosd1031 smart_failure[2806096]: Device: /dev/sdj [SAT], 2472 Offline uncorrectable sectors
1935Oct 04 10:16:04 cloudcephosd1031 smart_failure[2976317]: Device: /dev/sdf [SAT], 1880 Offline uncorrectable sectors
1936Oct 04 12:46:04 cloudcephosd1031 smart_failure[3032878]: Device: /dev/sdg [SAT], 1312 Offline uncorrectable sectors
1937Oct 04 16:46:04 cloudcephosd1031 smart_failure[3123971]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
1938Oct 04 17:16:04 cloudcephosd1031 smart_failure[3135602]: Device: /dev/sdi [SAT], 1824 Offline uncorrectable sectors
1939Oct 04 18:16:04 cloudcephosd1031 smart_failure[3158133]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
1940Oct 05 01:16:04 cloudcephosd1031 smart_failure[3316049]: Device: /dev/sdd [SAT], 1536 Offline uncorrectable sectors
1941Oct 05 02:16:04 cloudcephosd1031 smart_failure[3338494]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
1942Oct 05 03:16:01 cloudcephosd1031 smart_failure[3361140]: Device: /dev/sdj [SAT], 2472 Offline uncorrectable sectors
1943Oct 05 10:46:01 cloudcephosd1031 smart_failure[3530565]: Device: /dev/sdf [SAT], 1904 Offline uncorrectable sectors
1944Oct 05 13:16:01 cloudcephosd1031 smart_failure[3598663]: Device: /dev/sdg [SAT], 1312 Offline uncorrectable sectors
1945Oct 05 17:16:01 cloudcephosd1031 smart_failure[3689232]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
1946Oct 05 17:46:02 cloudcephosd1031 smart_failure[3700453]: Device: /dev/sdi [SAT], 1824 Offline uncorrectable sectors
1947Oct 05 18:46:01 cloudcephosd1031 smart_failure[3722732]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
1948Oct 06 01:46:01 cloudcephosd1031 smart_failure[3880529]: Device: /dev/sdd [SAT], 1536 Offline uncorrectable sectors
1949Oct 06 02:46:01 cloudcephosd1031 smart_failure[3902940]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
1950Oct 06 03:16:01 cloudcephosd1031 smart_failure[3914346]: Device: /dev/sdj [SAT], 2472 Offline uncorrectable sectors
1951Oct 06 10:46:01 cloudcephosd1031 smart_failure[4083359]: Device: /dev/sdf [SAT], 1904 Offline uncorrectable sectors
1952Oct 06 13:16:01 cloudcephosd1031 smart_failure[4140316]: Device: /dev/sdg [SAT], 1320 Offline uncorrectable sectors
1953Oct 06 17:16:01 cloudcephosd1031 smart_failure[36608]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
1954Oct 06 17:46:01 cloudcephosd1031 smart_failure[47933]: Device: /dev/sdi [SAT], 1824 Offline uncorrectable sectors
1955Oct 06 18:46:01 cloudcephosd1031 smart_failure[70048]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
1956Oct 07 01:46:01 cloudcephosd1031 smart_failure[227299]: Device: /dev/sdd [SAT], 1544 Offline uncorrectable sectors (changed +8)
1957Oct 07 02:46:01 cloudcephosd1031 smart_failure[249584]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
1958Oct 07 03:16:01 cloudcephosd1031 smart_failure[261027]: Device: /dev/sdj [SAT], 2472 Offline uncorrectable sectors
1959Oct 07 10:46:01 cloudcephosd1031 smart_failure[428546]: Device: /dev/sdf [SAT], 1904 Offline uncorrectable sectors
1960Oct 07 13:16:01 cloudcephosd1031 smart_failure[484114]: Device: /dev/sdg [SAT], 1328 Offline uncorrectable sectors
1961Oct 07 17:16:01 cloudcephosd1031 smart_failure[573416]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
1962Oct 07 17:46:01 cloudcephosd1031 smart_failure[584622]: Device: /dev/sdi [SAT], 1824 Offline uncorrectable sectors
1963Oct 07 18:46:02 cloudcephosd1031 smart_failure[607065]: Device: /dev/sdc [SAT], 1520 Offline uncorrectable sectors
1964Oct 08 01:46:01 cloudcephosd1031 smart_failure[763684]: Device: /dev/sdd [SAT], 1544 Offline uncorrectable sectors
1965Oct 08 02:46:01 cloudcephosd1031 smart_failure[786121]: Device: /dev/sde [SAT], 1424 Offline uncorrectable sectors
1966Oct 08 03:16:01 cloudcephosd1031 smart_failure[797192]: Device: /dev/sdj [SAT], 2480 Offline uncorrectable sectors
1967Oct 08 10:46:02 cloudcephosd1031 smart_failure[964501]: Device: /dev/sdf [SAT], 1912 Offline uncorrectable sectors
1968Oct 08 13:16:01 cloudcephosd1031 smart_failure[1019934]: Device: /dev/sdg [SAT], 1336 Offline uncorrectable sectors
1969Oct 08 17:16:01 cloudcephosd1031 smart_failure[1109561]: Device: /dev/sdh [SAT], 784 Offline uncorrectable sectors
1970Oct 08 17:46:01 cloudcephosd1031 smart_failure[1120769]: Device: /dev/sdi [SAT], 1832 Offline uncorrectable sectors
1971Oct 08 19:16:01 cloudcephosd1031 smart_failure[1154562]: Device: /dev/sdc [SAT], 1520 Offline uncorrectable sectors
1972Oct 09 01:46:01 cloudcephosd1031 smart_failure[1299463]: Device: /dev/sdd [SAT], 1544 Offline uncorrectable sectors
1973Oct 09 02:46:02 cloudcephosd1031 smart_failure[1321894]: Device: /dev/sde [SAT], 1424 Offline uncorrectable sectors
1974Oct 09 03:16:01 cloudcephosd1031 smart_failure[1333086]: Device: /dev/sdj [SAT], 2504 Offline uncorrectable sectors
1975Oct 09 10:46:01 cloudcephosd1031 smart_failure[1500172]: Device: /dev/sdf [SAT], 1928 Offline uncorrectable sectors
1976Oct 09 13:16:01 cloudcephosd1031 smart_failure[1556129]: Device: /dev/sdg [SAT], 1336 Offline uncorrectable sectors
1977Oct 09 17:16:01 cloudcephosd1031 smart_failure[1645084]: Device: /dev/sdh [SAT], 784 Offline uncorrectable sectors
1978Oct 09 17:46:01 cloudcephosd1031 smart_failure[1656410]: Device: /dev/sdi [SAT], 1832 Offline uncorrectable sectors
1979Oct 09 19:16:02 cloudcephosd1031 smart_failure[1690308]: Device: /dev/sdc [SAT], 1520 Offline uncorrectable sectors
1980Oct 10 01:46:01 cloudcephosd1031 smart_failure[1835542]: Device: /dev/sdd [SAT], 1560 Offline uncorrectable sectors
1981Oct 10 02:46:01 cloudcephosd1031 smart_failure[1857734]: Device: /dev/sde [SAT], 1432 Offline uncorrectable sectors
1982Oct 10 03:16:01 cloudcephosd1031 smart_failure[1868910]: Device: /dev/sdj [SAT], 2504 Offline uncorrectable sectors
1983Oct 10 10:46:01 cloudcephosd1031 smart_failure[2036597]: Device: /dev/sdf [SAT], 1928 Offline uncorrectable sectors
1984Oct 10 13:16:01 cloudcephosd1031 smart_failure[2092859]: Device: /dev/sdg [SAT], 1336 Offline uncorrectable sectors
1985Oct 10 17:16:01 cloudcephosd1031 smart_failure[2182986]: Device: /dev/sdh [SAT], 784 Offline uncorrectable sectors
1986Oct 10 17:46:01 cloudcephosd1031 smart_failure[2194346]: Device: /dev/sdi [SAT], 1856 Offline uncorrectable sectors
1987Oct 10 19:46:01 cloudcephosd1031 smart_failure[2240190]: Device: /dev/sdc [SAT], 1520 Offline uncorrectable sectors
1988Oct 11 01:46:01 cloudcephosd1031 smart_failure[2375177]: Device: /dev/sdd [SAT], 1568 Offline uncorrectable sectors
1989Oct 11 02:46:01 cloudcephosd1031 smart_failure[2397785]: Device: /dev/sde [SAT], 1448 Offline uncorrectable sectors
1990Oct 11 03:16:02 cloudcephosd1031 smart_failure[2409139]: Device: /dev/sdj [SAT], 2504 Offline uncorrectable sectors
1991Oct 11 10:46:01 cloudcephosd1031 smart_failure[2578970]: Device: /dev/sdf [SAT], 1936 Offline uncorrectable sectors
1992Sep 15 00:46:04 cloudcephosd1031 smart_failure[871831]: Device: /dev/sdd [SAT], 1416 Offline uncorrectable sectors
1993Sep 15 02:16:04 cloudcephosd1031 smart_failure[905645]: Device: /dev/sde [SAT], 1280 Offline uncorrectable sectors
1994Sep 15 02:46:04 cloudcephosd1031 smart_failure[917059]: Device: /dev/sdj [SAT], 2280 Offline uncorrectable sectors
1995Sep 15 10:16:04 cloudcephosd1031 smart_failure[1086662]: Device: /dev/sdf [SAT], 1760 Offline uncorrectable sectors
1996Sep 15 12:46:04 cloudcephosd1031 smart_failure[1142920]: Device: /dev/sdg [SAT], 1272 Offline uncorrectable sectors
1997Sep 15 16:16:04 cloudcephosd1031 smart_failure[1221312]: Device: /dev/sdh [SAT], 728 Offline uncorrectable sectors
1998Sep 15 16:16:04 cloudcephosd1031 smart_failure[1221331]: Device: /dev/sdi [SAT], 1664 Offline uncorrectable sectors
1999Sep 15 17:46:04 cloudcephosd1031 smart_failure[1255528]: Device: /dev/sdc [SAT], 1480 Offline uncorrectable sectors
2000Sep 16 00:46:04 cloudcephosd1031 smart_failure[1413605]: Device: /dev/sdd [SAT], 1416 Offline uncorrectable sectors
2001Sep 16 02:16:04 cloudcephosd1031 smart_failure[1447409]: Device: /dev/sde [SAT], 1288 Offline uncorrectable sectors
2002Sep 16 02:46:05 cloudcephosd1031 smart_failure[1458664]: Device: /dev/sdj [SAT], 2288 Offline uncorrectable sectors
2003Sep 16 10:16:04 cloudcephosd1031 smart_failure[1627645]: Device: /dev/sdf [SAT], 1760 Offline uncorrectable sectors
2004Sep 16 12:46:04 cloudcephosd1031 smart_failure[1683657]: Device: /dev/sdg [SAT], 1272 Offline uncorrectable sectors
2005Sep 16 16:16:04 cloudcephosd1031 smart_failure[1761998]: Device: /dev/sdh [SAT], 728 Offline uncorrectable sectors
2006Sep 16 16:16:04 cloudcephosd1031 smart_failure[1762017]: Device: /dev/sdi [SAT], 1680 Offline uncorrectable sectors
2007Sep 16 17:46:04 cloudcephosd1031 smart_failure[1795813]: Device: /dev/sdc [SAT], 1480 Offline uncorrectable sectors
2008Sep 17 00:46:04 cloudcephosd1031 smart_failure[1953018]: Device: /dev/sdd [SAT], 1416 Offline uncorrectable sectors
2009Sep 17 02:16:04 cloudcephosd1031 smart_failure[1986858]: Device: /dev/sde [SAT], 1288 Offline uncorrectable sectors
2010Sep 17 02:46:04 cloudcephosd1031 smart_failure[1998111]: Device: /dev/sdj [SAT], 2296 Offline uncorrectable sectors
2011Sep 17 10:16:04 cloudcephosd1031 smart_failure[2166567]: Device: /dev/sdf [SAT], 1760 Offline uncorrectable sectors
2012Sep 17 12:46:04 cloudcephosd1031 smart_failure[2222897]: Device: /dev/sdg [SAT], 1272 Offline uncorrectable sectors
2013Sep 17 16:16:04 cloudcephosd1031 smart_failure[2301189]: Device: /dev/sdh [SAT], 736 Offline uncorrectable sectors
2014Sep 17 16:16:04 cloudcephosd1031 smart_failure[2301232]: Device: /dev/sdi [SAT], 1704 Offline uncorrectable sectors
2015Sep 17 17:46:04 cloudcephosd1031 smart_failure[2334804]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2016Sep 18 00:46:04 cloudcephosd1031 smart_failure[2492567]: Device: /dev/sdd [SAT], 1424 Offline uncorrectable sectors
2017Sep 18 02:16:04 cloudcephosd1031 smart_failure[2526240]: Device: /dev/sde [SAT], 1296 Offline uncorrectable sectors
2018Sep 18 02:46:04 cloudcephosd1031 smart_failure[2537617]: Device: /dev/sdj [SAT], 2296 Offline uncorrectable sectors
2019Sep 18 10:16:04 cloudcephosd1031 smart_failure[2705856]: Device: /dev/sdf [SAT], 1760 Offline uncorrectable sectors
2020Sep 18 12:46:04 cloudcephosd1031 smart_failure[2763351]: Device: /dev/sdg [SAT], 1280 Offline uncorrectable sectors
2021Sep 18 16:16:04 cloudcephosd1031 smart_failure[2843030]: Device: /dev/sdh [SAT], 736 Offline uncorrectable sectors
2022Sep 18 16:16:04 cloudcephosd1031 smart_failure[2843049]: Device: /dev/sdi [SAT], 1712 Offline uncorrectable sectors
2023Sep 18 17:46:04 cloudcephosd1031 smart_failure[2876652]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2024Sep 19 00:46:04 cloudcephosd1031 smart_failure[3034481]: Device: /dev/sdd [SAT], 1424 Offline uncorrectable sectors
2025Sep 19 02:16:04 cloudcephosd1031 smart_failure[3068352]: Device: /dev/sde [SAT], 1312 Offline uncorrectable sectors
2026Sep 19 02:46:04 cloudcephosd1031 smart_failure[3079632]: Device: /dev/sdj [SAT], 2320 Offline uncorrectable sectors
2027Sep 19 10:16:04 cloudcephosd1031 smart_failure[3249770]: Device: /dev/sdf [SAT], 1784 Offline uncorrectable sectors
2028Sep 19 12:46:05 cloudcephosd1031 smart_failure[3306246]: Device: /dev/sdg [SAT], 1280 Offline uncorrectable sectors
2029Sep 19 16:16:04 cloudcephosd1031 smart_failure[3385388]: Device: /dev/sdh [SAT], 736 Offline uncorrectable sectors
2030Sep 19 16:16:04 cloudcephosd1031 smart_failure[3385432]: Device: /dev/sdi [SAT], 1720 Offline uncorrectable sectors
2031Sep 19 17:46:04 cloudcephosd1031 smart_failure[3419473]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2032Sep 20 00:46:04 cloudcephosd1031 smart_failure[3579951]: Device: /dev/sdd [SAT], 1440 Offline uncorrectable sectors
2033Sep 20 02:16:04 cloudcephosd1031 smart_failure[3613853]: Device: /dev/sde [SAT], 1336 Offline uncorrectable sectors
2034Sep 20 02:46:04 cloudcephosd1031 smart_failure[3624975]: Device: /dev/sdj [SAT], 2328 Offline uncorrectable sectors
2035Sep 20 10:16:04 cloudcephosd1031 smart_failure[3794048]: Device: /dev/sdf [SAT], 1808 Offline uncorrectable sectors
2036Sep 20 12:46:04 cloudcephosd1031 smart_failure[3850180]: Device: /dev/sdg [SAT], 1288 Offline uncorrectable sectors
2037Sep 20 16:16:04 cloudcephosd1031 smart_failure[3929464]: Device: /dev/sdh [SAT], 744 Offline uncorrectable sectors
2038Sep 20 16:16:04 cloudcephosd1031 smart_failure[3929507]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2039Sep 20 17:46:04 cloudcephosd1031 smart_failure[3962943]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2040Sep 21 00:46:04 cloudcephosd1031 smart_failure[4120629]: Device: /dev/sdd [SAT], 1464 Offline uncorrectable sectors
2041Sep 21 02:16:04 cloudcephosd1031 smart_failure[4154328]: Device: /dev/sde [SAT], 1344 Offline uncorrectable sectors
2042Sep 21 02:46:04 cloudcephosd1031 smart_failure[4165493]: Device: /dev/sdj [SAT], 2344 Offline uncorrectable sectors
2043Sep 21 10:16:04 cloudcephosd1031 smart_failure[139911]: Device: /dev/sdf [SAT], 1808 Offline uncorrectable sectors
2044Sep 21 12:46:04 cloudcephosd1031 smart_failure[196170]: Device: /dev/sdg [SAT], 1296 Offline uncorrectable sectors
2045Sep 21 16:16:04 cloudcephosd1031 smart_failure[275190]: Device: /dev/sdh [SAT], 744 Offline uncorrectable sectors
2046Sep 21 16:16:04 cloudcephosd1031 smart_failure[275233]: Device: /dev/sdi [SAT], 1744 Offline uncorrectable sectors
2047Sep 21 17:46:04 cloudcephosd1031 smart_failure[309529]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2048Sep 22 00:46:04 cloudcephosd1031 smart_failure[466709]: Device: /dev/sdd [SAT], 1464 Offline uncorrectable sectors
2049Sep 22 02:16:04 cloudcephosd1031 smart_failure[500077]: Device: /dev/sde [SAT], 1352 Offline uncorrectable sectors
2050Sep 22 02:46:04 cloudcephosd1031 smart_failure[511586]: Device: /dev/sdj [SAT], 2344 Offline uncorrectable sectors
2051Sep 22 10:16:04 cloudcephosd1031 smart_failure[679912]: Device: /dev/sdf [SAT], 1808 Offline uncorrectable sectors
2052Sep 22 12:46:04 cloudcephosd1031 smart_failure[735628]: Device: /dev/sdg [SAT], 1296 Offline uncorrectable sectors
2053Sep 22 16:16:04 cloudcephosd1031 smart_failure[815107]: Device: /dev/sdh [SAT], 744 Offline uncorrectable sectors
2054Sep 22 16:16:04 cloudcephosd1031 smart_failure[815126]: Device: /dev/sdi [SAT], 1744 Offline uncorrectable sectors
2055Sep 22 17:46:04 cloudcephosd1031 smart_failure[848817]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2056Sep 23 00:46:04 cloudcephosd1031 smart_failure[1007220]: Device: /dev/sdd [SAT], 1464 Offline uncorrectable sectors
2057Sep 23 02:16:04 cloudcephosd1031 smart_failure[1040451]: Device: /dev/sde [SAT], 1360 Offline uncorrectable sectors
2058Sep 23 02:46:04 cloudcephosd1031 smart_failure[1051872]: Device: /dev/sdj [SAT], 2344 Offline uncorrectable sectors
2059Sep 23 10:16:04 cloudcephosd1031 smart_failure[1221176]: Device: /dev/sdf [SAT], 1808 Offline uncorrectable sectors
2060Sep 23 12:46:04 cloudcephosd1031 smart_failure[1277485]: Device: /dev/sdg [SAT], 1296 Offline uncorrectable sectors
2061Sep 23 16:16:04 cloudcephosd1031 smart_failure[1356045]: Device: /dev/sdh [SAT], 744 Offline uncorrectable sectors
2062Sep 23 16:16:04 cloudcephosd1031 smart_failure[1356088]: Device: /dev/sdi [SAT], 1744 Offline uncorrectable sectors
2063Sep 23 17:46:04 cloudcephosd1031 smart_failure[1389570]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2064Sep 24 00:46:04 cloudcephosd1031 smart_failure[1548091]: Device: /dev/sdd [SAT], 1464 Offline uncorrectable sectors
2065Sep 24 02:16:04 cloudcephosd1031 smart_failure[1582723]: Device: /dev/sde [SAT], 1368 Offline uncorrectable sectors
2066Sep 24 02:46:04 cloudcephosd1031 smart_failure[1593812]: Device: /dev/sdj [SAT], 2344 Offline uncorrectable sectors
2067Sep 24 10:16:04 cloudcephosd1031 smart_failure[1762469]: Device: /dev/sdf [SAT], 1816 Offline uncorrectable sectors
2068Sep 24 12:46:04 cloudcephosd1031 smart_failure[1818889]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2069Sep 24 16:16:04 cloudcephosd1031 smart_failure[1897773]: Device: /dev/sdh [SAT], 760 Offline uncorrectable sectors
2070Sep 24 16:16:04 cloudcephosd1031 smart_failure[1897815]: Device: /dev/sdi [SAT], 1760 Offline uncorrectable sectors
2071Sep 24 17:46:04 cloudcephosd1031 smart_failure[1931749]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2072Sep 25 00:46:04 cloudcephosd1031 smart_failure[2088955]: Device: /dev/sdd [SAT], 1464 Offline uncorrectable sectors
2073Sep 25 02:16:04 cloudcephosd1031 smart_failure[2122616]: Device: /dev/sde [SAT], 1384 Offline uncorrectable sectors
2074Sep 25 02:46:04 cloudcephosd1031 smart_failure[2133711]: Device: /dev/sdj [SAT], 2360 Offline uncorrectable sectors
2075Sep 25 10:16:04 cloudcephosd1031 smart_failure[2302247]: Device: /dev/sdf [SAT], 1856 Offline uncorrectable sectors
2076Sep 25 12:46:04 cloudcephosd1031 smart_failure[2359086]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2077Sep 25 16:16:04 cloudcephosd1031 smart_failure[2437319]: Device: /dev/sdh [SAT], 768 Offline uncorrectable sectors
2078Sep 25 16:16:04 cloudcephosd1031 smart_failure[2437361]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
2079Sep 25 17:46:04 cloudcephosd1031 smart_failure[2471438]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2080Sep 26 00:46:04 cloudcephosd1031 smart_failure[2629866]: Device: /dev/sdd [SAT], 1464 Offline uncorrectable sectors
2081Sep 26 02:16:04 cloudcephosd1031 smart_failure[2664145]: Device: /dev/sde [SAT], 1392 Offline uncorrectable sectors
2082Sep 26 02:46:04 cloudcephosd1031 smart_failure[2675700]: Device: /dev/sdj [SAT], 2360 Offline uncorrectable sectors
2083Sep 26 10:16:04 cloudcephosd1031 smart_failure[2844884]: Device: /dev/sdf [SAT], 1856 Offline uncorrectable sectors
2084Sep 26 12:46:04 cloudcephosd1031 smart_failure[2901139]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2085Sep 26 16:16:04 cloudcephosd1031 smart_failure[2979774]: Device: /dev/sdh [SAT], 768 Offline uncorrectable sectors
2086Sep 26 16:16:04 cloudcephosd1031 smart_failure[2979820]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
2087Sep 26 17:46:04 cloudcephosd1031 smart_failure[3013593]: Device: /dev/sdc [SAT], 1496 Offline uncorrectable sectors
2088Sep 27 00:46:04 cloudcephosd1031 smart_failure[3171921]: Device: /dev/sdd [SAT], 1496 Offline uncorrectable sectors
2089Sep 27 02:16:04 cloudcephosd1031 smart_failure[3206084]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
2090Sep 27 02:46:04 cloudcephosd1031 smart_failure[3217232]: Device: /dev/sdj [SAT], 2408 Offline uncorrectable sectors
2091Sep 27 10:16:04 cloudcephosd1031 smart_failure[3385373]: Device: /dev/sdf [SAT], 1864 Offline uncorrectable sectors
2092Sep 27 12:46:04 cloudcephosd1031 smart_failure[3441677]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2093Sep 27 16:16:05 cloudcephosd1031 smart_failure[3521131]: Device: /dev/sdh [SAT], 768 Offline uncorrectable sectors
2094Sep 27 16:16:05 cloudcephosd1031 smart_failure[3521174]: Device: /dev/sdi [SAT], 1776 Offline uncorrectable sectors
2095Sep 27 17:46:04 cloudcephosd1031 smart_failure[3554555]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
2096Sep 28 00:46:04 cloudcephosd1031 smart_failure[3713102]: Device: /dev/sdd [SAT], 1504 Offline uncorrectable sectors
2097Sep 28 02:16:04 cloudcephosd1031 smart_failure[3747371]: Device: /dev/sde [SAT], 1400 Offline uncorrectable sectors
2098Sep 28 02:46:04 cloudcephosd1031 smart_failure[3758658]: Device: /dev/sdj [SAT], 2408 Offline uncorrectable sectors
2099Sep 28 10:16:04 cloudcephosd1031 smart_failure[3927431]: Device: /dev/sdf [SAT], 1872 Offline uncorrectable sectors
2100Sep 28 12:46:04 cloudcephosd1031 smart_failure[3983979]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2101Sep 28 16:46:04 cloudcephosd1031 smart_failure[4073762]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
2102Sep 28 16:46:04 cloudcephosd1031 smart_failure[4073781]: Device: /dev/sdi [SAT], 1784 Offline uncorrectable sectors
2103Sep 28 17:46:04 cloudcephosd1031 smart_failure[4096116]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
2104Sep 29 00:46:05 cloudcephosd1031 smart_failure[60729]: Device: /dev/sdd [SAT], 1504 Offline uncorrectable sectors
2105Sep 29 02:16:04 cloudcephosd1031 smart_failure[94608]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
2106Sep 29 02:46:04 cloudcephosd1031 smart_failure[105793]: Device: /dev/sdj [SAT], 2424 Offline uncorrectable sectors
2107Sep 29 10:16:04 cloudcephosd1031 smart_failure[275322]: Device: /dev/sdf [SAT], 1872 Offline uncorrectable sectors
2108Sep 29 12:46:04 cloudcephosd1031 smart_failure[331972]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2109Sep 29 16:46:05 cloudcephosd1031 smart_failure[422088]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
2110Sep 29 16:46:05 cloudcephosd1031 smart_failure[422107]: Device: /dev/sdi [SAT], 1784 Offline uncorrectable sectors
2111Sep 29 17:46:04 cloudcephosd1031 smart_failure[444698]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
2112Sep 30 01:16:04 cloudcephosd1031 smart_failure[614337]: Device: /dev/sdd [SAT], 1512 Offline uncorrectable sectors
2113Sep 30 02:16:04 cloudcephosd1031 smart_failure[636887]: Device: /dev/sde [SAT], 1408 Offline uncorrectable sectors
2114Sep 30 02:46:04 cloudcephosd1031 smart_failure[647952]: Device: /dev/sdj [SAT], 2424 Offline uncorrectable sectors
2115Sep 30 10:16:04 cloudcephosd1031 smart_failure[818099]: Device: /dev/sdf [SAT], 1872 Offline uncorrectable sectors
2116Sep 30 12:46:04 cloudcephosd1031 smart_failure[874291]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2117Sep 30 16:46:04 cloudcephosd1031 smart_failure[964886]: Device: /dev/sdh [SAT], 776 Offline uncorrectable sectors
2118Sep 30 17:16:04 cloudcephosd1031 smart_failure[976250]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
2119Sep 30 17:46:04 cloudcephosd1031 smart_failure[987397]: Device: /dev/sdc [SAT], 1512 Offline uncorrectable sectors
2120===== NODE GROUP =====
2121(1) cloudcephosd1021.eqiad.wmnet
2122----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
2123Oct 01 02:04:03 cloudcephosd1021 smart_failure[1679504]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
2124Oct 01 09:34:03 cloudcephosd1021 smart_failure[1847836]: Device: /dev/sdh [SAT], 1416 Offline uncorrectable sectors
2125Oct 01 13:04:03 cloudcephosd1021 smart_failure[1926438]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2126Oct 01 14:34:03 cloudcephosd1021 smart_failure[1960315]: Device: /dev/sdg [SAT], 1872 Offline uncorrectable sectors
2127Oct 01 18:04:03 cloudcephosd1021 smart_failure[2038573]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
2128Oct 01 20:34:03 cloudcephosd1021 smart_failure[2095006]: Device: /dev/sdi [SAT], 1744 Offline uncorrectable sectors
2129Oct 01 21:34:03 cloudcephosd1021 smart_failure[2117002]: Device: /dev/sdf [SAT], 1808 Offline uncorrectable sectors
2130Oct 01 21:34:03 cloudcephosd1021 smart_failure[2117051]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2131Oct 02 02:04:03 cloudcephosd1021 smart_failure[2217395]: Device: /dev/sde [SAT], 1008 Offline uncorrectable sectors
2132Oct 02 09:34:03 cloudcephosd1021 smart_failure[2385796]: Device: /dev/sdh [SAT], 1432 Offline uncorrectable sectors
2133Oct 02 13:04:03 cloudcephosd1021 smart_failure[2464713]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2134Oct 02 14:34:03 cloudcephosd1021 smart_failure[2498356]: Device: /dev/sdg [SAT], 1888 Offline uncorrectable sectors
2135Oct 02 18:04:03 cloudcephosd1021 smart_failure[2576952]: Device: /dev/sdc [SAT], 1280 Offline uncorrectable sectors
2136Oct 02 20:34:03 cloudcephosd1021 smart_failure[2633222]: Device: /dev/sdi [SAT], 1760 Offline uncorrectable sectors
2137Oct 02 21:34:03 cloudcephosd1021 smart_failure[2655596]: Device: /dev/sdf [SAT], 1816 Offline uncorrectable sectors
2138Oct 02 21:34:03 cloudcephosd1021 smart_failure[2655642]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
2139Oct 03 02:04:03 cloudcephosd1021 smart_failure[2746639]: Device: /dev/sde [SAT], 1008 Offline uncorrectable sectors
2140Oct 03 09:34:03 cloudcephosd1021 smart_failure[2914790]: Device: /dev/sdh [SAT], 1432 Offline uncorrectable sectors
2141Oct 03 13:04:03 cloudcephosd1021 smart_failure[2993083]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2142Oct 03 14:34:03 cloudcephosd1021 smart_failure[3026351]: Device: /dev/sdg [SAT], 1888 Offline uncorrectable sectors
2143Oct 03 18:04:03 cloudcephosd1021 smart_failure[3104438]: Device: /dev/sdc [SAT], 1304 Offline uncorrectable sectors
2144Oct 03 20:34:03 cloudcephosd1021 smart_failure[3160448]: Device: /dev/sdi [SAT], 1768 Offline uncorrectable sectors
2145Oct 03 21:34:03 cloudcephosd1021 smart_failure[3182921]: Device: /dev/sdf [SAT], 1816 Offline uncorrectable sectors
2146Oct 03 21:34:03 cloudcephosd1021 smart_failure[3182969]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
2147Oct 04 02:04:03 cloudcephosd1021 smart_failure[3283458]: Device: /dev/sde [SAT], 1008 Offline uncorrectable sectors
2148Oct 04 09:34:03 cloudcephosd1021 smart_failure[3452553]: Device: /dev/sdh [SAT], 1448 Offline uncorrectable sectors
2149Oct 04 13:04:03 cloudcephosd1021 smart_failure[3530837]: Device: /dev/sdd [SAT], 552 Offline uncorrectable sectors
2150Oct 04 14:34:03 cloudcephosd1021 smart_failure[3564318]: Device: /dev/sdg [SAT], 1904 Offline uncorrectable sectors
2151Oct 04 18:04:03 cloudcephosd1021 smart_failure[3643542]: Device: /dev/sdc [SAT], 1320 Offline uncorrectable sectors
2152Oct 04 20:34:03 cloudcephosd1021 smart_failure[3699863]: Device: /dev/sdi [SAT], 1776 Offline uncorrectable sectors
2153Oct 04 21:34:03 cloudcephosd1021 smart_failure[3722308]: Device: /dev/sdf [SAT], 1816 Offline uncorrectable sectors
2154Oct 04 21:34:03 cloudcephosd1021 smart_failure[3722327]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
2155Oct 05 02:04:03 cloudcephosd1021 smart_failure[3823593]: Device: /dev/sde [SAT], 1008 Offline uncorrectable sectors
2156Oct 05 09:34:03 cloudcephosd1021 smart_failure[3992200]: Device: /dev/sdh [SAT], 1448 Offline uncorrectable sectors
2157Oct 05 13:04:03 cloudcephosd1021 smart_failure[4083092]: Device: /dev/sdd [SAT], 552 Offline uncorrectable sectors
2158Oct 05 14:34:03 cloudcephosd1021 smart_failure[4116731]: Device: /dev/sdg [SAT], 1928 Offline uncorrectable sectors (changed +8)
2159Oct 05 18:04:03 cloudcephosd1021 smart_failure[1513]: Device: /dev/sdc [SAT], 1320 Offline uncorrectable sectors
2160Oct 05 20:34:03 cloudcephosd1021 smart_failure[57703]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
2161Oct 05 21:34:03 cloudcephosd1021 smart_failure[80363]: Device: /dev/sdf [SAT], 1816 Offline uncorrectable sectors
2162Oct 05 21:34:03 cloudcephosd1021 smart_failure[80412]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
2163Oct 06 02:04:03 cloudcephosd1021 smart_failure[180903]: Device: /dev/sde [SAT], 1008 Offline uncorrectable sectors
2164Oct 06 09:34:03 cloudcephosd1021 smart_failure[350201]: Device: /dev/sdh [SAT], 1448 Offline uncorrectable sectors
2165Oct 06 13:04:03 cloudcephosd1021 smart_failure[429018]: Device: /dev/sdd [SAT], 552 Offline uncorrectable sectors
2166Oct 06 14:34:03 cloudcephosd1021 smart_failure[462925]: Device: /dev/sdg [SAT], 1936 Offline uncorrectable sectors
2167Oct 06 18:04:03 cloudcephosd1021 smart_failure[541818]: Device: /dev/sdc [SAT], 1320 Offline uncorrectable sectors
2168Oct 06 20:34:03 cloudcephosd1021 smart_failure[598359]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
2169Oct 06 21:34:03 cloudcephosd1021 smart_failure[621164]: Device: /dev/sdf [SAT], 1816 Offline uncorrectable sectors
2170Oct 06 21:34:03 cloudcephosd1021 smart_failure[621183]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
2171Oct 07 02:04:03 cloudcephosd1021 smart_failure[722091]: Device: /dev/sde [SAT], 1016 Offline uncorrectable sectors
2172Oct 07 09:34:03 cloudcephosd1021 smart_failure[892344]: Device: /dev/sdh [SAT], 1448 Offline uncorrectable sectors
2173Oct 07 13:04:03 cloudcephosd1021 smart_failure[971143]: Device: /dev/sdd [SAT], 552 Offline uncorrectable sectors
2174Oct 07 14:34:03 cloudcephosd1021 smart_failure[1005223]: Device: /dev/sdg [SAT], 1944 Offline uncorrectable sectors
2175Oct 07 18:04:03 cloudcephosd1021 smart_failure[1084458]: Device: /dev/sdc [SAT], 1328 Offline uncorrectable sectors
2176Oct 07 20:34:03 cloudcephosd1021 smart_failure[1141025]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
2177Oct 07 21:34:03 cloudcephosd1021 smart_failure[1163792]: Device: /dev/sdf [SAT], 1816 Offline uncorrectable sectors
2178Oct 07 21:34:03 cloudcephosd1021 smart_failure[1163841]: Device: /dev/sdj [SAT], 472 Offline uncorrectable sectors
2179Oct 08 02:04:03 cloudcephosd1021 smart_failure[1266183]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2180Oct 08 09:34:03 cloudcephosd1021 smart_failure[1436139]: Device: /dev/sdh [SAT], 1456 Offline uncorrectable sectors
2181Oct 08 13:04:03 cloudcephosd1021 smart_failure[1514882]: Device: /dev/sdd [SAT], 552 Offline uncorrectable sectors
2182Oct 08 14:34:03 cloudcephosd1021 smart_failure[1548948]: Device: /dev/sdg [SAT], 1944 Offline uncorrectable sectors
2183Oct 08 18:04:04 cloudcephosd1021 smart_failure[1628759]: Device: /dev/sdc [SAT], 1328 Offline uncorrectable sectors
2184Oct 08 20:34:03 cloudcephosd1021 smart_failure[1685259]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
2185Oct 08 21:34:03 cloudcephosd1021 smart_failure[1707790]: Device: /dev/sdf [SAT], 1832 Offline uncorrectable sectors
2186Oct 08 21:34:03 cloudcephosd1021 smart_failure[1707839]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2187Oct 09 02:04:03 cloudcephosd1021 smart_failure[1810216]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2188Oct 09 09:34:03 cloudcephosd1021 smart_failure[1979510]: Device: /dev/sdh [SAT], 1456 Offline uncorrectable sectors
2189Oct 09 13:04:03 cloudcephosd1021 smart_failure[2058846]: Device: /dev/sdd [SAT], 552 Offline uncorrectable sectors
2190Oct 09 14:34:03 cloudcephosd1021 smart_failure[2092771]: Device: /dev/sdg [SAT], 1944 Offline uncorrectable sectors
2191Oct 09 18:04:03 cloudcephosd1021 smart_failure[2171803]: Device: /dev/sdc [SAT], 1336 Offline uncorrectable sectors
2192Oct 09 20:34:03 cloudcephosd1021 smart_failure[2228702]: Device: /dev/sdi [SAT], 1792 Offline uncorrectable sectors
2193Oct 09 21:34:03 cloudcephosd1021 smart_failure[2251426]: Device: /dev/sdf [SAT], 1840 Offline uncorrectable sectors
2194Oct 09 21:34:03 cloudcephosd1021 smart_failure[2251480]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2195Oct 10 02:04:03 cloudcephosd1021 smart_failure[2353473]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2196Oct 10 09:34:03 cloudcephosd1021 smart_failure[2523603]: Device: /dev/sdh [SAT], 1472 Offline uncorrectable sectors
2197Oct 10 13:04:03 cloudcephosd1021 smart_failure[2602920]: Device: /dev/sdd [SAT], 552 Offline uncorrectable sectors
2198Oct 10 14:34:03 cloudcephosd1021 smart_failure[2636508]: Device: /dev/sdg [SAT], 1952 Offline uncorrectable sectors
2199Oct 10 18:04:03 cloudcephosd1021 smart_failure[2715512]: Device: /dev/sdc [SAT], 1336 Offline uncorrectable sectors
2200Oct 10 20:34:03 cloudcephosd1021 smart_failure[2772066]: Device: /dev/sdi [SAT], 1800 Offline uncorrectable sectors
2201Oct 10 21:34:03 cloudcephosd1021 smart_failure[2795067]: Device: /dev/sdf [SAT], 1840 Offline uncorrectable sectors
2202Oct 10 21:34:04 cloudcephosd1021 smart_failure[2795086]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2203Oct 11 02:04:03 cloudcephosd1021 smart_failure[2896054]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2204Oct 11 09:34:03 cloudcephosd1021 smart_failure[3065821]: Device: /dev/sdh [SAT], 1472 Offline uncorrectable sectors
2205Sep 08 09:34:03 cloudcephosd1021 smart_failure[1960550]: Device: /dev/sdh [SAT], 1336 Offline uncorrectable sectors
2206Sep 08 12:34:03 cloudcephosd1021 smart_failure[2028644]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2207Sep 08 14:34:03 cloudcephosd1021 smart_failure[2074360]: Device: /dev/sdg [SAT], 1712 Offline uncorrectable sectors
2208Sep 08 17:34:04 cloudcephosd1021 smart_failure[2142441]: Device: /dev/sdc [SAT], 1160 Offline uncorrectable sectors
2209Sep 08 20:34:03 cloudcephosd1021 smart_failure[2210841]: Device: /dev/sdi [SAT], 1656 Offline uncorrectable sectors
2210Sep 08 21:04:03 cloudcephosd1021 smart_failure[2222243]: Device: /dev/sdj [SAT], 440 Offline uncorrectable sectors
2211Sep 08 21:34:03 cloudcephosd1021 smart_failure[2233744]: Device: /dev/sdf [SAT], 1656 Offline uncorrectable sectors
2212Sep 09 01:34:03 cloudcephosd1021 smart_failure[2325206]: Device: /dev/sde [SAT], 960 Offline uncorrectable sectors
2213Sep 09 09:34:03 cloudcephosd1021 smart_failure[2507500]: Device: /dev/sdh [SAT], 1336 Offline uncorrectable sectors
2214Sep 09 12:34:03 cloudcephosd1021 smart_failure[2575401]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2215Sep 09 14:34:03 cloudcephosd1021 smart_failure[2620761]: Device: /dev/sdg [SAT], 1720 Offline uncorrectable sectors
2216Sep 09 18:04:03 cloudcephosd1021 smart_failure[2700241]: Device: /dev/sdc [SAT], 1184 Offline uncorrectable sectors
2217Sep 09 20:34:03 cloudcephosd1021 smart_failure[2756608]: Device: /dev/sdi [SAT], 1656 Offline uncorrectable sectors
2218Sep 09 21:04:04 cloudcephosd1021 smart_failure[2768169]: Device: /dev/sdj [SAT], 448 Offline uncorrectable sectors (changed +8)
2219Sep 09 21:34:03 cloudcephosd1021 smart_failure[2779471]: Device: /dev/sdf [SAT], 1672 Offline uncorrectable sectors
2220Sep 10 01:34:03 cloudcephosd1021 smart_failure[2870962]: Device: /dev/sde [SAT], 960 Offline uncorrectable sectors
2221Sep 10 09:34:03 cloudcephosd1021 smart_failure[3053152]: Device: /dev/sdh [SAT], 1336 Offline uncorrectable sectors
2222Sep 10 12:34:03 cloudcephosd1021 smart_failure[3121523]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2223Sep 10 14:34:03 cloudcephosd1021 smart_failure[3167215]: Device: /dev/sdg [SAT], 1720 Offline uncorrectable sectors
2224Sep 10 18:04:03 cloudcephosd1021 smart_failure[3246855]: Device: /dev/sdc [SAT], 1184 Offline uncorrectable sectors
2225Sep 10 20:34:03 cloudcephosd1021 smart_failure[3303468]: Device: /dev/sdi [SAT], 1656 Offline uncorrectable sectors
2226Sep 10 21:34:03 cloudcephosd1021 smart_failure[3326080]: Device: /dev/sdf [SAT], 1680 Offline uncorrectable sectors
2227Sep 10 21:34:03 cloudcephosd1021 smart_failure[3326128]: Device: /dev/sdj [SAT], 448 Offline uncorrectable sectors
2228Sep 11 01:34:04 cloudcephosd1021 smart_failure[3417543]: Device: /dev/sde [SAT], 960 Offline uncorrectable sectors
2229Sep 11 09:34:03 cloudcephosd1021 smart_failure[3599091]: Device: /dev/sdh [SAT], 1336 Offline uncorrectable sectors
2230Sep 11 12:34:03 cloudcephosd1021 smart_failure[3667506]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2231Sep 11 14:34:03 cloudcephosd1021 smart_failure[3712300]: Device: /dev/sdg [SAT], 1728 Offline uncorrectable sectors
2232Sep 11 18:04:03 cloudcephosd1021 smart_failure[3791387]: Device: /dev/sdc [SAT], 1184 Offline uncorrectable sectors
2233Sep 11 20:34:03 cloudcephosd1021 smart_failure[3848158]: Device: /dev/sdi [SAT], 1664 Offline uncorrectable sectors
2234Sep 11 21:34:03 cloudcephosd1021 smart_failure[3870891]: Device: /dev/sdf [SAT], 1688 Offline uncorrectable sectors
2235Sep 11 21:34:03 cloudcephosd1021 smart_failure[3870910]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2236Sep 12 02:04:03 cloudcephosd1021 smart_failure[3972119]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
2237Sep 12 09:34:03 cloudcephosd1021 smart_failure[4141434]: Device: /dev/sdh [SAT], 1336 Offline uncorrectable sectors
2238Sep 12 12:34:03 cloudcephosd1021 smart_failure[15413]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2239Sep 12 14:34:03 cloudcephosd1021 smart_failure[60460]: Device: /dev/sdg [SAT], 1736 Offline uncorrectable sectors
2240Sep 12 18:04:03 cloudcephosd1021 smart_failure[139206]: Device: /dev/sdc [SAT], 1192 Offline uncorrectable sectors
2241Sep 12 20:34:03 cloudcephosd1021 smart_failure[195591]: Device: /dev/sdi [SAT], 1672 Offline uncorrectable sectors
2242Sep 12 21:34:03 cloudcephosd1021 smart_failure[217762]: Device: /dev/sdf [SAT], 1688 Offline uncorrectable sectors
2243Sep 12 21:34:03 cloudcephosd1021 smart_failure[217811]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2244Sep 13 02:04:03 cloudcephosd1021 smart_failure[319545]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
2245Sep 13 09:34:03 cloudcephosd1021 smart_failure[489269]: Device: /dev/sdh [SAT], 1344 Offline uncorrectable sectors
2246Sep 13 12:34:03 cloudcephosd1021 smart_failure[556523]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2247Sep 13 14:34:03 cloudcephosd1021 smart_failure[601960]: Device: /dev/sdg [SAT], 1736 Offline uncorrectable sectors
2248Sep 13 18:04:03 cloudcephosd1021 smart_failure[680533]: Device: /dev/sdc [SAT], 1192 Offline uncorrectable sectors
2249Sep 13 20:34:03 cloudcephosd1021 smart_failure[736983]: Device: /dev/sdi [SAT], 1672 Offline uncorrectable sectors
2250Sep 13 21:34:03 cloudcephosd1021 smart_failure[759806]: Device: /dev/sdf [SAT], 1688 Offline uncorrectable sectors
2251Sep 13 21:34:03 cloudcephosd1021 smart_failure[759825]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2252Sep 14 02:04:03 cloudcephosd1021 smart_failure[861395]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
2253Sep 14 09:34:03 cloudcephosd1021 smart_failure[1031520]: Device: /dev/sdh [SAT], 1344 Offline uncorrectable sectors
2254Sep 14 12:34:04 cloudcephosd1021 smart_failure[1098890]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2255Sep 14 14:34:03 cloudcephosd1021 smart_failure[1143747]: Device: /dev/sdg [SAT], 1744 Offline uncorrectable sectors
2256Sep 14 18:04:03 cloudcephosd1021 smart_failure[1223041]: Device: /dev/sdc [SAT], 1192 Offline uncorrectable sectors
2257Sep 14 20:34:03 cloudcephosd1021 smart_failure[1279555]: Device: /dev/sdi [SAT], 1680 Offline uncorrectable sectors
2258Sep 14 21:34:03 cloudcephosd1021 smart_failure[1302043]: Device: /dev/sdf [SAT], 1696 Offline uncorrectable sectors
2259Sep 14 21:34:03 cloudcephosd1021 smart_failure[1302092]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2260Sep 15 02:04:03 cloudcephosd1021 smart_failure[1404485]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
2261Sep 15 09:34:03 cloudcephosd1021 smart_failure[1573969]: Device: /dev/sdh [SAT], 1344 Offline uncorrectable sectors
2262Sep 15 13:04:03 cloudcephosd1021 smart_failure[1653175]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2263Sep 15 14:34:03 cloudcephosd1021 smart_failure[1687152]: Device: /dev/sdg [SAT], 1744 Offline uncorrectable sectors
2264Sep 15 18:04:03 cloudcephosd1021 smart_failure[1766069]: Device: /dev/sdc [SAT], 1200 Offline uncorrectable sectors
2265Sep 15 20:34:03 cloudcephosd1021 smart_failure[1822967]: Device: /dev/sdi [SAT], 1688 Offline uncorrectable sectors
2266Sep 15 21:34:03 cloudcephosd1021 smart_failure[1845442]: Device: /dev/sdf [SAT], 1704 Offline uncorrectable sectors
2267Sep 15 21:34:03 cloudcephosd1021 smart_failure[1845490]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2268Sep 16 02:04:03 cloudcephosd1021 smart_failure[1947719]: Device: /dev/sde [SAT], 968 Offline uncorrectable sectors
2269Sep 16 09:34:03 cloudcephosd1021 smart_failure[2117344]: Device: /dev/sdh [SAT], 1344 Offline uncorrectable sectors
2270Sep 16 13:04:03 cloudcephosd1021 smart_failure[2196151]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2271Sep 16 14:34:03 cloudcephosd1021 smart_failure[2230325]: Device: /dev/sdg [SAT], 1752 Offline uncorrectable sectors
2272Sep 16 18:04:03 cloudcephosd1021 smart_failure[2309011]: Device: /dev/sdc [SAT], 1216 Offline uncorrectable sectors
2273Sep 16 20:34:03 cloudcephosd1021 smart_failure[2365668]: Device: /dev/sdi [SAT], 1688 Offline uncorrectable sectors
2274Sep 16 21:34:03 cloudcephosd1021 smart_failure[2388314]: Device: /dev/sdf [SAT], 1712 Offline uncorrectable sectors
2275Sep 16 21:34:03 cloudcephosd1021 smart_failure[2388372]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2276Sep 17 02:04:03 cloudcephosd1021 smart_failure[2489931]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
2277Sep 17 09:34:03 cloudcephosd1021 smart_failure[2659394]: Device: /dev/sdh [SAT], 1344 Offline uncorrectable sectors
2278Sep 17 13:04:03 cloudcephosd1021 smart_failure[2738313]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2279Sep 17 14:34:03 cloudcephosd1021 smart_failure[2772308]: Device: /dev/sdg [SAT], 1776 Offline uncorrectable sectors
2280Sep 17 18:04:03 cloudcephosd1021 smart_failure[2851088]: Device: /dev/sdc [SAT], 1216 Offline uncorrectable sectors
2281Sep 17 20:34:03 cloudcephosd1021 smart_failure[2907659]: Device: /dev/sdi [SAT], 1688 Offline uncorrectable sectors
2282Sep 17 21:34:03 cloudcephosd1021 smart_failure[2930298]: Device: /dev/sdf [SAT], 1712 Offline uncorrectable sectors
2283Sep 17 21:34:03 cloudcephosd1021 smart_failure[2930347]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2284Sep 18 02:04:03 cloudcephosd1021 smart_failure[3032190]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
2285Sep 18 09:34:03 cloudcephosd1021 smart_failure[3201265]: Device: /dev/sdh [SAT], 1360 Offline uncorrectable sectors
2286Sep 18 13:04:03 cloudcephosd1021 smart_failure[3280457]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2287Sep 18 14:34:03 cloudcephosd1021 smart_failure[3314482]: Device: /dev/sdg [SAT], 1792 Offline uncorrectable sectors
2288Sep 18 18:04:03 cloudcephosd1021 smart_failure[3393173]: Device: /dev/sdc [SAT], 1216 Offline uncorrectable sectors
2289Sep 18 20:34:03 cloudcephosd1021 smart_failure[3449225]: Device: /dev/sdi [SAT], 1688 Offline uncorrectable sectors
2290Sep 18 21:34:03 cloudcephosd1021 smart_failure[3471826]: Device: /dev/sdf [SAT], 1744 Offline uncorrectable sectors
2291Sep 18 21:34:03 cloudcephosd1021 smart_failure[3471876]: Device: /dev/sdj [SAT], 456 Offline uncorrectable sectors
2292Sep 19 02:04:03 cloudcephosd1021 smart_failure[3573520]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
2293Sep 19 09:34:03 cloudcephosd1021 smart_failure[3742703]: Device: /dev/sdh [SAT], 1360 Offline uncorrectable sectors
2294Sep 19 13:04:03 cloudcephosd1021 smart_failure[3821469]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2295Sep 19 14:34:03 cloudcephosd1021 smart_failure[3855830]: Device: /dev/sdg [SAT], 1792 Offline uncorrectable sectors
2296Sep 19 18:04:03 cloudcephosd1021 smart_failure[3935508]: Device: /dev/sdc [SAT], 1224 Offline uncorrectable sectors
2297Sep 19 20:34:03 cloudcephosd1021 smart_failure[3992848]: Device: /dev/sdi [SAT], 1704 Offline uncorrectable sectors
2298Sep 19 21:34:03 cloudcephosd1021 smart_failure[4015604]: Device: /dev/sdf [SAT], 1744 Offline uncorrectable sectors
2299Sep 19 21:34:03 cloudcephosd1021 smart_failure[4015654]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2300Sep 20 02:04:03 cloudcephosd1021 smart_failure[4118602]: Device: /dev/sde [SAT], 976 Offline uncorrectable sectors
2301Sep 20 09:34:03 cloudcephosd1021 smart_failure[95337]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
2302Sep 20 13:04:03 cloudcephosd1021 smart_failure[174307]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2303Sep 20 14:34:03 cloudcephosd1021 smart_failure[208659]: Device: /dev/sdg [SAT], 1792 Offline uncorrectable sectors
2304Sep 20 18:04:03 cloudcephosd1021 smart_failure[287167]: Device: /dev/sdc [SAT], 1224 Offline uncorrectable sectors
2305Sep 20 20:34:03 cloudcephosd1021 smart_failure[343813]: Device: /dev/sdi [SAT], 1712 Offline uncorrectable sectors
2306Sep 20 21:34:03 cloudcephosd1021 smart_failure[366556]: Device: /dev/sdf [SAT], 1768 Offline uncorrectable sectors
2307Sep 20 21:34:03 cloudcephosd1021 smart_failure[366604]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2308Sep 21 02:04:03 cloudcephosd1021 smart_failure[468749]: Device: /dev/sde [SAT], 984 Offline uncorrectable sectors
2309Sep 21 09:34:03 cloudcephosd1021 smart_failure[637805]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
2310Sep 21 13:04:03 cloudcephosd1021 smart_failure[716714]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2311Sep 21 14:34:03 cloudcephosd1021 smart_failure[750950]: Device: /dev/sdg [SAT], 1800 Offline uncorrectable sectors
2312Sep 21 18:04:03 cloudcephosd1021 smart_failure[830766]: Device: /dev/sdc [SAT], 1224 Offline uncorrectable sectors
2313Sep 21 20:34:03 cloudcephosd1021 smart_failure[886887]: Device: /dev/sdi [SAT], 1712 Offline uncorrectable sectors
2314Sep 21 21:34:03 cloudcephosd1021 smart_failure[909639]: Device: /dev/sdf [SAT], 1784 Offline uncorrectable sectors
2315Sep 21 21:34:03 cloudcephosd1021 smart_failure[909689]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2316Sep 22 02:04:03 cloudcephosd1021 smart_failure[1011090]: Device: /dev/sde [SAT], 984 Offline uncorrectable sectors
2317Sep 22 09:34:03 cloudcephosd1021 smart_failure[1180956]: Device: /dev/sdh [SAT], 1368 Offline uncorrectable sectors
2318Sep 22 13:04:03 cloudcephosd1021 smart_failure[1259244]: Device: /dev/sdd [SAT], 528 Offline uncorrectable sectors
2319Sep 22 14:34:03 cloudcephosd1021 smart_failure[1293384]: Device: /dev/sdg [SAT], 1800 Offline uncorrectable sectors
2320Sep 22 18:04:03 cloudcephosd1021 smart_failure[1372707]: Device: /dev/sdc [SAT], 1224 Offline uncorrectable sectors
2321Sep 22 20:34:03 cloudcephosd1021 smart_failure[1429156]: Device: /dev/sdi [SAT], 1712 Offline uncorrectable sectors
2322Sep 22 21:34:03 cloudcephosd1021 smart_failure[1451622]: Device: /dev/sdf [SAT], 1784 Offline uncorrectable sectors
2323Sep 22 21:34:03 cloudcephosd1021 smart_failure[1451671]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2324Sep 23 02:04:03 cloudcephosd1021 smart_failure[1553256]: Device: /dev/sde [SAT], 984 Offline uncorrectable sectors
2325Sep 23 09:34:03 cloudcephosd1021 smart_failure[1723149]: Device: /dev/sdh [SAT], 1384 Offline uncorrectable sectors
2326Sep 23 13:04:03 cloudcephosd1021 smart_failure[1801509]: Device: /dev/sdd [SAT], 536 Offline uncorrectable sectors
2327Sep 23 14:34:03 cloudcephosd1021 smart_failure[1835201]: Device: /dev/sdg [SAT], 1816 Offline uncorrectable sectors
2328Sep 23 18:04:03 cloudcephosd1021 smart_failure[1913894]: Device: /dev/sdc [SAT], 1240 Offline uncorrectable sectors
2329Sep 23 20:34:03 cloudcephosd1021 smart_failure[1970283]: Device: /dev/sdi [SAT], 1712 Offline uncorrectable sectors
2330Sep 23 21:34:03 cloudcephosd1021 smart_failure[1992640]: Device: /dev/sdf [SAT], 1792 Offline uncorrectable sectors
2331Sep 23 21:34:03 cloudcephosd1021 smart_failure[1992691]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2332Sep 24 02:04:03 cloudcephosd1021 smart_failure[2094253]: Device: /dev/sde [SAT], 984 Offline uncorrectable sectors
2333Sep 24 09:34:03 cloudcephosd1021 smart_failure[2263675]: Device: /dev/sdh [SAT], 1392 Offline uncorrectable sectors
2334Sep 24 13:04:03 cloudcephosd1021 smart_failure[2341636]: Device: /dev/sdd [SAT], 536 Offline uncorrectable sectors
2335Sep 24 14:34:03 cloudcephosd1021 smart_failure[2375204]: Device: /dev/sdg [SAT], 1824 Offline uncorrectable sectors
2336Sep 24 18:04:03 cloudcephosd1021 smart_failure[2454057]: Device: /dev/sdc [SAT], 1248 Offline uncorrectable sectors
2337Sep 24 20:34:03 cloudcephosd1021 smart_failure[2510928]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2338Sep 24 21:34:03 cloudcephosd1021 smart_failure[2533281]: Device: /dev/sdf [SAT], 1800 Offline uncorrectable sectors
2339Sep 24 21:34:03 cloudcephosd1021 smart_failure[2533329]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2340Sep 25 02:04:03 cloudcephosd1021 smart_failure[2634478]: Device: /dev/sde [SAT], 984 Offline uncorrectable sectors
2341Sep 25 09:34:03 cloudcephosd1021 smart_failure[2803647]: Device: /dev/sdh [SAT], 1392 Offline uncorrectable sectors
2342Sep 25 13:04:03 cloudcephosd1021 smart_failure[2882627]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2343Sep 25 14:34:03 cloudcephosd1021 smart_failure[2916824]: Device: /dev/sdg [SAT], 1824 Offline uncorrectable sectors
2344Sep 25 18:04:03 cloudcephosd1021 smart_failure[2995613]: Device: /dev/sdc [SAT], 1248 Offline uncorrectable sectors
2345Sep 25 20:34:03 cloudcephosd1021 smart_failure[3051954]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2346Sep 25 21:34:03 cloudcephosd1021 smart_failure[3074595]: Device: /dev/sdf [SAT], 1800 Offline uncorrectable sectors
2347Sep 25 21:34:03 cloudcephosd1021 smart_failure[3074644]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2348Sep 26 02:04:03 cloudcephosd1021 smart_failure[3175887]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
2349Sep 26 09:34:03 cloudcephosd1021 smart_failure[3345170]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
2350Sep 26 13:04:03 cloudcephosd1021 smart_failure[3424141]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2351Sep 26 14:34:03 cloudcephosd1021 smart_failure[3458231]: Device: /dev/sdg [SAT], 1824 Offline uncorrectable sectors
2352Sep 26 18:04:03 cloudcephosd1021 smart_failure[3536571]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
2353Sep 26 20:34:03 cloudcephosd1021 smart_failure[3592805]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2354Sep 26 21:34:03 cloudcephosd1021 smart_failure[3615396]: Device: /dev/sdf [SAT], 1800 Offline uncorrectable sectors
2355Sep 26 21:34:04 cloudcephosd1021 smart_failure[3615445]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2356Sep 27 02:04:03 cloudcephosd1021 smart_failure[3716594]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
2357Sep 27 09:34:03 cloudcephosd1021 smart_failure[3884550]: Device: /dev/sdh [SAT], 1400 Offline uncorrectable sectors
2358Sep 27 13:04:03 cloudcephosd1021 smart_failure[3963560]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2359Sep 27 14:34:03 cloudcephosd1021 smart_failure[3997891]: Device: /dev/sdg [SAT], 1848 Offline uncorrectable sectors
2360Sep 27 18:04:03 cloudcephosd1021 smart_failure[4076830]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
2361Sep 27 20:34:03 cloudcephosd1021 smart_failure[4132675]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2362Sep 27 21:34:03 cloudcephosd1021 smart_failure[4155379]: Device: /dev/sdf [SAT], 1800 Offline uncorrectable sectors
2363Sep 27 21:34:03 cloudcephosd1021 smart_failure[4155428]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2364Sep 28 02:04:03 cloudcephosd1021 smart_failure[63196]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
2365Sep 28 09:34:03 cloudcephosd1021 smart_failure[231713]: Device: /dev/sdh [SAT], 1408 Offline uncorrectable sectors
2366Sep 28 13:04:03 cloudcephosd1021 smart_failure[310861]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2367Sep 28 14:34:03 cloudcephosd1021 smart_failure[344480]: Device: /dev/sdg [SAT], 1864 Offline uncorrectable sectors
2368Sep 28 18:04:03 cloudcephosd1021 smart_failure[423374]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
2369Sep 28 20:34:03 cloudcephosd1021 smart_failure[479913]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2370Sep 28 21:34:03 cloudcephosd1021 smart_failure[502273]: Device: /dev/sdf [SAT], 1800 Offline uncorrectable sectors
2371Sep 28 21:34:03 cloudcephosd1021 smart_failure[502321]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2372Sep 29 02:04:03 cloudcephosd1021 smart_failure[603266]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
2373Sep 29 09:34:03 cloudcephosd1021 smart_failure[771657]: Device: /dev/sdh [SAT], 1416 Offline uncorrectable sectors
2374Sep 29 13:04:03 cloudcephosd1021 smart_failure[849010]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2375Sep 29 14:34:03 cloudcephosd1021 smart_failure[882432]: Device: /dev/sdg [SAT], 1864 Offline uncorrectable sectors
2376Sep 29 18:04:03 cloudcephosd1021 smart_failure[960797]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
2377Sep 29 20:34:03 cloudcephosd1021 smart_failure[1017186]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2378Sep 29 21:34:03 cloudcephosd1021 smart_failure[1039716]: Device: /dev/sdf [SAT], 1800 Offline uncorrectable sectors
2379Sep 29 21:34:03 cloudcephosd1021 smart_failure[1039764]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2380Sep 30 02:04:03 cloudcephosd1021 smart_failure[1140323]: Device: /dev/sde [SAT], 992 Offline uncorrectable sectors
2381Sep 30 09:34:03 cloudcephosd1021 smart_failure[1309236]: Device: /dev/sdh [SAT], 1416 Offline uncorrectable sectors
2382Sep 30 13:04:03 cloudcephosd1021 smart_failure[1387478]: Device: /dev/sdd [SAT], 544 Offline uncorrectable sectors
2383Sep 30 14:34:03 cloudcephosd1021 smart_failure[1421444]: Device: /dev/sdg [SAT], 1872 Offline uncorrectable sectors (changed +8)
2384Sep 30 18:04:03 cloudcephosd1021 smart_failure[1499685]: Device: /dev/sdc [SAT], 1272 Offline uncorrectable sectors
2385Sep 30 20:34:03 cloudcephosd1021 smart_failure[1556241]: Device: /dev/sdi [SAT], 1728 Offline uncorrectable sectors
2386Sep 30 21:34:03 cloudcephosd1021 smart_failure[1578589]: Device: /dev/sdf [SAT], 1800 Offline uncorrectable sectors
2387Sep 30 21:34:03 cloudcephosd1021 smart_failure[1578638]: Device: /dev/sdj [SAT], 464 Offline uncorrectable sectors
2388===== NODE GROUP =====
2389(1) cloudcephosd1028.eqiad.wmnet
2390----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
2391Oct 01 01:37:06 cloudcephosd1028 smart_failure[1397252]: Device: /dev/sde [SAT], 1816 Offline uncorrectable sectors
2392Oct 01 03:37:06 cloudcephosd1028 smart_failure[1441900]: Device: /dev/sdf [SAT], 5336 Offline uncorrectable sectors
2393Oct 01 06:07:06 cloudcephosd1028 smart_failure[1497664]: Device: /dev/sdi [SAT], 992 Offline uncorrectable sectors
2394Oct 01 09:37:06 cloudcephosd1028 smart_failure[1575638]: Device: /dev/sdh [SAT], 856 Offline uncorrectable sectors
2395Oct 01 17:37:06 cloudcephosd1028 smart_failure[1753963]: Device: /dev/sdg [SAT], 1864 Offline uncorrectable sectors
2396Oct 01 19:07:06 cloudcephosd1028 smart_failure[1787979]: Device: /dev/sdc [SAT], 1008 Offline uncorrectable sectors
2397Oct 01 19:07:06 cloudcephosd1028 smart_failure[1787998]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2398Oct 01 22:37:06 cloudcephosd1028 smart_failure[1865509]: Device: /dev/sdj [SAT], 2920 Offline uncorrectable sectors
2399Oct 02 01:37:06 cloudcephosd1028 smart_failure[1932375]: Device: /dev/sde [SAT], 1824 Offline uncorrectable sectors
2400Oct 02 03:37:06 cloudcephosd1028 smart_failure[1976928]: Device: /dev/sdf [SAT], 5360 Offline uncorrectable sectors
2401Oct 02 06:07:06 cloudcephosd1028 smart_failure[2032407]: Device: /dev/sdi [SAT], 992 Offline uncorrectable sectors
2402Oct 02 09:37:06 cloudcephosd1028 smart_failure[2110215]: Device: /dev/sdh [SAT], 856 Offline uncorrectable sectors
2403Oct 02 17:37:06 cloudcephosd1028 smart_failure[2289792]: Device: /dev/sdg [SAT], 1872 Offline uncorrectable sectors
2404Oct 02 19:07:06 cloudcephosd1028 smart_failure[2323437]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2405Oct 02 19:07:06 cloudcephosd1028 smart_failure[2323482]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2406Oct 02 22:37:06 cloudcephosd1028 smart_failure[2399442]: Device: /dev/sdj [SAT], 2936 Offline uncorrectable sectors
2407Oct 03 01:37:06 cloudcephosd1028 smart_failure[2459096]: Device: /dev/sde [SAT], 1824 Offline uncorrectable sectors
2408Oct 03 03:37:06 cloudcephosd1028 smart_failure[2503963]: Device: /dev/sdf [SAT], 5400 Offline uncorrectable sectors
2409Oct 03 06:07:06 cloudcephosd1028 smart_failure[2559517]: Device: /dev/sdi [SAT], 1000 Offline uncorrectable sectors
2410Oct 03 09:37:06 cloudcephosd1028 smart_failure[2637025]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2411Oct 03 17:37:06 cloudcephosd1028 smart_failure[2817100]: Device: /dev/sdg [SAT], 1872 Offline uncorrectable sectors
2412Oct 03 19:07:06 cloudcephosd1028 smart_failure[2850402]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2413Oct 03 19:07:06 cloudcephosd1028 smart_failure[2850421]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2414Oct 03 22:37:06 cloudcephosd1028 smart_failure[2928677]: Device: /dev/sdj [SAT], 2968 Offline uncorrectable sectors
2415Oct 04 01:37:06 cloudcephosd1028 smart_failure[2995462]: Device: /dev/sde [SAT], 1824 Offline uncorrectable sectors
2416Oct 04 03:37:06 cloudcephosd1028 smart_failure[3039978]: Device: /dev/sdf [SAT], 5416 Offline uncorrectable sectors
2417Oct 04 06:07:06 cloudcephosd1028 smart_failure[3095525]: Device: /dev/sdi [SAT], 1008 Offline uncorrectable sectors
2418Oct 04 09:37:06 cloudcephosd1028 smart_failure[3174680]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2419Oct 04 17:37:06 cloudcephosd1028 smart_failure[3355165]: Device: /dev/sdg [SAT], 1880 Offline uncorrectable sectors
2420Oct 04 19:07:06 cloudcephosd1028 smart_failure[3388456]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2421Oct 04 19:07:06 cloudcephosd1028 smart_failure[3388475]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2422Oct 04 22:37:07 cloudcephosd1028 smart_failure[3466964]: Device: /dev/sdj [SAT], 2968 Offline uncorrectable sectors
2423Oct 05 01:37:06 cloudcephosd1028 smart_failure[3534683]: Device: /dev/sde [SAT], 1832 Offline uncorrectable sectors
2424Oct 05 04:07:05 cloudcephosd1028 smart_failure[3590658]: Device: /dev/sdf [SAT], 5448 Offline uncorrectable sectors
2425Oct 05 06:37:05 cloudcephosd1028 smart_failure[3646964]: Device: /dev/sdi [SAT], 1016 Offline uncorrectable sectors
2426Oct 05 10:07:05 cloudcephosd1028 smart_failure[3725507]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2427Oct 05 18:07:05 cloudcephosd1028 smart_failure[3916793]: Device: /dev/sdg [SAT], 1888 Offline uncorrectable sectors
2428Oct 05 19:37:05 cloudcephosd1028 smart_failure[3950376]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2429Oct 05 19:37:05 cloudcephosd1028 smart_failure[3950424]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2430Oct 05 23:07:05 cloudcephosd1028 smart_failure[4028413]: Device: /dev/sdj [SAT], 2976 Offline uncorrectable sectors
2431Oct 06 02:07:05 cloudcephosd1028 smart_failure[4095445]: Device: /dev/sde [SAT], 1832 Offline uncorrectable sectors
2432Oct 06 04:07:05 cloudcephosd1028 smart_failure[4140644]: Device: /dev/sdf [SAT], 5472 Offline uncorrectable sectors
2433Oct 06 06:37:05 cloudcephosd1028 smart_failure[3027]: Device: /dev/sdi [SAT], 1016 Offline uncorrectable sectors
2434Oct 06 10:07:05 cloudcephosd1028 smart_failure[81485]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2435Oct 06 18:07:05 cloudcephosd1028 smart_failure[262509]: Device: /dev/sdg [SAT], 1896 Offline uncorrectable sectors
2436Oct 06 19:37:05 cloudcephosd1028 smart_failure[296193]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2437Oct 06 19:37:05 cloudcephosd1028 smart_failure[296235]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2438Oct 06 23:07:05 cloudcephosd1028 smart_failure[374563]: Device: /dev/sdj [SAT], 2976 Offline uncorrectable sectors
2439Oct 07 02:07:05 cloudcephosd1028 smart_failure[442435]: Device: /dev/sde [SAT], 1832 Offline uncorrectable sectors
2440Oct 07 04:07:05 cloudcephosd1028 smart_failure[487649]: Device: /dev/sdf [SAT], 5472 Offline uncorrectable sectors
2441Oct 07 06:37:05 cloudcephosd1028 smart_failure[543964]: Device: /dev/sdi [SAT], 1016 Offline uncorrectable sectors
2442Oct 07 10:07:05 cloudcephosd1028 smart_failure[623078]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2443Oct 07 18:07:05 cloudcephosd1028 smart_failure[803392]: Device: /dev/sdg [SAT], 1912 Offline uncorrectable sectors
2444Oct 07 19:37:05 cloudcephosd1028 smart_failure[837034]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2445Oct 07 19:37:05 cloudcephosd1028 smart_failure[837129]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2446Oct 07 23:07:05 cloudcephosd1028 smart_failure[915848]: Device: /dev/sdj [SAT], 2976 Offline uncorrectable sectors
2447Oct 08 02:07:05 cloudcephosd1028 smart_failure[984036]: Device: /dev/sde [SAT], 1832 Offline uncorrectable sectors
2448Oct 08 04:07:05 cloudcephosd1028 smart_failure[1029015]: Device: /dev/sdf [SAT], 5504 Offline uncorrectable sectors
2449Oct 08 06:37:05 cloudcephosd1028 smart_failure[1085490]: Device: /dev/sdi [SAT], 1024 Offline uncorrectable sectors
2450Oct 08 10:07:05 cloudcephosd1028 smart_failure[1164335]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2451Oct 08 18:07:05 cloudcephosd1028 smart_failure[1344758]: Device: /dev/sdg [SAT], 1912 Offline uncorrectable sectors
2452Oct 08 19:37:05 cloudcephosd1028 smart_failure[1378522]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2453Oct 08 19:37:05 cloudcephosd1028 smart_failure[1378565]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2454Oct 08 23:07:05 cloudcephosd1028 smart_failure[1457094]: Device: /dev/sdj [SAT], 2984 Offline uncorrectable sectors
2455Oct 09 02:07:05 cloudcephosd1028 smart_failure[1525223]: Device: /dev/sde [SAT], 1832 Offline uncorrectable sectors
2456Oct 09 04:07:05 cloudcephosd1028 smart_failure[1570438]: Device: /dev/sdf [SAT], 5520 Offline uncorrectable sectors
2457Oct 09 06:37:05 cloudcephosd1028 smart_failure[1626364]: Device: /dev/sdi [SAT], 1024 Offline uncorrectable sectors
2458Oct 09 10:07:05 cloudcephosd1028 smart_failure[1705039]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2459Oct 09 18:07:05 cloudcephosd1028 smart_failure[1885223]: Device: /dev/sdg [SAT], 1912 Offline uncorrectable sectors
2460Oct 09 19:37:05 cloudcephosd1028 smart_failure[1919210]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2461Oct 09 19:37:05 cloudcephosd1028 smart_failure[1919254]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2462Oct 09 23:07:05 cloudcephosd1028 smart_failure[1998045]: Device: /dev/sdj [SAT], 2984 Offline uncorrectable sectors
2463Oct 10 02:07:05 cloudcephosd1028 smart_failure[2066106]: Device: /dev/sde [SAT], 1840 Offline uncorrectable sectors
2464Oct 10 04:07:05 cloudcephosd1028 smart_failure[2111139]: Device: /dev/sdf [SAT], 5528 Offline uncorrectable sectors
2465Oct 10 06:37:05 cloudcephosd1028 smart_failure[2167457]: Device: /dev/sdi [SAT], 1032 Offline uncorrectable sectors
2466Oct 10 10:07:05 cloudcephosd1028 smart_failure[2246590]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2467Oct 10 18:07:05 cloudcephosd1028 smart_failure[2427627]: Device: /dev/sdg [SAT], 1912 Offline uncorrectable sectors
2468Oct 10 19:37:05 cloudcephosd1028 smart_failure[2461710]: Device: /dev/sdc [SAT], 1032 Offline uncorrectable sectors
2469Oct 10 19:37:05 cloudcephosd1028 smart_failure[2461754]: Device: /dev/sdd [SAT], 192 Offline uncorrectable sectors
2470Oct 10 23:07:05 cloudcephosd1028 smart_failure[2541227]: Device: /dev/sdj [SAT], 2992 Offline uncorrectable sectors
2471Oct 11 02:07:05 cloudcephosd1028 smart_failure[2609238]: Device: /dev/sde [SAT], 1840 Offline uncorrectable sectors
2472Oct 11 04:07:05 cloudcephosd1028 smart_failure[2654675]: Device: /dev/sdf [SAT], 5528 Offline uncorrectable sectors
2473Oct 11 06:37:05 cloudcephosd1028 smart_failure[2711199]: Device: /dev/sdi [SAT], 1040 Offline uncorrectable sectors
2474Oct 11 10:07:05 cloudcephosd1028 smart_failure[2789769]: Device: /dev/sdh [SAT], 864 Offline uncorrectable sectors
2475Sep 09 09:37:06 cloudcephosd1028 smart_failure[2256452]: Device: /dev/sdh [SAT], 816 Offline uncorrectable sectors
2476Sep 09 17:37:06 cloudcephosd1028 smart_failure[2434897]: Device: /dev/sdg [SAT], 1704 Offline uncorrectable sectors
2477Sep 09 19:07:06 cloudcephosd1028 smart_failure[2468655]: Device: /dev/sdc [SAT], 968 Offline uncorrectable sectors
2478Sep 09 19:07:06 cloudcephosd1028 smart_failure[2468674]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2479Sep 09 21:07:06 cloudcephosd1028 smart_failure[2512967]: Device: /dev/sdj [SAT], 2800 Offline uncorrectable sectors
2480Sep 10 00:37:07 cloudcephosd1028 smart_failure[2591710]: Device: /dev/sde [SAT], 1720 Offline uncorrectable sectors
2481Sep 10 03:37:06 cloudcephosd1028 smart_failure[2658745]: Device: /dev/sdf [SAT], 4936 Offline uncorrectable sectors (changed +8)
2482Sep 10 04:37:06 cloudcephosd1028 smart_failure[2681203]: Device: /dev/sdi [SAT], 888 Offline uncorrectable sectors
2483Sep 10 09:37:06 cloudcephosd1028 smart_failure[2793157]: Device: /dev/sdh [SAT], 816 Offline uncorrectable sectors
2484Sep 10 17:37:06 cloudcephosd1028 smart_failure[2972197]: Device: /dev/sdg [SAT], 1728 Offline uncorrectable sectors
2485Sep 10 19:07:06 cloudcephosd1028 smart_failure[3005477]: Device: /dev/sdc [SAT], 968 Offline uncorrectable sectors
2486Sep 10 19:07:06 cloudcephosd1028 smart_failure[3005520]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2487Sep 10 21:07:06 cloudcephosd1028 smart_failure[3050267]: Device: /dev/sdj [SAT], 2808 Offline uncorrectable sectors
2488Sep 11 01:07:06 cloudcephosd1028 smart_failure[3139591]: Device: /dev/sde [SAT], 1720 Offline uncorrectable sectors
2489Sep 11 03:37:06 cloudcephosd1028 smart_failure[3195521]: Device: /dev/sdf [SAT], 4952 Offline uncorrectable sectors
2490Sep 11 04:37:06 cloudcephosd1028 smart_failure[3217960]: Device: /dev/sdi [SAT], 888 Offline uncorrectable sectors
2491Sep 11 09:37:06 cloudcephosd1028 smart_failure[3329791]: Device: /dev/sdh [SAT], 816 Offline uncorrectable sectors
2492Sep 11 17:37:06 cloudcephosd1028 smart_failure[3509951]: Device: /dev/sdg [SAT], 1760 Offline uncorrectable sectors
2493Sep 11 19:07:06 cloudcephosd1028 smart_failure[3543722]: Device: /dev/sdc [SAT], 968 Offline uncorrectable sectors
2494Sep 11 19:07:06 cloudcephosd1028 smart_failure[3543741]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2495Sep 11 21:07:06 cloudcephosd1028 smart_failure[3588708]: Device: /dev/sdj [SAT], 2808 Offline uncorrectable sectors
2496Sep 12 01:07:07 cloudcephosd1028 smart_failure[3679073]: Device: /dev/sde [SAT], 1728 Offline uncorrectable sectors
2497Sep 12 03:37:06 cloudcephosd1028 smart_failure[3735468]: Device: /dev/sdf [SAT], 4968 Offline uncorrectable sectors
2498Sep 12 04:37:06 cloudcephosd1028 smart_failure[3758383]: Device: /dev/sdi [SAT], 888 Offline uncorrectable sectors
2499Sep 12 09:37:06 cloudcephosd1028 smart_failure[3871090]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2500Sep 12 17:37:06 cloudcephosd1028 smart_failure[4052065]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2501Sep 12 19:07:06 cloudcephosd1028 smart_failure[4086157]: Device: /dev/sdc [SAT], 984 Offline uncorrectable sectors
2502Sep 12 19:07:06 cloudcephosd1028 smart_failure[4086176]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2503Sep 12 21:07:06 cloudcephosd1028 smart_failure[4131651]: Device: /dev/sdj [SAT], 2816 Offline uncorrectable sectors
2504Sep 13 01:37:06 cloudcephosd1028 smart_failure[40447]: Device: /dev/sde [SAT], 1736 Offline uncorrectable sectors
2505Sep 13 03:37:06 cloudcephosd1028 smart_failure[86217]: Device: /dev/sdf [SAT], 4976 Offline uncorrectable sectors
2506Sep 13 04:37:06 cloudcephosd1028 smart_failure[109053]: Device: /dev/sdi [SAT], 888 Offline uncorrectable sectors
2507Sep 13 09:37:06 cloudcephosd1028 smart_failure[222427]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2508Sep 13 17:37:06 cloudcephosd1028 smart_failure[402963]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2509Sep 13 19:07:06 cloudcephosd1028 smart_failure[436922]: Device: /dev/sdc [SAT], 984 Offline uncorrectable sectors
2510Sep 13 19:07:06 cloudcephosd1028 smart_failure[436966]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2511Sep 13 21:07:06 cloudcephosd1028 smart_failure[482858]: Device: /dev/sdj [SAT], 2816 Offline uncorrectable sectors
2512Sep 14 01:37:07 cloudcephosd1028 smart_failure[584635]: Device: /dev/sde [SAT], 1736 Offline uncorrectable sectors
2513Sep 14 03:37:06 cloudcephosd1028 smart_failure[629213]: Device: /dev/sdf [SAT], 4984 Offline uncorrectable sectors
2514Sep 14 04:37:06 cloudcephosd1028 smart_failure[652066]: Device: /dev/sdi [SAT], 888 Offline uncorrectable sectors
2515Sep 14 09:37:06 cloudcephosd1028 smart_failure[765169]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2516Sep 14 17:37:07 cloudcephosd1028 smart_failure[945825]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2517Sep 14 19:07:06 cloudcephosd1028 smart_failure[979857]: Device: /dev/sdc [SAT], 984 Offline uncorrectable sectors
2518Sep 14 19:07:06 cloudcephosd1028 smart_failure[979876]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2519Sep 14 21:07:06 cloudcephosd1028 smart_failure[1025485]: Device: /dev/sdj [SAT], 2816 Offline uncorrectable sectors
2520Sep 15 01:37:06 cloudcephosd1028 smart_failure[1126160]: Device: /dev/sde [SAT], 1736 Offline uncorrectable sectors
2521Sep 15 03:37:06 cloudcephosd1028 smart_failure[1171649]: Device: /dev/sdf [SAT], 5024 Offline uncorrectable sectors
2522Sep 15 04:37:06 cloudcephosd1028 smart_failure[1194213]: Device: /dev/sdi [SAT], 888 Offline uncorrectable sectors
2523Sep 15 09:37:07 cloudcephosd1028 smart_failure[1307532]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2524Sep 15 17:37:06 cloudcephosd1028 smart_failure[1488925]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2525Sep 15 19:07:06 cloudcephosd1028 smart_failure[1523318]: Device: /dev/sdc [SAT], 984 Offline uncorrectable sectors
2526Sep 15 19:07:06 cloudcephosd1028 smart_failure[1523337]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2527Sep 15 21:07:06 cloudcephosd1028 smart_failure[1568726]: Device: /dev/sdj [SAT], 2816 Offline uncorrectable sectors
2528Sep 16 01:37:07 cloudcephosd1028 smart_failure[1671355]: Device: /dev/sde [SAT], 1736 Offline uncorrectable sectors
2529Sep 16 03:37:06 cloudcephosd1028 smart_failure[1716984]: Device: /dev/sdf [SAT], 5040 Offline uncorrectable sectors
2530Sep 16 04:37:06 cloudcephosd1028 smart_failure[1739782]: Device: /dev/sdi [SAT], 896 Offline uncorrectable sectors
2531Sep 16 09:37:06 cloudcephosd1028 smart_failure[1853787]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2532Sep 16 17:37:06 cloudcephosd1028 smart_failure[2035101]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2533Sep 16 19:07:06 cloudcephosd1028 smart_failure[2069224]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2534Sep 16 19:07:06 cloudcephosd1028 smart_failure[2069267]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2535Sep 16 21:07:06 cloudcephosd1028 smart_failure[2114865]: Device: /dev/sdj [SAT], 2816 Offline uncorrectable sectors
2536Sep 17 01:37:06 cloudcephosd1028 smart_failure[2217200]: Device: /dev/sde [SAT], 1736 Offline uncorrectable sectors
2537Sep 17 03:37:06 cloudcephosd1028 smart_failure[2262780]: Device: /dev/sdf [SAT], 5064 Offline uncorrectable sectors
2538Sep 17 04:37:06 cloudcephosd1028 smart_failure[2285566]: Device: /dev/sdi [SAT], 896 Offline uncorrectable sectors
2539Sep 17 09:37:06 cloudcephosd1028 smart_failure[2398735]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2540Sep 17 17:37:06 cloudcephosd1028 smart_failure[2580212]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2541Sep 17 19:07:06 cloudcephosd1028 smart_failure[2614336]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2542Sep 17 19:07:06 cloudcephosd1028 smart_failure[2614355]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2543Sep 17 21:07:06 cloudcephosd1028 smart_failure[2659940]: Device: /dev/sdj [SAT], 2816 Offline uncorrectable sectors
2544Sep 18 01:37:06 cloudcephosd1028 smart_failure[2762499]: Device: /dev/sde [SAT], 1752 Offline uncorrectable sectors
2545Sep 18 03:37:07 cloudcephosd1028 smart_failure[2808120]: Device: /dev/sdf [SAT], 5104 Offline uncorrectable sectors
2546Sep 18 04:37:06 cloudcephosd1028 smart_failure[2830894]: Device: /dev/sdi [SAT], 904 Offline uncorrectable sectors
2547Sep 18 09:37:06 cloudcephosd1028 smart_failure[2943846]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2548Sep 18 17:37:06 cloudcephosd1028 smart_failure[3124221]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2549Sep 18 19:07:06 cloudcephosd1028 smart_failure[3158755]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2550Sep 18 19:07:06 cloudcephosd1028 smart_failure[3158774]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2551Sep 18 21:07:06 cloudcephosd1028 smart_failure[3204463]: Device: /dev/sdj [SAT], 2824 Offline uncorrectable sectors
2552Sep 19 01:37:06 cloudcephosd1028 smart_failure[3307200]: Device: /dev/sde [SAT], 1752 Offline uncorrectable sectors
2553Sep 19 03:37:06 cloudcephosd1028 smart_failure[3353195]: Device: /dev/sdf [SAT], 5120 Offline uncorrectable sectors
2554Sep 19 04:37:06 cloudcephosd1028 smart_failure[3376076]: Device: /dev/sdi [SAT], 904 Offline uncorrectable sectors
2555Sep 19 09:37:06 cloudcephosd1028 smart_failure[3489595]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2556Sep 19 17:37:06 cloudcephosd1028 smart_failure[3671381]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2557Sep 19 19:07:06 cloudcephosd1028 smart_failure[3705647]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2558Sep 19 19:07:06 cloudcephosd1028 smart_failure[3705689]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2559Sep 19 21:07:06 cloudcephosd1028 smart_failure[3751579]: Device: /dev/sdj [SAT], 2840 Offline uncorrectable sectors
2560Sep 20 01:37:06 cloudcephosd1028 smart_failure[3854658]: Device: /dev/sde [SAT], 1752 Offline uncorrectable sectors
2561Sep 20 03:37:06 cloudcephosd1028 smart_failure[3899775]: Device: /dev/sdf [SAT], 5144 Offline uncorrectable sectors
2562Sep 20 04:37:07 cloudcephosd1028 smart_failure[3922615]: Device: /dev/sdi [SAT], 904 Offline uncorrectable sectors
2563Sep 20 09:37:06 cloudcephosd1028 smart_failure[4035795]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2564Sep 20 17:37:06 cloudcephosd1028 smart_failure[21680]: Device: /dev/sdg [SAT], 1768 Offline uncorrectable sectors
2565Sep 20 19:07:06 cloudcephosd1028 smart_failure[55562]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2566Sep 20 19:07:06 cloudcephosd1028 smart_failure[55581]: Device: /dev/sdd [SAT], 176 Offline uncorrectable sectors
2567Sep 20 21:07:07 cloudcephosd1028 smart_failure[100750]: Device: /dev/sdj [SAT], 2840 Offline uncorrectable sectors
2568Sep 21 01:37:06 cloudcephosd1028 smart_failure[202547]: Device: /dev/sde [SAT], 1752 Offline uncorrectable sectors
2569Sep 21 03:37:06 cloudcephosd1028 smart_failure[247415]: Device: /dev/sdf [SAT], 5160 Offline uncorrectable sectors
2570Sep 21 05:07:06 cloudcephosd1028 smart_failure[281256]: Device: /dev/sdi [SAT], 928 Offline uncorrectable sectors
2571Sep 21 09:37:06 cloudcephosd1028 smart_failure[382301]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2572Sep 21 17:37:06 cloudcephosd1028 smart_failure[563719]: Device: /dev/sdg [SAT], 1784 Offline uncorrectable sectors
2573Sep 21 19:07:06 cloudcephosd1028 smart_failure[597543]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2574Sep 21 19:07:06 cloudcephosd1028 smart_failure[597562]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2575Sep 21 21:37:06 cloudcephosd1028 smart_failure[653743]: Device: /dev/sdj [SAT], 2848 Offline uncorrectable sectors
2576Sep 22 01:37:06 cloudcephosd1028 smart_failure[743680]: Device: /dev/sde [SAT], 1752 Offline uncorrectable sectors
2577Sep 22 03:37:06 cloudcephosd1028 smart_failure[788657]: Device: /dev/sdf [SAT], 5176 Offline uncorrectable sectors
2578Sep 22 05:07:07 cloudcephosd1028 smart_failure[822522]: Device: /dev/sdi [SAT], 928 Offline uncorrectable sectors
2579Sep 22 09:37:06 cloudcephosd1028 smart_failure[924253]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2580Sep 22 17:37:06 cloudcephosd1028 smart_failure[1104208]: Device: /dev/sdg [SAT], 1784 Offline uncorrectable sectors
2581Sep 22 19:07:06 cloudcephosd1028 smart_failure[1137614]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2582Sep 22 19:07:06 cloudcephosd1028 smart_failure[1137658]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2583Sep 22 21:37:07 cloudcephosd1028 smart_failure[1193843]: Device: /dev/sdj [SAT], 2856 Offline uncorrectable sectors
2584Sep 23 01:37:06 cloudcephosd1028 smart_failure[1283984]: Device: /dev/sde [SAT], 1784 Offline uncorrectable sectors
2585Sep 23 03:37:06 cloudcephosd1028 smart_failure[1329127]: Device: /dev/sdf [SAT], 5208 Offline uncorrectable sectors
2586Sep 23 05:37:06 cloudcephosd1028 smart_failure[1374218]: Device: /dev/sdi [SAT], 936 Offline uncorrectable sectors
2587Sep 23 09:37:06 cloudcephosd1028 smart_failure[1463992]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2588Sep 23 17:37:06 cloudcephosd1028 smart_failure[1644294]: Device: /dev/sdg [SAT], 1784 Offline uncorrectable sectors
2589Sep 23 19:07:06 cloudcephosd1028 smart_failure[1678405]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2590Sep 23 19:07:06 cloudcephosd1028 smart_failure[1678448]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2591Sep 23 22:07:06 cloudcephosd1028 smart_failure[1745577]: Device: /dev/sdj [SAT], 2864 Offline uncorrectable sectors
2592Sep 24 01:37:06 cloudcephosd1028 smart_failure[1824681]: Device: /dev/sde [SAT], 1792 Offline uncorrectable sectors
2593Sep 24 03:37:06 cloudcephosd1028 smart_failure[1869947]: Device: /dev/sdf [SAT], 5216 Offline uncorrectable sectors
2594Sep 24 05:37:07 cloudcephosd1028 smart_failure[1914952]: Device: /dev/sdi [SAT], 952 Offline uncorrectable sectors
2595Sep 24 09:37:06 cloudcephosd1028 smart_failure[2004804]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2596Sep 24 17:37:06 cloudcephosd1028 smart_failure[2184274]: Device: /dev/sdg [SAT], 1816 Offline uncorrectable sectors
2597Sep 24 19:07:06 cloudcephosd1028 smart_failure[2217778]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2598Sep 24 19:07:06 cloudcephosd1028 smart_failure[2217821]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2599Sep 24 22:07:07 cloudcephosd1028 smart_failure[2285370]: Device: /dev/sdj [SAT], 2888 Offline uncorrectable sectors
2600Sep 25 01:37:06 cloudcephosd1028 smart_failure[2364451]: Device: /dev/sde [SAT], 1792 Offline uncorrectable sectors
2601Sep 25 03:37:06 cloudcephosd1028 smart_failure[2409261]: Device: /dev/sdf [SAT], 5248 Offline uncorrectable sectors
2602Sep 25 05:37:06 cloudcephosd1028 smart_failure[2454299]: Device: /dev/sdi [SAT], 952 Offline uncorrectable sectors
2603Sep 25 09:37:06 cloudcephosd1028 smart_failure[2544158]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2604Sep 25 17:37:06 cloudcephosd1028 smart_failure[2724105]: Device: /dev/sdg [SAT], 1824 Offline uncorrectable sectors
2605Sep 25 19:07:06 cloudcephosd1028 smart_failure[2757832]: Device: /dev/sdc [SAT], 992 Offline uncorrectable sectors
2606Sep 25 19:07:06 cloudcephosd1028 smart_failure[2757851]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2607Sep 25 22:37:06 cloudcephosd1028 smart_failure[2835751]: Device: /dev/sdj [SAT], 2896 Offline uncorrectable sectors
2608Sep 26 01:37:06 cloudcephosd1028 smart_failure[2903506]: Device: /dev/sde [SAT], 1792 Offline uncorrectable sectors
2609Sep 26 03:37:06 cloudcephosd1028 smart_failure[2948041]: Device: /dev/sdf [SAT], 5248 Offline uncorrectable sectors
2610Sep 26 05:37:06 cloudcephosd1028 smart_failure[2993234]: Device: /dev/sdi [SAT], 960 Offline uncorrectable sectors
2611Sep 26 09:37:06 cloudcephosd1028 smart_failure[3083378]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2612Sep 26 17:37:06 cloudcephosd1028 smart_failure[3263762]: Device: /dev/sdg [SAT], 1832 Offline uncorrectable sectors
2613Sep 26 19:07:06 cloudcephosd1028 smart_failure[3297088]: Device: /dev/sdc [SAT], 1008 Offline uncorrectable sectors
2614Sep 26 19:07:06 cloudcephosd1028 smart_failure[3297131]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2615Sep 26 22:37:06 cloudcephosd1028 smart_failure[3375603]: Device: /dev/sdj [SAT], 2896 Offline uncorrectable sectors
2616Sep 27 01:37:06 cloudcephosd1028 smart_failure[3442576]: Device: /dev/sde [SAT], 1792 Offline uncorrectable sectors
2617Sep 27 03:37:06 cloudcephosd1028 smart_failure[3487948]: Device: /dev/sdf [SAT], 5256 Offline uncorrectable sectors
2618Sep 27 05:37:06 cloudcephosd1028 smart_failure[3532474]: Device: /dev/sdi [SAT], 960 Offline uncorrectable sectors
2619Sep 27 09:37:06 cloudcephosd1028 smart_failure[3621933]: Device: /dev/sdh [SAT], 824 Offline uncorrectable sectors
2620Sep 27 17:37:06 cloudcephosd1028 smart_failure[3801663]: Device: /dev/sdg [SAT], 1840 Offline uncorrectable sectors
2621Sep 27 19:07:06 cloudcephosd1028 smart_failure[3834986]: Device: /dev/sdc [SAT], 1008 Offline uncorrectable sectors
2622Sep 27 19:07:06 cloudcephosd1028 smart_failure[3835030]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2623Sep 27 22:37:06 cloudcephosd1028 smart_failure[3913493]: Device: /dev/sdj [SAT], 2912 Offline uncorrectable sectors
2624Sep 28 01:37:06 cloudcephosd1028 smart_failure[3981370]: Device: /dev/sde [SAT], 1792 Offline uncorrectable sectors
2625Sep 28 03:37:06 cloudcephosd1028 smart_failure[4026090]: Device: /dev/sdf [SAT], 5280 Offline uncorrectable sectors
2626Sep 28 05:37:06 cloudcephosd1028 smart_failure[4070942]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
2627Sep 28 09:37:06 cloudcephosd1028 smart_failure[4160836]: Device: /dev/sdh [SAT], 832 Offline uncorrectable sectors
2628Sep 28 17:37:06 cloudcephosd1028 smart_failure[146284]: Device: /dev/sdg [SAT], 1848 Offline uncorrectable sectors
2629Sep 28 19:07:06 cloudcephosd1028 smart_failure[180387]: Device: /dev/sdc [SAT], 1008 Offline uncorrectable sectors
2630Sep 28 19:07:06 cloudcephosd1028 smart_failure[180406]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2631Sep 28 22:37:06 cloudcephosd1028 smart_failure[259064]: Device: /dev/sdj [SAT], 2912 Offline uncorrectable sectors
2632Sep 29 01:37:06 cloudcephosd1028 smart_failure[325838]: Device: /dev/sde [SAT], 1800 Offline uncorrectable sectors
2633Sep 29 03:37:06 cloudcephosd1028 smart_failure[370609]: Device: /dev/sdf [SAT], 5296 Offline uncorrectable sectors
2634Sep 29 05:37:06 cloudcephosd1028 smart_failure[414824]: Device: /dev/sdi [SAT], 976 Offline uncorrectable sectors
2635Sep 29 09:37:06 cloudcephosd1028 smart_failure[504304]: Device: /dev/sdh [SAT], 840 Offline uncorrectable sectors
2636Sep 29 17:37:06 cloudcephosd1028 smart_failure[682753]: Device: /dev/sdg [SAT], 1856 Offline uncorrectable sectors
2637Sep 29 19:07:06 cloudcephosd1028 smart_failure[716439]: Device: /dev/sdc [SAT], 1008 Offline uncorrectable sectors
2638Sep 29 19:07:06 cloudcephosd1028 smart_failure[716458]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2639Sep 29 22:37:06 cloudcephosd1028 smart_failure[794618]: Device: /dev/sdj [SAT], 2912 Offline uncorrectable sectors
2640Sep 30 01:37:06 cloudcephosd1028 smart_failure[861246]: Device: /dev/sde [SAT], 1808 Offline uncorrectable sectors
2641Sep 30 03:37:06 cloudcephosd1028 smart_failure[906049]: Device: /dev/sdf [SAT], 5312 Offline uncorrectable sectors
2642Sep 30 05:37:07 cloudcephosd1028 smart_failure[950561]: Device: /dev/sdi [SAT], 984 Offline uncorrectable sectors
2643Sep 30 09:37:06 cloudcephosd1028 smart_failure[1040153]: Device: /dev/sdh [SAT], 856 Offline uncorrectable sectors
2644Sep 30 17:37:06 cloudcephosd1028 smart_failure[1218456]: Device: /dev/sdg [SAT], 1856 Offline uncorrectable sectors
2645Sep 30 19:07:06 cloudcephosd1028 smart_failure[1251971]: Device: /dev/sdc [SAT], 1008 Offline uncorrectable sectors
2646Sep 30 19:07:06 cloudcephosd1028 smart_failure[1252020]: Device: /dev/sdd [SAT], 184 Offline uncorrectable sectors
2647Sep 30 22:37:06 cloudcephosd1028 smart_failure[1330341]: Device: /dev/sdj [SAT], 2912 Offline uncorrectable sectors
2648===== NODE GROUP =====
2649(1) cloudcephosd1023.eqiad.wmnet
2650----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
2651Oct 06 11:15:28 cloudcephosd1023 smart_failure[15043]: Device: /dev/sdd [SAT], 584 Offline uncorrectable sectors
2652Oct 06 11:45:28 cloudcephosd1023 smart_failure[27283]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
2653Oct 06 14:15:28 cloudcephosd1023 smart_failure[38886]: Device: /dev/sdc [SAT], 1240 Offline uncorrectable sectors
2654Oct 06 14:45:28 cloudcephosd1023 smart_failure[1950]: Device: /dev/sdi [SAT], 2344 Offline uncorrectable sectors
2655Oct 06 18:45:29 cloudcephosd1023 smart_failure[978]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2656Oct 06 20:15:28 cloudcephosd1023 smart_failure[37594]: Device: /dev/sdh [SAT], 1656 Offline uncorrectable sectors
2657Oct 06 23:45:28 cloudcephosd1023 smart_failure[24269]: Device: /dev/sdf [SAT], 1056 Offline uncorrectable sectors
2658Oct 07 03:45:28 cloudcephosd1023 smart_failure[22672]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2659Oct 07 11:15:28 cloudcephosd1023 smart_failure[7072]: Device: /dev/sdd [SAT], 584 Offline uncorrectable sectors
2660Oct 07 11:45:28 cloudcephosd1023 smart_failure[19303]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
2661Oct 07 14:15:28 cloudcephosd1023 smart_failure[30258]: Device: /dev/sdc [SAT], 1240 Offline uncorrectable sectors
2662Oct 07 14:45:28 cloudcephosd1023 smart_failure[42177]: Device: /dev/sdi [SAT], 2344 Offline uncorrectable sectors
2663Oct 07 18:45:28 cloudcephosd1023 smart_failure[40515]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2664Oct 07 20:15:28 cloudcephosd1023 smart_failure[27683]: Device: /dev/sdh [SAT], 1656 Offline uncorrectable sectors
2665Oct 07 23:45:28 cloudcephosd1023 smart_failure[13790]: Device: /dev/sdf [SAT], 1056 Offline uncorrectable sectors
2666Oct 08 03:45:28 cloudcephosd1023 smart_failure[12283]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2667Oct 08 11:15:28 cloudcephosd1023 smart_failure[45542]: Device: /dev/sdd [SAT], 584 Offline uncorrectable sectors
2668Oct 08 11:45:28 cloudcephosd1023 smart_failure[8761]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
2669Oct 08 14:15:28 cloudcephosd1023 smart_failure[19888]: Device: /dev/sdc [SAT], 1240 Offline uncorrectable sectors
2670Oct 08 14:45:28 cloudcephosd1023 smart_failure[31738]: Device: /dev/sdi [SAT], 2344 Offline uncorrectable sectors
2671Oct 08 18:45:28 cloudcephosd1023 smart_failure[30327]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2672Oct 08 20:15:28 cloudcephosd1023 smart_failure[17696]: Device: /dev/sdh [SAT], 1656 Offline uncorrectable sectors
2673Oct 08 23:45:28 cloudcephosd1023 smart_failure[3407]: Device: /dev/sdf [SAT], 1056 Offline uncorrectable sectors
2674Oct 09 03:45:28 cloudcephosd1023 smart_failure[1890]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2675Oct 09 11:15:28 cloudcephosd1023 smart_failure[34776]: Device: /dev/sdd [SAT], 584 Offline uncorrectable sectors
2676Oct 09 11:45:28 cloudcephosd1023 smart_failure[46559]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
2677Oct 09 14:15:28 cloudcephosd1023 smart_failure[9001]: Device: /dev/sdc [SAT], 1240 Offline uncorrectable sectors
2678Oct 09 14:45:28 cloudcephosd1023 smart_failure[21145]: Device: /dev/sdi [SAT], 2352 Offline uncorrectable sectors
2679Oct 09 18:45:28 cloudcephosd1023 smart_failure[19261]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2680Oct 09 20:15:28 cloudcephosd1023 smart_failure[6279]: Device: /dev/sdh [SAT], 1656 Offline uncorrectable sectors
2681Oct 09 23:45:28 cloudcephosd1023 smart_failure[41206]: Device: /dev/sdf [SAT], 1056 Offline uncorrectable sectors
2682Oct 10 03:45:28 cloudcephosd1023 smart_failure[39611]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2683Oct 10 11:15:28 cloudcephosd1023 smart_failure[24543]: Device: /dev/sdd [SAT], 584 Offline uncorrectable sectors
2684Oct 10 11:45:28 cloudcephosd1023 smart_failure[36566]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
2685Oct 10 14:15:28 cloudcephosd1023 smart_failure[47516]: Device: /dev/sdc [SAT], 1240 Offline uncorrectable sectors
2686Oct 10 14:45:28 cloudcephosd1023 smart_failure[10901]: Device: /dev/sdi [SAT], 2352 Offline uncorrectable sectors
2687Oct 10 18:45:28 cloudcephosd1023 smart_failure[10354]: Device: /dev/sde [SAT], 1032 Offline uncorrectable sectors
2688Oct 10 20:15:28 cloudcephosd1023 smart_failure[46217]: Device: /dev/sdh [SAT], 1656 Offline uncorrectable sectors
2689Oct 10 23:45:28 cloudcephosd1023 smart_failure[33159]: Device: /dev/sdf [SAT], 1056 Offline uncorrectable sectors
2690Oct 11 03:45:28 cloudcephosd1023 smart_failure[32576]: Device: /dev/sdj [SAT], 480 Offline uncorrectable sectors
2691Oct 11 11:15:28 cloudcephosd1023 smart_failure[19577]: Device: /dev/sdd [SAT], 584 Offline uncorrectable sectors
2692Oct 11 11:45:28 cloudcephosd1023 smart_failure[31484]: Device: /dev/sdg [SAT], 1584 Offline uncorrectable sectors
2693===== NODE GROUP =====
2694(1) cloudcephosd1022.eqiad.wmnet
2695----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
2696Oct 06 07:20:40 cloudcephosd1022 smart_failure[20906]: Device: /dev/sdh [SAT], 2064 Offline uncorrectable sectors
2697Oct 06 08:20:40 cloudcephosd1022 smart_failure[45298]: Device: /dev/sdd [SAT], 1568 Offline uncorrectable sectors
2698Oct 06 11:20:40 cloudcephosd1022 smart_failure[20084]: Device: /dev/sdc [SAT], 1800 Offline uncorrectable sectors
2699Oct 06 14:20:40 cloudcephosd1022 smart_failure[44033]: Device: /dev/sdj [SAT], 1296 Offline uncorrectable sectors
2700Oct 06 15:20:40 cloudcephosd1022 smart_failure[19294]: Device: /dev/sde [SAT], 824 Offline uncorrectable sectors
2701Oct 06 20:20:40 cloudcephosd1022 smart_failure[41929]: Device: /dev/sdi [SAT], 984 Offline uncorrectable sectors
2702Oct 07 01:20:40 cloudcephosd1022 smart_failure[17238]: Device: /dev/sdf [SAT], 944 Offline uncorrectable sectors
2703Oct 07 01:20:40 cloudcephosd1022 smart_failure[17258]: Device: /dev/sdg [SAT], 1296 Offline uncorrectable sectors
2704Oct 07 07:20:40 cloudcephosd1022 smart_failure[15786]: Device: /dev/sdh [SAT], 2080 Offline uncorrectable sectors
2705Oct 07 08:20:40 cloudcephosd1022 smart_failure[39640]: Device: /dev/sdd [SAT], 1568 Offline uncorrectable sectors
2706Oct 07 11:20:40 cloudcephosd1022 smart_failure[15011]: Device: /dev/sdc [SAT], 1800 Offline uncorrectable sectors
2707Oct 07 14:20:40 cloudcephosd1022 smart_failure[38439]: Device: /dev/sdj [SAT], 1296 Offline uncorrectable sectors
2708Oct 07 15:20:40 cloudcephosd1022 smart_failure[14131]: Device: /dev/sde [SAT], 824 Offline uncorrectable sectors
2709Oct 07 20:20:40 cloudcephosd1022 smart_failure[37089]: Device: /dev/sdi [SAT], 984 Offline uncorrectable sectors
2710Oct 08 01:20:40 cloudcephosd1022 smart_failure[12360]: Device: /dev/sdf [SAT], 944 Offline uncorrectable sectors
2711Oct 08 01:20:40 cloudcephosd1022 smart_failure[12380]: Device: /dev/sdg [SAT], 1296 Offline uncorrectable sectors
2712Oct 08 07:20:40 cloudcephosd1022 smart_failure[11206]: Device: /dev/sdh [SAT], 2080 Offline uncorrectable sectors
2713Oct 08 08:20:40 cloudcephosd1022 smart_failure[34995]: Device: /dev/sdd [SAT], 1584 Offline uncorrectable sectors
2714Oct 08 11:20:40 cloudcephosd1022 smart_failure[10164]: Device: /dev/sdc [SAT], 1800 Offline uncorrectable sectors
2715Oct 08 14:20:40 cloudcephosd1022 smart_failure[33281]: Device: /dev/sdj [SAT], 1296 Offline uncorrectable sectors
2716Oct 08 15:20:40 cloudcephosd1022 smart_failure[9081]: Device: /dev/sde [SAT], 824 Offline uncorrectable sectors
2717Oct 08 20:20:40 cloudcephosd1022 smart_failure[32300]: Device: /dev/sdi [SAT], 984 Offline uncorrectable sectors
2718Oct 09 01:20:40 cloudcephosd1022 smart_failure[7089]: Device: /dev/sdf [SAT], 944 Offline uncorrectable sectors
2719Oct 09 01:20:40 cloudcephosd1022 smart_failure[7109]: Device: /dev/sdg [SAT], 1296 Offline uncorrectable sectors
2720Oct 09 07:20:40 cloudcephosd1022 smart_failure[5541]: Device: /dev/sdh [SAT], 2088 Offline uncorrectable sectors
2721Oct 09 08:20:40 cloudcephosd1022 smart_failure[29335]: Device: /dev/sdd [SAT], 1616 Offline uncorrectable sectors
2722Oct 09 11:20:40 cloudcephosd1022 smart_failure[4459]: Device: /dev/sdc [SAT], 1800 Offline uncorrectable sectors
2723Oct 09 14:20:40 cloudcephosd1022 smart_failure[27851]: Device: /dev/sdj [SAT], 1296 Offline uncorrectable sectors
2724Oct 09 15:20:40 cloudcephosd1022 smart_failure[3435]: Device: /dev/sde [SAT], 824 Offline uncorrectable sectors
2725Oct 09 20:20:40 cloudcephosd1022 smart_failure[26537]: Device: /dev/sdi [SAT], 984 Offline uncorrectable sectors
2726Oct 10 01:20:40 cloudcephosd1022 smart_failure[1203]: Device: /dev/sdf [SAT], 944 Offline uncorrectable sectors
2727Oct 10 01:20:40 cloudcephosd1022 smart_failure[1224]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2728Oct 10 07:20:40 cloudcephosd1022 smart_failure[48454]: Device: /dev/sdh [SAT], 2088 Offline uncorrectable sectors
2729Oct 10 08:20:40 cloudcephosd1022 smart_failure[23893]: Device: /dev/sdd [SAT], 1616 Offline uncorrectable sectors
2730Oct 10 11:20:40 cloudcephosd1022 smart_failure[48356]: Device: /dev/sdc [SAT], 1800 Offline uncorrectable sectors
2731Oct 10 14:20:40 cloudcephosd1022 smart_failure[25299]: Device: /dev/sdj [SAT], 1296 Offline uncorrectable sectors
2732Oct 10 15:20:40 cloudcephosd1022 smart_failure[908]: Device: /dev/sde [SAT], 824 Offline uncorrectable sectors
2733Oct 10 20:20:40 cloudcephosd1022 smart_failure[25196]: Device: /dev/sdi [SAT], 984 Offline uncorrectable sectors
2734Oct 11 01:20:40 cloudcephosd1022 smart_failure[49123]: Device: /dev/sdf [SAT], 952 Offline uncorrectable sectors
2735Oct 11 01:20:40 cloudcephosd1022 smart_failure[49143]: Device: /dev/sdg [SAT], 1304 Offline uncorrectable sectors
2736Oct 11 07:20:40 cloudcephosd1022 smart_failure[48539]: Device: /dev/sdh [SAT], 2096 Offline uncorrectable sectors
2737Oct 11 08:20:40 cloudcephosd1022 smart_failure[23874]: Device: /dev/sdd [SAT], 1616 Offline uncorrectable sectors
2738Oct 11 11:20:40 cloudcephosd1022 smart_failure[47870]: Device: /dev/sdc [SAT], 1800 Offline uncorrectable sectors
2739===== NODE GROUP =====
2740(1) cloudcephosd1024.eqiad.wmnet
2741----- OUTPUT of 'journalctl | gre...}' | sort | uniq' -----
2742Oct 06 07:09:34 cloudcephosd1024 smart_failure[17946]: Device: /dev/sdf [SAT], 3208 Offline uncorrectable sectors
2743Oct 06 09:09:34 cloudcephosd1024 smart_failure[17730]: Device: /dev/sdj [SAT], 2832 Offline uncorrectable sectors
2744Oct 06 11:09:34 cloudcephosd1024 smart_failure[17393]: Device: /dev/sde [SAT], 1200 Offline uncorrectable sectors
2745Oct 06 15:09:34 cloudcephosd1024 smart_failure[16306]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
2746Oct 06 16:09:34 cloudcephosd1024 smart_failure[40200]: Device: /dev/sdh [SAT], 1360 Offline uncorrectable sectors
2747Oct 06 20:09:34 cloudcephosd1024 smart_failure[39046]: Device: /dev/sdc [SAT], 4984 Offline uncorrectable sectors
2748Oct 06 20:39:34 cloudcephosd1024 smart_failure[2328]: Device: /dev/sdd [SAT], 872 Offline uncorrectable sectors
2749Oct 07 01:39:34 cloudcephosd1024 smart_failure[25491]: Device: /dev/sdg [SAT], 1944 Offline uncorrectable sectors
2750Oct 07 07:09:34 cloudcephosd1024 smart_failure[11205]: Device: /dev/sdf [SAT], 3208 Offline uncorrectable sectors
2751Oct 07 09:09:34 cloudcephosd1024 smart_failure[10544]: Device: /dev/sdj [SAT], 2832 Offline uncorrectable sectors
2752Oct 07 11:09:34 cloudcephosd1024 smart_failure[9423]: Device: /dev/sde [SAT], 1200 Offline uncorrectable sectors
2753Oct 07 15:09:34 cloudcephosd1024 smart_failure[7900]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
2754Oct 07 16:09:34 cloudcephosd1024 smart_failure[31902]: Device: /dev/sdh [SAT], 1360 Offline uncorrectable sectors
2755Oct 07 20:09:34 cloudcephosd1024 smart_failure[30578]: Device: /dev/sdc [SAT], 4984 Offline uncorrectable sectors
2756Oct 07 20:39:34 cloudcephosd1024 smart_failure[42345]: Device: /dev/sdd [SAT], 872 Offline uncorrectable sectors
2757Oct 08 01:39:34 cloudcephosd1024 smart_failure[16905]: Device: /dev/sdg [SAT], 1952 Offline uncorrectable sectors
2758Oct 08 07:09:34 cloudcephosd1024 smart_failure[2703]: Device: /dev/sdf [SAT], 3208 Offline uncorrectable sectors
2759Oct 08 09:09:34 cloudcephosd1024 smart_failure[1633]: Device: /dev/sdj [SAT], 2832 Offline uncorrectable sectors
2760Oct 08 11:09:34 cloudcephosd1024 smart_failure[880]: Device: /dev/sde [SAT], 1200 Offline uncorrectable sectors
2761Oct 08 15:09:34 cloudcephosd1024 smart_failure[47862]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
2762Oct 08 16:09:34 cloudcephosd1024 smart_failure[23655]: Device: /dev/sdh [SAT], 1360 Offline uncorrectable sectors
2763Oct 08 20:09:34 cloudcephosd1024 smart_failure[22723]: Device: /dev/sdc [SAT], 4984 Offline uncorrectable sectors
2764Oct 08 20:39:34 cloudcephosd1024 smart_failure[34596]: Device: /dev/sdd [SAT], 872 Offline uncorrectable sectors
2765Oct 09 01:39:34 cloudcephosd1024 smart_failure[8511]: Device: /dev/sdg [SAT], 1952 Offline uncorrectable sectors
2766Oct 09 07:09:34 cloudcephosd1024 smart_failure[42712]: Device: /dev/sdf [SAT], 3208 Offline uncorrectable sectors
2767Oct 09 09:09:34 cloudcephosd1024 smart_failure[41776]: Device: /dev/sdj [SAT], 2832 Offline uncorrectable sectors
2768Oct 09 11:09:34 cloudcephosd1024 smart_failure[40801]: Device: /dev/sde [SAT], 1200 Offline uncorrectable sectors
2769Oct 09 15:09:34 cloudcephosd1024 smart_failure[39308]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
2770Oct 09 16:09:34 cloudcephosd1024 smart_failure[14664]: Device: /dev/sdh [SAT], 1360 Offline uncorrectable sectors
2771Oct 09 20:09:34 cloudcephosd1024 smart_failure[13480]: Device: /dev/sdc [SAT], 4984 Offline uncorrectable sectors
2772Oct 09 20:39:34 cloudcephosd1024 smart_failure[25520]: Device: /dev/sdd [SAT], 872 Offline uncorrectable sectors
2773Oct 10 01:39:34 cloudcephosd1024 smart_failure[47766]: Device: /dev/sdg [SAT], 1952 Offline uncorrectable sectors
2774Oct 10 07:09:34 cloudcephosd1024 smart_failure[33613]: Device: /dev/sdf [SAT], 3208 Offline uncorrectable sectors
2775Oct 10 09:09:34 cloudcephosd1024 smart_failure[33152]: Device: /dev/sdj [SAT], 2832 Offline uncorrectable sectors
2776Oct 10 11:09:34 cloudcephosd1024 smart_failure[32819]: Device: /dev/sde [SAT], 1200 Offline uncorrectable sectors
2777Oct 10 15:09:34 cloudcephosd1024 smart_failure[31852]: Device: /dev/sdi [SAT], 968 Offline uncorrectable sectors
2778Oct 10 16:09:34 cloudcephosd1024 smart_failure[7511]: Device: /dev/sdh [SAT], 1360 Offline uncorrectable sectors
2779Oct 10 20:09:34 cloudcephosd1024 smart_failure[7074]: Device: /dev/sdc [SAT], 4984 Offline uncorrectable sectors
2780Oct 10 20:39:34 cloudcephosd1024 smart_failure[19401]: Device: /dev/sdd [SAT], 872 Offline uncorrectable sectors
2781Oct 11 01:39:34 cloudcephosd1024 smart_failure[42955]: Device: /dev/sdg [SAT], 1952 Offline uncorrectable sectors
2782Oct 11 07:09:34 cloudcephosd1024 smart_failure[30474]: Device: /dev/sdf [SAT], 3208 Offline uncorrectable sectors
2783Oct 11 09:09:34 cloudcephosd1024 smart_failure[29918]: Device: /dev/sdj [SAT], 2832 Offline uncorrectable sectors
2784Oct 11 11:09:34 cloudcephosd1024 smart_failure[29987]: Device: /dev/sde [SAT], 1200 Offline uncorrectable sectors
2785================

All those machines were bought in two batches, of which they are all the machines, so it might be a bad hard drive batch:
{T291987}
{T283888}

These hosts are in service, and we can't take them all out at the same time, so we'll have to coordinate to replace/debug.

Note that one of the drives (cloudcephosd1034-sdh) was replaced in T316673: hw troubleshooting: one disk not working properly in cloudcephosd1034.eqiad.wmnet and it has not had any errors.

I'll fill up the details with the logs/debugging from the wiki in a bit
Thanks!

List of affected hosts:

cloudcephosd1021 - back online
cloudcephosd1022 - back online
cloudcephosd1023 - back online
cloudcephosd1024 - back online
cloudcephosd1025 - back online
cloudcephosd1026 - back online
cloudcephosd1027 - back online
cloudcephosd1028 - back online
cloudcephosd1029 - back online
cloudcephosd1030 - back online
cloudcephosd1031 - back online
cloudcephosd1032 - back online
cloudcephosd1033 - back online
cloudcephosd1034 - back online

Current status

  • cloudcephosd1030 will have 4 disks shipped in to replace the current ones and ship those out
    • Smartctl + performance tests of the new drives (taking advantage that they are empty, and for reference):

P58861 Masterwork From Distant Lands
1######## cloudcephosd1030: /dev/sdf
2#### Smartctl
3smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-28-amd64] (local build)
4Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
5
6=== START OF INFORMATION SECTION ===
7Device Model: SSDSC2KG019TZR
8Serial Number: PHYJ3251006P1P9CGN
9LU WWN Device Id: 5 5cd2e4 156472d94
10Add. Product Id: DELL(tm)
11Firmware Version: 7CV1DL74
12User Capacity: 1,920,383,410,176 bytes [1.92 TB]
13Sector Sizes: 512 bytes logical, 4096 bytes physical
14Rotation Rate: Solid State Device
15Form Factor: 2.5 inches
16TRIM Command: Available, deterministic, zeroed
17Device is: Not in smartctl database [for details use: -P showall]
18ATA Version is: ACS-3 T13/2161-D revision 5
19SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
20Local Time is: Thu Mar 21 11:11:53 2024 UTC
21SMART support is: Available - device has SMART capability.
22SMART support is: Enabled
23
24=== START OF READ SMART DATA SECTION ===
25SMART overall-health self-assessment test result: PASSED
26
27General SMART Values:
28Offline data collection status: (0x00) Offline data collection activity
29 was never started.
30 Auto Offline Data Collection: Disabled.
31Self-test execution status: ( 0) The previous self-test routine completed
32 without error or no self-test has ever
33 been run.
34Total time to complete Offline
35data collection: ( 0) seconds.
36Offline data collection
37capabilities: (0x79) SMART execute Offline immediate.
38 No Auto Offline data collection support.
39 Suspend Offline collection upon new
40 command.
41 Offline surface scan supported.
42 Self-test supported.
43 Conveyance Self-test supported.
44 Selective Self-test supported.
45SMART capabilities: (0x0003) Saves SMART data before entering
46 power-saving mode.
47 Supports SMART auto save timer.
48Error logging capability: (0x01) Error logging supported.
49 General Purpose Logging supported.
50Short self-test routine
51recommended polling time: ( 1) minutes.
52Extended self-test routine
53recommended polling time: ( 60) minutes.
54Conveyance self-test routine
55recommended polling time: ( 60) minutes.
56SCT capabilities: (0x003d) SCT Status supported.
57 SCT Error Recovery Control supported.
58 SCT Feature Control supported.
59 SCT Data Table supported.
60
61SMART Attributes Data Structure revision number: 1
62Vendor Specific SMART Attributes with Thresholds:
63ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
64 1 Raw_Read_Error_Rate 0x000e 130 130 039 Old_age Always - 90320
65 5 Reallocated_Sector_Ct 0x0033 076 076 001 Pre-fail Always - 0
66 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 18
67 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
68 13 Read_Soft_Error_Rate 0x001e 130 130 000 Old_age Always - 90320
69170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
70171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
71172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
72173 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 1
73174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 2
74175 Program_Fail_Count_Chip 0x0012 100 100 010 Old_age Always - 0
75179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 010 Pre-fail Always - 0
76180 Unused_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 9804
77181 Program_Fail_Cnt_Total 0x003a 100 100 000 Old_age Always - 0
78182 Erase_Fail_Count_Total 0x003a 100 100 000 Old_age Always - 0
79184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0
80194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 30
81195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
82197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
83198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
84199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
85201 Unknown_SSD_Attribute 0x0033 100 100 010 Pre-fail Always - 8663599296
86202 Unknown_SSD_Attribute 0x0027 100 100 000 Pre-fail Always - 0
87233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0
88234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
89235 Unknown_Attribute 0x000b 100 100 000 Pre-fail Always - 0
90241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 79
91242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 1
92245 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 100
93
94SMART Error Log not supported
95
96SMART Self-test log structure revision number 1
97No self-tests have been logged. [To run self-tests, use: smartctl -t]
98
99Read SMART Selective Self-test Log failed: scsi error aborted command
100
101
102
103
104#### FIO
105test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
106fio-3.25
107Starting 1 process
108
109test: (groupid=0, jobs=1): err= 0: pid=392111: Thu Mar 21 11:13:53 2024
110 read: IOPS=1078, BW=539MiB/s (565MB/s)(63.2GiB/120007msec)
111 slat (usec): min=3, max=635, avg=28.60, stdev=14.86
112 clat (usec): min=1161, max=12536, avg=7386.37, stdev=47.43
113 lat (usec): min=1390, max=12549, avg=7415.21, stdev=43.36
114 clat percentiles (usec):
115 | 1.00th=[ 7308], 5.00th=[ 7373], 10.00th=[ 7373], 20.00th=[ 7373],
116 | 30.00th=[ 7373], 40.00th=[ 7373], 50.00th=[ 7373], 60.00th=[ 7373],
117 | 70.00th=[ 7373], 80.00th=[ 7373], 90.00th=[ 7373], 95.00th=[ 7439],
118 | 99.00th=[ 7439], 99.50th=[ 7504], 99.90th=[ 7570], 99.95th=[ 7767],
119 | 99.99th=[ 8094]
120 bw ( KiB/s): min=550912, max=553984, per=100.00%, avg=552660.08, stdev=537.19, samples=239
121 iops : min= 1076, max= 1082, avg=1079.41, stdev= 1.05, samples=239
122 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
123 cpu : usr=0.55%, sys=4.20%, ctx=129439, majf=0, minf=530
124 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
125 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
126 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
127 issued rwts: total=129435,0,0,0 short=0,0,0,0 dropped=0,0,0,0
128 latency : target=0, window=0, percentile=100.00%, depth=8
129
130Run status group 0 (all jobs):
131 READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=63.2GiB (67.9GB), run=120007-120007msec
132
133Disk stats (read/write):
134 sdf: ios=258632/0, merge=0/0, ticks=1858843/0, in_queue=1858843, util=100.00%
135
136
137
138######## cloudcephosd1030: /dev/sdg
139#### Smartctl
140smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-28-amd64] (local build)
141Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
142
143=== START OF INFORMATION SECTION ===
144Device Model: SSDSC2KG019TZR
145Serial Number: PHYJ325009381P9CGN
146LU WWN Device Id: 5 5cd2e4 15646eaa6
147Add. Product Id: DELL(tm)
148Firmware Version: 7CV1DL74
149User Capacity: 1,920,383,410,176 bytes [1.92 TB]
150Sector Sizes: 512 bytes logical, 4096 bytes physical
151Rotation Rate: Solid State Device
152Form Factor: 2.5 inches
153TRIM Command: Available, deterministic, zeroed
154Device is: Not in smartctl database [for details use: -P showall]
155ATA Version is: ACS-3 T13/2161-D revision 5
156SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
157Local Time is: Thu Mar 21 11:13:53 2024 UTC
158SMART support is: Available - device has SMART capability.
159SMART support is: Enabled
160
161=== START OF READ SMART DATA SECTION ===
162SMART overall-health self-assessment test result: PASSED
163
164General SMART Values:
165Offline data collection status: (0x00) Offline data collection activity
166 was never started.
167 Auto Offline Data Collection: Disabled.
168Self-test execution status: ( 0) The previous self-test routine completed
169 without error or no self-test has ever
170 been run.
171Total time to complete Offline
172data collection: ( 0) seconds.
173Offline data collection
174capabilities: (0x79) SMART execute Offline immediate.
175 No Auto Offline data collection support.
176 Suspend Offline collection upon new
177 command.
178 Offline surface scan supported.
179 Self-test supported.
180 Conveyance Self-test supported.
181 Selective Self-test supported.
182SMART capabilities: (0x0003) Saves SMART data before entering
183 power-saving mode.
184 Supports SMART auto save timer.
185Error logging capability: (0x01) Error logging supported.
186 General Purpose Logging supported.
187Short self-test routine
188recommended polling time: ( 1) minutes.
189Extended self-test routine
190recommended polling time: ( 60) minutes.
191Conveyance self-test routine
192recommended polling time: ( 60) minutes.
193SCT capabilities: (0x003d) SCT Status supported.
194 SCT Error Recovery Control supported.
195 SCT Feature Control supported.
196 SCT Data Table supported.
197
198SMART Attributes Data Structure revision number: 1
199Vendor Specific SMART Attributes with Thresholds:
200ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
201 1 Raw_Read_Error_Rate 0x000e 130 130 039 Old_age Always - 90320
202 5 Reallocated_Sector_Ct 0x0033 076 076 001 Pre-fail Always - 0
203 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 18
204 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
205 13 Read_Soft_Error_Rate 0x001e 130 130 000 Old_age Always - 90320
206170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
207171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
208172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
209173 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 1
210174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 2
211175 Program_Fail_Count_Chip 0x0012 100 100 010 Old_age Always - 0
212179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 010 Pre-fail Always - 0
213180 Unused_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 9851
214181 Program_Fail_Cnt_Total 0x003a 100 100 000 Old_age Always - 0
215182 Erase_Fail_Count_Total 0x003a 100 100 000 Old_age Always - 0
216184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0
217194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 29
218195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
219197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
220198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
221199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
222201 Unknown_SSD_Attribute 0x0033 100 100 010 Pre-fail Always - 8663730401
223202 Unknown_SSD_Attribute 0x0027 100 100 000 Pre-fail Always - 0
224233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0
225234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
226235 Unknown_Attribute 0x000b 100 100 000 Pre-fail Always - 0
227241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 79
228242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 1
229245 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 100
230
231SMART Error Log not supported
232
233SMART Self-test log structure revision number 1
234No self-tests have been logged. [To run self-tests, use: smartctl -t]
235
236Read SMART Selective Self-test Log failed: scsi error aborted command
237
238
239
240
241#### FIO
242test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
243fio-3.25
244Starting 1 process
245
246test: (groupid=0, jobs=1): err= 0: pid=392739: Thu Mar 21 11:15:54 2024
247 read: IOPS=1079, BW=540MiB/s (566MB/s)(63.2GiB/120007msec)
248 slat (usec): min=3, max=755, avg=28.62, stdev=16.23
249 clat (usec): min=1122, max=12989, avg=7382.72, stdev=47.75
250 lat (usec): min=1312, max=13002, avg=7411.63, stdev=43.21
251 clat percentiles (usec):
252 | 1.00th=[ 7308], 5.00th=[ 7373], 10.00th=[ 7373], 20.00th=[ 7373],
253 | 30.00th=[ 7373], 40.00th=[ 7373], 50.00th=[ 7373], 60.00th=[ 7373],
254 | 70.00th=[ 7373], 80.00th=[ 7373], 90.00th=[ 7373], 95.00th=[ 7439],
255 | 99.00th=[ 7439], 99.50th=[ 7439], 99.90th=[ 7570], 99.95th=[ 7701],
256 | 99.99th=[ 8094]
257 bw ( KiB/s): min=551936, max=553984, per=100.00%, avg=552900.02, stdev=510.62, samples=239
258 iops : min= 1078, max= 1082, avg=1079.88, stdev= 1.01, samples=239
259 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
260 cpu : usr=0.60%, sys=4.19%, ctx=129501, majf=0, minf=916
261 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
262 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
263 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
264 issued rwts: total=129492,0,0,0 short=0,0,0,0 dropped=0,0,0,0
265 latency : target=0, window=0, percentile=100.00%, depth=8
266
267Run status group 0 (all jobs):
268 READ: bw=540MiB/s (566MB/s), 540MiB/s-540MiB/s (566MB/s-566MB/s), io=63.2GiB (67.9GB), run=120007-120007msec
269
270Disk stats (read/write):
271 sdg: ios=258737/0, merge=0/0, ticks=1858633/0, in_queue=1858633, util=100.00%
272
273
274
275######## cloudcephosd1030: /dev/sdh
276#### Smartctl
277smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-28-amd64] (local build)
278Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
279
280=== START OF INFORMATION SECTION ===
281Device Model: SSDSC2KG019TZR
282Serial Number: PHYJ326601721P9CGN
283LU WWN Device Id: 5 5cd2e4 156496e10
284Add. Product Id: DELL(tm)
285Firmware Version: 7CV1DL74
286User Capacity: 1,920,383,410,176 bytes [1.92 TB]
287Sector Sizes: 512 bytes logical, 4096 bytes physical
288Rotation Rate: Solid State Device
289Form Factor: 2.5 inches
290TRIM Command: Available, deterministic, zeroed
291Device is: Not in smartctl database [for details use: -P showall]
292ATA Version is: ACS-3 T13/2161-D revision 5
293SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
294Local Time is: Thu Mar 21 11:15:54 2024 UTC
295SMART support is: Available - device has SMART capability.
296SMART support is: Enabled
297
298=== START OF READ SMART DATA SECTION ===
299SMART overall-health self-assessment test result: PASSED
300
301General SMART Values:
302Offline data collection status: (0x00) Offline data collection activity
303 was never started.
304 Auto Offline Data Collection: Disabled.
305Self-test execution status: ( 0) The previous self-test routine completed
306 without error or no self-test has ever
307 been run.
308Total time to complete Offline
309data collection: ( 0) seconds.
310Offline data collection
311capabilities: (0x79) SMART execute Offline immediate.
312 No Auto Offline data collection support.
313 Suspend Offline collection upon new
314 command.
315 Offline surface scan supported.
316 Self-test supported.
317 Conveyance Self-test supported.
318 Selective Self-test supported.
319SMART capabilities: (0x0003) Saves SMART data before entering
320 power-saving mode.
321 Supports SMART auto save timer.
322Error logging capability: (0x01) Error logging supported.
323 General Purpose Logging supported.
324Short self-test routine
325recommended polling time: ( 1) minutes.
326Extended self-test routine
327recommended polling time: ( 60) minutes.
328Conveyance self-test routine
329recommended polling time: ( 60) minutes.
330SCT capabilities: (0x003d) SCT Status supported.
331 SCT Error Recovery Control supported.
332 SCT Feature Control supported.
333 SCT Data Table supported.
334
335SMART Attributes Data Structure revision number: 1
336Vendor Specific SMART Attributes with Thresholds:
337ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
338 1 Raw_Read_Error_Rate 0x000e 130 130 039 Old_age Always - 90320
339 5 Reallocated_Sector_Ct 0x0033 076 076 001 Pre-fail Always - 0
340 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 18
341 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
342 13 Read_Soft_Error_Rate 0x001e 130 130 000 Old_age Always - 90320
343170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
344171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
345172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
346173 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 1
347174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 2
348175 Program_Fail_Count_Chip 0x0012 100 100 010 Old_age Always - 0
349179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 010 Pre-fail Always - 0
350180 Unused_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 9843
351181 Program_Fail_Cnt_Total 0x003a 100 100 000 Old_age Always - 0
352182 Erase_Fail_Count_Total 0x003a 100 100 000 Old_age Always - 0
353184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0
354194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 29
355195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
356197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
357198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
358199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
359201 Unknown_SSD_Attribute 0x0033 100 100 010 Pre-fail Always - 8663861541
360202 Unknown_SSD_Attribute 0x0027 100 100 000 Pre-fail Always - 0
361233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0
362234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
363235 Unknown_Attribute 0x000b 100 100 000 Pre-fail Always - 0
364241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 79
365242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 1
366245 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 100
367
368SMART Error Log not supported
369
370SMART Self-test log structure revision number 1
371No self-tests have been logged. [To run self-tests, use: smartctl -t]
372
373Read SMART Selective Self-test Log failed: scsi error aborted command
374
375
376
377
378#### FIO
379test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
380fio-3.25
381Starting 1 process
382
383test: (groupid=0, jobs=1): err= 0: pid=393337: Thu Mar 21 11:17:54 2024
384 read: IOPS=1078, BW=539MiB/s (565MB/s)(63.2GiB/120007msec)
385 slat (usec): min=3, max=608, avg=28.57, stdev=15.12
386 clat (usec): min=1050, max=12509, avg=7387.90, stdev=47.34
387 lat (usec): min=1252, max=12545, avg=7416.73, stdev=43.14
388 clat percentiles (usec):
389 | 1.00th=[ 7308], 5.00th=[ 7373], 10.00th=[ 7373], 20.00th=[ 7373],
390 | 30.00th=[ 7373], 40.00th=[ 7373], 50.00th=[ 7373], 60.00th=[ 7373],
391 | 70.00th=[ 7373], 80.00th=[ 7373], 90.00th=[ 7439], 95.00th=[ 7439],
392 | 99.00th=[ 7439], 99.50th=[ 7439], 99.90th=[ 7635], 99.95th=[ 7701],
393 | 99.99th=[ 8094]
394 bw ( KiB/s): min=550912, max=553984, per=100.00%, avg=552522.98, stdev=565.05, samples=239
395 iops : min= 1076, max= 1082, avg=1079.15, stdev= 1.10, samples=239
396 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
397 cpu : usr=0.63%, sys=4.11%, ctx=129414, majf=0, minf=532
398 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
399 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
400 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
401 issued rwts: total=129408,0,0,0 short=0,0,0,0 dropped=0,0,0,0
402 latency : target=0, window=0, percentile=100.00%, depth=8
403
404Run status group 0 (all jobs):
405 READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=63.2GiB (67.8GB), run=120007-120007msec
406
407Disk stats (read/write):
408 sdh: ios=258562/0, merge=0/0, ticks=1858768/0, in_queue=1858768, util=99.99%
409
410
411
412######## cloudcephosd1030: /dev/sdi
413#### Smartctl
414smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-28-amd64] (local build)
415Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
416
417=== START OF INFORMATION SECTION ===
418Device Model: SSDSC2KG019TZR
419Serial Number: PHYJ3251009U1P9CGN
420LU WWN Device Id: 5 5cd2e4 156472dff
421Add. Product Id: DELL(tm)
422Firmware Version: 7CV1DL74
423User Capacity: 1,920,383,410,176 bytes [1.92 TB]
424Sector Sizes: 512 bytes logical, 4096 bytes physical
425Rotation Rate: Solid State Device
426Form Factor: 2.5 inches
427TRIM Command: Available, deterministic, zeroed
428Device is: Not in smartctl database [for details use: -P showall]
429ATA Version is: ACS-3 T13/2161-D revision 5
430SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
431Local Time is: Thu Mar 21 11:17:54 2024 UTC
432SMART support is: Available - device has SMART capability.
433SMART support is: Enabled
434
435=== START OF READ SMART DATA SECTION ===
436SMART overall-health self-assessment test result: PASSED
437
438General SMART Values:
439Offline data collection status: (0x00) Offline data collection activity
440 was never started.
441 Auto Offline Data Collection: Disabled.
442Self-test execution status: ( 0) The previous self-test routine completed
443 without error or no self-test has ever
444 been run.
445Total time to complete Offline
446data collection: ( 0) seconds.
447Offline data collection
448capabilities: (0x79) SMART execute Offline immediate.
449 No Auto Offline data collection support.
450 Suspend Offline collection upon new
451 command.
452 Offline surface scan supported.
453 Self-test supported.
454 Conveyance Self-test supported.
455 Selective Self-test supported.
456SMART capabilities: (0x0003) Saves SMART data before entering
457 power-saving mode.
458 Supports SMART auto save timer.
459Error logging capability: (0x01) Error logging supported.
460 General Purpose Logging supported.
461Short self-test routine
462recommended polling time: ( 1) minutes.
463Extended self-test routine
464recommended polling time: ( 60) minutes.
465Conveyance self-test routine
466recommended polling time: ( 60) minutes.
467SCT capabilities: (0x003d) SCT Status supported.
468 SCT Error Recovery Control supported.
469 SCT Feature Control supported.
470 SCT Data Table supported.
471
472SMART Attributes Data Structure revision number: 1
473Vendor Specific SMART Attributes with Thresholds:
474ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
475 1 Raw_Read_Error_Rate 0x000e 130 130 039 Old_age Always - 90320
476 5 Reallocated_Sector_Ct 0x0033 076 076 001 Pre-fail Always - 0
477 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 18
478 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
479 13 Read_Soft_Error_Rate 0x001e 130 130 000 Old_age Always - 90320
480170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
481171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
482172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
483173 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 1
484174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 2
485175 Program_Fail_Count_Chip 0x0012 100 100 010 Old_age Always - 0
486179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 010 Pre-fail Always - 0
487180 Unused_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 9752
488181 Program_Fail_Cnt_Total 0x003a 100 100 000 Old_age Always - 0
489182 Erase_Fail_Count_Total 0x003a 100 100 000 Old_age Always - 0
490184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0
491194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 29
492195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
493197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
494198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
495199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
496201 Unknown_SSD_Attribute 0x0033 100 100 010 Pre-fail Always - 8663992256
497202 Unknown_SSD_Attribute 0x0027 100 100 000 Pre-fail Always - 0
498233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0
499234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
500235 Unknown_Attribute 0x000b 100 100 000 Pre-fail Always - 0
501241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 79
502242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 1
503245 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 100
504
505SMART Error Log not supported
506
507SMART Self-test log structure revision number 1
508No self-tests have been logged. [To run self-tests, use: smartctl -t]
509
510Read SMART Selective Self-test Log failed: scsi error aborted command
511
512
513
514
515#### FIO
516test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
517fio-3.25
518Starting 1 process
519
520test: (groupid=0, jobs=1): err= 0: pid=394011: Thu Mar 21 11:19:54 2024
521 read: IOPS=1078, BW=539MiB/s (565MB/s)(63.2GiB/120007msec)
522 slat (usec): min=3, max=658, avg=28.52, stdev=14.28
523 clat (usec): min=1100, max=12531, avg=7388.15, stdev=46.35
524 lat (usec): min=1290, max=12545, avg=7416.95, stdev=42.46
525 clat percentiles (usec):
526 | 1.00th=[ 7308], 5.00th=[ 7373], 10.00th=[ 7373], 20.00th=[ 7373],
527 | 30.00th=[ 7373], 40.00th=[ 7373], 50.00th=[ 7373], 60.00th=[ 7373],
528 | 70.00th=[ 7373], 80.00th=[ 7373], 90.00th=[ 7439], 95.00th=[ 7439],
529 | 99.00th=[ 7439], 99.50th=[ 7504], 99.90th=[ 7570], 99.95th=[ 7701],
530 | 99.99th=[ 8094]
531 bw ( KiB/s): min=551936, max=553984, per=100.00%, avg=552522.98, stdev=524.62, samples=239
532 iops : min= 1078, max= 1082, avg=1079.15, stdev= 1.02, samples=239
533 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
534 cpu : usr=0.58%, sys=4.16%, ctx=129412, majf=0, minf=787
535 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
536 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
537 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
538 issued rwts: total=129404,0,0,0 short=0,0,0,0 dropped=0,0,0,0
539 latency : target=0, window=0, percentile=100.00%, depth=8
540
541Run status group 0 (all jobs):
542 READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=63.2GiB (67.8GB), run=120007-120007msec
543
544Disk stats (read/write):
545 sdi: ios=258566/0, merge=0/0, ticks=1858844/0, in_queue=1858844, util=100.00%
546
547
548
549######## cloudcephosd1030: /dev/sdj
550#### Smartctl
551smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-28-amd64] (local build)
552Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
553
554=== START OF INFORMATION SECTION ===
555Device Model: SSDSC2KG019TZR
556Serial Number: PHYJ325008V71P9CGN
557LU WWN Device Id: 5 5cd2e4 15646e1c8
558Add. Product Id: DELL(tm)
559Firmware Version: 7CV1DL74
560User Capacity: 1,920,383,410,176 bytes [1.92 TB]
561Sector Sizes: 512 bytes logical, 4096 bytes physical
562Rotation Rate: Solid State Device
563Form Factor: 2.5 inches
564TRIM Command: Available, deterministic, zeroed
565Device is: Not in smartctl database [for details use: -P showall]
566ATA Version is: ACS-3 T13/2161-D revision 5
567SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
568Local Time is: Thu Mar 21 11:19:54 2024 UTC
569SMART support is: Available - device has SMART capability.
570SMART support is: Enabled
571
572=== START OF READ SMART DATA SECTION ===
573SMART overall-health self-assessment test result: PASSED
574
575General SMART Values:
576Offline data collection status: (0x00) Offline data collection activity
577 was never started.
578 Auto Offline Data Collection: Disabled.
579Self-test execution status: ( 0) The previous self-test routine completed
580 without error or no self-test has ever
581 been run.
582Total time to complete Offline
583data collection: ( 0) seconds.
584Offline data collection
585capabilities: (0x79) SMART execute Offline immediate.
586 No Auto Offline data collection support.
587 Suspend Offline collection upon new
588 command.
589 Offline surface scan supported.
590 Self-test supported.
591 Conveyance Self-test supported.
592 Selective Self-test supported.
593SMART capabilities: (0x0003) Saves SMART data before entering
594 power-saving mode.
595 Supports SMART auto save timer.
596Error logging capability: (0x01) Error logging supported.
597 General Purpose Logging supported.
598Short self-test routine
599recommended polling time: ( 1) minutes.
600Extended self-test routine
601recommended polling time: ( 60) minutes.
602Conveyance self-test routine
603recommended polling time: ( 60) minutes.
604SCT capabilities: (0x003d) SCT Status supported.
605 SCT Error Recovery Control supported.
606 SCT Feature Control supported.
607 SCT Data Table supported.
608
609SMART Attributes Data Structure revision number: 1
610Vendor Specific SMART Attributes with Thresholds:
611ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
612 1 Raw_Read_Error_Rate 0x000e 130 130 039 Old_age Always - 132673744
613 5 Reallocated_Sector_Ct 0x0033 076 076 001 Pre-fail Always - 0
614 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 18
615 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
616 13 Read_Soft_Error_Rate 0x001e 090 090 000 Old_age Always - 132673744
617170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
618171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
619172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
620173 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 1
621174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 2
622175 Program_Fail_Count_Chip 0x0012 100 100 010 Old_age Always - 0
623179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 010 Pre-fail Always - 0
624180 Unused_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 9874
625181 Program_Fail_Cnt_Total 0x003a 100 100 000 Old_age Always - 0
626182 Erase_Fail_Count_Total 0x003a 100 100 000 Old_age Always - 0
627184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0
628194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 29
629195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
630197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
631198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
632199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
633201 Unknown_SSD_Attribute 0x0033 100 100 010 Pre-fail Always - 8664123632
634202 Unknown_SSD_Attribute 0x0027 100 100 000 Pre-fail Always - 0
635233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0
636234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
637235 Unknown_Attribute 0x000b 100 100 000 Pre-fail Always - 0
638241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 79
639242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 2024
640245 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 100
641
642SMART Error Log not supported
643
644SMART Self-test log structure revision number 1
645No self-tests have been logged. [To run self-tests, use: smartctl -t]
646
647Read SMART Selective Self-test Log failed: scsi error aborted command
648
649
650
651
652#### FIO
653test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
654fio-3.25
655Starting 1 process
656
657test: (groupid=0, jobs=1): err= 0: pid=394610: Thu Mar 21 11:21:55 2024
658 read: IOPS=1078, BW=539MiB/s (566MB/s)(63.2GiB/120007msec)
659 slat (usec): min=3, max=783, avg=28.81, stdev=15.21
660 clat (usec): min=1512, max=12423, avg=7383.61, stdev=45.96
661 lat (usec): min=2296, max=12443, avg=7412.72, stdev=41.73
662 clat percentiles (usec):
663 | 1.00th=[ 7308], 5.00th=[ 7373], 10.00th=[ 7373], 20.00th=[ 7373],
664 | 30.00th=[ 7373], 40.00th=[ 7373], 50.00th=[ 7373], 60.00th=[ 7373],
665 | 70.00th=[ 7373], 80.00th=[ 7373], 90.00th=[ 7373], 95.00th=[ 7439],
666 | 99.00th=[ 7439], 99.50th=[ 7504], 99.90th=[ 7570], 99.95th=[ 7832],
667 | 99.99th=[ 8160]
668 bw ( KiB/s): min=550912, max=553984, per=100.00%, avg=552810.04, stdev=496.16, samples=239
669 iops : min= 1076, max= 1082, avg=1079.71, stdev= 0.97, samples=239
670 lat (msec) : 2=0.01%, 4=0.01%, 10=100.00%, 20=0.01%
671 cpu : usr=0.64%, sys=4.15%, ctx=129484, majf=0, minf=804
672 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
673 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
674 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
675 issued rwts: total=129475,0,0,0 short=0,0,0,0 dropped=0,0,0,0
676 latency : target=0, window=0, percentile=100.00%, depth=8
677
678Run status group 0 (all jobs):
679 READ: bw=539MiB/s (566MB/s), 539MiB/s-539MiB/s (566MB/s-566MB/s), io=63.2GiB (67.9GB), run=120007-120007msec
680
681Disk stats (read/write):
682 sdj: ios=258696/0, merge=0/0, ticks=1858626/0, in_queue=1858626, util=99.99%
683
684
685

  • cloudcephosd1034 will be used to run some performance tests on the original drives and the replaced drive
sudo fio \
    --ioengine=libaio \
    --direct=1 \
    --numjobs=1 \
    --time_based=1 \
    --runtime=120 \
    --rw=randread \
    --bs=512k \
    --iodepth=8 \
    --name=test \
    --group_reporting=1 \
    --filename=/dev/sdb

Tests results:

P58897 Masterwork From Distant Lands
1######## cloudcephosd1034: /dev/sdc
2#### Smartctl
3smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
4Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
5
6=== START OF INFORMATION SECTION ===
7Device Model: HFS1T9G32FEH-BA10A
8Serial Number: KJA8N5701I0808I68
9LU WWN Device Id: 5 ace42e 0253539ac
10Add. Product Id: DELL(tm)
11Firmware Version: DD02
12User Capacity: 1,920,383,410,176 bytes [1.92 TB]
13Sector Sizes: 512 bytes logical, 4096 bytes physical
14Rotation Rate: Solid State Device
15Form Factor: 2.5 inches
16TRIM Command: Available, deterministic, zeroed
17Device is: Not in smartctl database [for details use: -P showall]
18ATA Version is: ACS-4 (minor revision not indicated)
19SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
20Local Time is: Fri Mar 22 09:51:48 2024 UTC
21SMART support is: Available - device has SMART capability.
22SMART support is: Enabled
23
24=== START OF READ SMART DATA SECTION ===
25SMART overall-health self-assessment test result: PASSED
26
27General SMART Values:
28Offline data collection status: (0x02) Offline data collection activity
29 was completed without error.
30 Auto Offline Data Collection: Disabled.
31Self-test execution status: ( 0) The previous self-test routine completed
32 without error or no self-test has ever
33 been run.
34Total time to complete Offline
35data collection: ( 1730) seconds.
36Offline data collection
37capabilities: (0x19) SMART execute Offline immediate.
38 No Auto Offline data collection support.
39 Suspend Offline collection upon new
40 command.
41 Offline surface scan supported.
42 Self-test supported.
43 No Conveyance Self-test supported.
44 No Selective Self-test supported.
45SMART capabilities: (0x0002) Does not save SMART data before
46 entering power-saving mode.
47 Supports SMART auto save timer.
48Error logging capability: (0x01) Error logging supported.
49 General Purpose Logging supported.
50Short self-test routine
51recommended polling time: ( 2) minutes.
52Extended self-test routine
53recommended polling time: ( 30) minutes.
54SCT capabilities: (0x003d) SCT Status supported.
55 SCT Error Recovery Control supported.
56 SCT Feature Control supported.
57 SCT Data Table supported.
58
59SMART Attributes Data Structure revision number: 0
60Vendor Specific SMART Attributes with Thresholds:
61ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
62 1 Raw_Read_Error_Rate 0x000e 100 100 006 Old_age Always - 0
63 5 Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
64 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15173
65 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 40
66 13 Read_Soft_Error_Rate 0x002e 100 100 000 Old_age Always - 0
67173 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 271
68175 Program_Fail_Count_Chip 0x0032 099 099 000 Old_age Always - 255
69179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 002 Pre-fail Always - 0
70180 Unused_Rsvd_Blk_Cnt_Tot 0x0010 100 100 000 Old_age Offline - 4706
71181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always - 0
72182 Erase_Fail_Count_Total 0x0032 100 100 000 Old_age Always - 0
73194 Temperature_Celsius 0x0002 066 054 000 Old_age Always - 34 (Min/Max 19/46)
74195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
75198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 3248
76199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
77201 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
78202 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
79233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 224254
80235 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 224254
81241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 556700
82245 Unknown_Attribute 0x0033 099 099 001 Pre-fail Always - 99
83
84SMART Error Log not supported
85
86SMART Self-test Log not supported
87
88Selective Self-tests/Logging not supported
89
90
91
92
93#### FIO
94test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
95fio-3.25
96Starting 1 process
97
98test: (groupid=0, jobs=1): err= 0: pid=271391: Fri Mar 22 09:53:49 2024
99 read: IOPS=1034, BW=517MiB/s (542MB/s)(60.6GiB/120008msec)
100 slat (usec): min=4, max=696, avg=29.55, stdev=13.78
101 clat (usec): min=1716, max=13210, avg=7703.77, stdev=240.79
102 lat (usec): min=2172, max=13236, avg=7733.59, stdev=240.01
103 clat percentiles (usec):
104 | 1.00th=[ 7177], 5.00th=[ 7242], 10.00th=[ 7439], 20.00th=[ 7635],
105 | 30.00th=[ 7635], 40.00th=[ 7635], 50.00th=[ 7701], 60.00th=[ 7701],
106 | 70.00th=[ 7767], 80.00th=[ 7767], 90.00th=[ 7963], 95.00th=[ 8160],
107 | 99.00th=[ 8455], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9372],
108 | 99.99th=[10421]
109 bw ( KiB/s): min=527360, max=532480, per=100.00%, avg=529896.44, stdev=1052.90, samples=239
110 iops : min= 1030, max= 1040, avg=1034.95, stdev= 2.06, samples=239
111 lat (msec) : 2=0.01%, 4=0.01%, 10=99.98%, 20=0.01%
112 cpu : usr=0.52%, sys=4.14%, ctx=124113, majf=0, minf=533
113 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
114 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
115 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
116 issued rwts: total=124105,0,0,0 short=0,0,0,0 dropped=0,0,0,0
117 latency : target=0, window=0, percentile=100.00%, depth=8
118
119Run status group 0 (all jobs):
120 READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=60.6GiB (65.1GB), run=120008-120008msec
121
122Disk stats (read/write):
123 sdc: ios=247966/0, merge=0/0, ticks=1842641/0, in_queue=1842641, util=100.00%
124
125
126
127######## cloudcephosd1034: /dev/sdd
128#### Smartctl
129smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
130Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
131
132=== START OF INFORMATION SECTION ===
133Device Model: HFS1T9G32FEH-BA10A
134Serial Number: KJA8N5701I0308I2R
135LU WWN Device Id: 5 ace42e 02535419a
136Add. Product Id: DELL(tm)
137Firmware Version: DD02
138User Capacity: 1,920,383,410,176 bytes [1.92 TB]
139Sector Sizes: 512 bytes logical, 4096 bytes physical
140Rotation Rate: Solid State Device
141Form Factor: 2.5 inches
142TRIM Command: Available, deterministic, zeroed
143Device is: Not in smartctl database [for details use: -P showall]
144ATA Version is: ACS-4 (minor revision not indicated)
145SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
146Local Time is: Fri Mar 22 09:53:49 2024 UTC
147SMART support is: Available - device has SMART capability.
148SMART support is: Enabled
149
150=== START OF READ SMART DATA SECTION ===
151SMART overall-health self-assessment test result: PASSED
152
153General SMART Values:
154Offline data collection status: (0x02) Offline data collection activity
155 was completed without error.
156 Auto Offline Data Collection: Disabled.
157Self-test execution status: ( 0) The previous self-test routine completed
158 without error or no self-test has ever
159 been run.
160Total time to complete Offline
161data collection: ( 1730) seconds.
162Offline data collection
163capabilities: (0x19) SMART execute Offline immediate.
164 No Auto Offline data collection support.
165 Suspend Offline collection upon new
166 command.
167 Offline surface scan supported.
168 Self-test supported.
169 No Conveyance Self-test supported.
170 No Selective Self-test supported.
171SMART capabilities: (0x0002) Does not save SMART data before
172 entering power-saving mode.
173 Supports SMART auto save timer.
174Error logging capability: (0x01) Error logging supported.
175 General Purpose Logging supported.
176Short self-test routine
177recommended polling time: ( 2) minutes.
178Extended self-test routine
179recommended polling time: ( 30) minutes.
180SCT capabilities: (0x003d) SCT Status supported.
181 SCT Error Recovery Control supported.
182 SCT Feature Control supported.
183 SCT Data Table supported.
184
185SMART Attributes Data Structure revision number: 0
186Vendor Specific SMART Attributes with Thresholds:
187ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
188 1 Raw_Read_Error_Rate 0x000e 100 100 006 Old_age Always - 0
189 5 Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
190 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15173
191 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 40
192 13 Read_Soft_Error_Rate 0x002e 100 100 000 Old_age Always - 0
193173 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 261
194175 Program_Fail_Count_Chip 0x0032 099 099 000 Old_age Always - 246
195179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 002 Pre-fail Always - 0
196180 Unused_Rsvd_Blk_Cnt_Tot 0x0010 100 100 000 Old_age Offline - 4635
197181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always - 0
198182 Erase_Fail_Count_Total 0x0032 100 100 000 Old_age Always - 0
199194 Temperature_Celsius 0x0002 064 052 000 Old_age Always - 36 (Min/Max 19/48)
200195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
201198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 512
202199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
203201 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
204202 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
205233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 216692
206235 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 216692
207241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 536468
208245 Unknown_Attribute 0x0033 099 099 001 Pre-fail Always - 99
209
210SMART Error Log not supported
211
212SMART Self-test Log not supported
213
214Selective Self-tests/Logging not supported
215
216
217
218
219#### FIO
220test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
221fio-3.25
222Starting 1 process
223
224test: (groupid=0, jobs=1): err= 0: pid=272032: Fri Mar 22 09:55:49 2024
225 read: IOPS=1033, BW=517MiB/s (542MB/s)(60.6GiB/120008msec)
226 slat (usec): min=3, max=650, avg=29.01, stdev=14.21
227 clat (usec): min=1366, max=13196, avg=7707.29, stdev=241.22
228 lat (usec): min=1594, max=13230, avg=7736.56, stdev=240.43
229 clat percentiles (usec):
230 | 1.00th=[ 7177], 5.00th=[ 7242], 10.00th=[ 7439], 20.00th=[ 7635],
231 | 30.00th=[ 7635], 40.00th=[ 7635], 50.00th=[ 7701], 60.00th=[ 7701],
232 | 70.00th=[ 7767], 80.00th=[ 7767], 90.00th=[ 7963], 95.00th=[ 8160],
233 | 99.00th=[ 8455], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9372],
234 | 99.99th=[10028]
235 bw ( KiB/s): min=526336, max=531456, per=100.00%, avg=529682.21, stdev=1025.85, samples=239
236 iops : min= 1028, max= 1038, avg=1034.54, stdev= 2.00, samples=239
237 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
238 cpu : usr=0.56%, sys=4.03%, ctx=124065, majf=0, minf=660
239 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
240 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
241 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
242 issued rwts: total=124056,0,0,0 short=0,0,0,0 dropped=0,0,0,0
243 latency : target=0, window=0, percentile=100.00%, depth=8
244
245Run status group 0 (all jobs):
246 READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=60.6GiB (65.0GB), run=120008-120008msec
247
248Disk stats (read/write):
249 sdd: ios=247883/0, merge=0/0, ticks=1843018/0, in_queue=1843018, util=100.00%
250
251
252
253######## cloudcephosd1034: /dev/sde
254#### Smartctl
255smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
256Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
257
258=== START OF INFORMATION SECTION ===
259Device Model: HFS1T9G32FEH-BA10A
260Serial Number: KJA8N5701I0308I4J
261LU WWN Device Id: 5 ace42e 0253541da
262Add. Product Id: DELL(tm)
263Firmware Version: DD02
264User Capacity: 1,920,383,410,176 bytes [1.92 TB]
265Sector Sizes: 512 bytes logical, 4096 bytes physical
266Rotation Rate: Solid State Device
267Form Factor: 2.5 inches
268TRIM Command: Available, deterministic, zeroed
269Device is: Not in smartctl database [for details use: -P showall]
270ATA Version is: ACS-4 (minor revision not indicated)
271SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
272Local Time is: Fri Mar 22 09:55:49 2024 UTC
273SMART support is: Available - device has SMART capability.
274SMART support is: Enabled
275
276=== START OF READ SMART DATA SECTION ===
277SMART overall-health self-assessment test result: PASSED
278
279General SMART Values:
280Offline data collection status: (0x02) Offline data collection activity
281 was completed without error.
282 Auto Offline Data Collection: Disabled.
283Self-test execution status: ( 0) The previous self-test routine completed
284 without error or no self-test has ever
285 been run.
286Total time to complete Offline
287data collection: ( 1730) seconds.
288Offline data collection
289capabilities: (0x19) SMART execute Offline immediate.
290 No Auto Offline data collection support.
291 Suspend Offline collection upon new
292 command.
293 Offline surface scan supported.
294 Self-test supported.
295 No Conveyance Self-test supported.
296 No Selective Self-test supported.
297SMART capabilities: (0x0002) Does not save SMART data before
298 entering power-saving mode.
299 Supports SMART auto save timer.
300Error logging capability: (0x01) Error logging supported.
301 General Purpose Logging supported.
302Short self-test routine
303recommended polling time: ( 2) minutes.
304Extended self-test routine
305recommended polling time: ( 30) minutes.
306SCT capabilities: (0x003d) SCT Status supported.
307 SCT Error Recovery Control supported.
308 SCT Feature Control supported.
309 SCT Data Table supported.
310
311SMART Attributes Data Structure revision number: 0
312Vendor Specific SMART Attributes with Thresholds:
313ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
314 1 Raw_Read_Error_Rate 0x000e 100 100 006 Old_age Always - 0
315 5 Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
316 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15177
317 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 41
318 13 Read_Soft_Error_Rate 0x002e 100 100 000 Old_age Always - 0
319173 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 247
320175 Program_Fail_Count_Chip 0x0032 099 099 000 Old_age Always - 233
321179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 002 Pre-fail Always - 0
322180 Unused_Rsvd_Blk_Cnt_Tot 0x0010 100 100 000 Old_age Offline - 4673
323181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always - 0
324182 Erase_Fail_Count_Total 0x0032 100 100 000 Old_age Always - 0
325194 Temperature_Celsius 0x0002 066 054 000 Old_age Always - 34 (Min/Max 19/46)
326195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
327198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 1208
328199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
329201 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
330202 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
331233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 200155
332235 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 200155
333241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 508683
334245 Unknown_Attribute 0x0033 099 099 001 Pre-fail Always - 99
335
336SMART Error Log not supported
337
338SMART Self-test Log not supported
339
340Selective Self-tests/Logging not supported
341
342
343
344
345#### FIO
346test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
347fio-3.25
348Starting 1 process
349
350test: (groupid=0, jobs=1): err= 0: pid=272645: Fri Mar 22 09:57:49 2024
351 read: IOPS=1034, BW=517MiB/s (542MB/s)(60.6GiB/120008msec)
352 slat (usec): min=4, max=822, avg=29.43, stdev=15.28
353 clat (usec): min=1237, max=13701, avg=7699.74, stdev=239.26
354 lat (usec): min=2061, max=13743, avg=7729.41, stdev=238.38
355 clat percentiles (usec):
356 | 1.00th=[ 7177], 5.00th=[ 7242], 10.00th=[ 7439], 20.00th=[ 7635],
357 | 30.00th=[ 7635], 40.00th=[ 7635], 50.00th=[ 7701], 60.00th=[ 7701],
358 | 70.00th=[ 7701], 80.00th=[ 7767], 90.00th=[ 7963], 95.00th=[ 8160],
359 | 99.00th=[ 8455], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9372],
360 | 99.99th=[ 9896]
361 bw ( KiB/s): min=527360, max=532480, per=100.00%, avg=530192.07, stdev=1008.56, samples=239
362 iops : min= 1030, max= 1040, avg=1035.53, stdev= 1.97, samples=239
363 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
364 cpu : usr=0.57%, sys=4.07%, ctx=124186, majf=0, minf=527
365 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
366 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
367 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
368 issued rwts: total=124174,0,0,0 short=0,0,0,0 dropped=0,0,0,0
369 latency : target=0, window=0, percentile=100.00%, depth=8
370
371Run status group 0 (all jobs):
372 READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=60.6GiB (65.1GB), run=120008-120008msec
373
374Disk stats (read/write):
375 sde: ios=248111/0, merge=0/0, ticks=1842571/0, in_queue=1842571, util=100.00%
376
377
378
379######## cloudcephosd1034: /dev/sdf
380#### Smartctl
381smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
382Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
383
384=== START OF INFORMATION SECTION ===
385Device Model: HFS1T9G32FEH-BA10A
386Serial Number: KJA8N5701I0808I62
387LU WWN Device Id: 5 ace42e 0253539a6
388Add. Product Id: DELL(tm)
389Firmware Version: DD02
390User Capacity: 1,920,383,410,176 bytes [1.92 TB]
391Sector Sizes: 512 bytes logical, 4096 bytes physical
392Rotation Rate: Solid State Device
393Form Factor: 2.5 inches
394TRIM Command: Available, deterministic, zeroed
395Device is: Not in smartctl database [for details use: -P showall]
396ATA Version is: ACS-4 (minor revision not indicated)
397SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
398Local Time is: Fri Mar 22 09:57:50 2024 UTC
399SMART support is: Available - device has SMART capability.
400SMART support is: Enabled
401
402=== START OF READ SMART DATA SECTION ===
403SMART overall-health self-assessment test result: PASSED
404
405General SMART Values:
406Offline data collection status: (0x02) Offline data collection activity
407 was completed without error.
408 Auto Offline Data Collection: Disabled.
409Self-test execution status: ( 0) The previous self-test routine completed
410 without error or no self-test has ever
411 been run.
412Total time to complete Offline
413data collection: ( 1730) seconds.
414Offline data collection
415capabilities: (0x19) SMART execute Offline immediate.
416 No Auto Offline data collection support.
417 Suspend Offline collection upon new
418 command.
419 Offline surface scan supported.
420 Self-test supported.
421 No Conveyance Self-test supported.
422 No Selective Self-test supported.
423SMART capabilities: (0x0002) Does not save SMART data before
424 entering power-saving mode.
425 Supports SMART auto save timer.
426Error logging capability: (0x01) Error logging supported.
427 General Purpose Logging supported.
428Short self-test routine
429recommended polling time: ( 2) minutes.
430Extended self-test routine
431recommended polling time: ( 30) minutes.
432SCT capabilities: (0x003d) SCT Status supported.
433 SCT Error Recovery Control supported.
434 SCT Feature Control supported.
435 SCT Data Table supported.
436
437SMART Attributes Data Structure revision number: 0
438Vendor Specific SMART Attributes with Thresholds:
439ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
440 1 Raw_Read_Error_Rate 0x000e 100 100 006 Old_age Always - 0
441 5 Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
442 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15173
443 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 40
444 13 Read_Soft_Error_Rate 0x002e 100 100 000 Old_age Always - 0
445173 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 243
446175 Program_Fail_Count_Chip 0x0032 099 099 000 Old_age Always - 229
447179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 002 Pre-fail Always - 0
448180 Unused_Rsvd_Blk_Cnt_Tot 0x0010 100 100 000 Old_age Offline - 4539
449181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always - 0
450182 Erase_Fail_Count_Total 0x0032 100 100 000 Old_age Always - 0
451194 Temperature_Celsius 0x0002 065 053 000 Old_age Always - 35 (Min/Max 19/47)
452195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
453198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 840
454199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
455201 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
456202 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
457233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 198085
458235 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 198085
459241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 500924
460245 Unknown_Attribute 0x0033 099 099 001 Pre-fail Always - 99
461
462SMART Error Log not supported
463
464SMART Self-test Log not supported
465
466Selective Self-tests/Logging not supported
467
468
469
470
471#### FIO
472test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
473fio-3.25
474Starting 1 process
475
476test: (groupid=0, jobs=1): err= 0: pid=273265: Fri Mar 22 09:59:50 2024
477 read: IOPS=1032, BW=516MiB/s (541MB/s)(60.5GiB/120008msec)
478 slat (usec): min=4, max=681, avg=27.54, stdev=14.56
479 clat (usec): min=1162, max=13247, avg=7717.36, stdev=245.87
480 lat (usec): min=1844, max=13289, avg=7745.17, stdev=245.05
481 clat percentiles (usec):
482 | 1.00th=[ 7177], 5.00th=[ 7242], 10.00th=[ 7439], 20.00th=[ 7635],
483 | 30.00th=[ 7635], 40.00th=[ 7701], 50.00th=[ 7701], 60.00th=[ 7701],
484 | 70.00th=[ 7767], 80.00th=[ 7832], 90.00th=[ 7963], 95.00th=[ 8160],
485 | 99.00th=[ 8455], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9372],
486 | 99.99th=[10028]
487 bw ( KiB/s): min=526336, max=531456, per=100.00%, avg=529099.51, stdev=980.73, samples=239
488 iops : min= 1028, max= 1038, avg=1033.40, stdev= 1.92, samples=239
489 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
490 cpu : usr=0.48%, sys=3.83%, ctx=123931, majf=0, minf=531
491 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
492 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
493 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
494 issued rwts: total=123926,0,0,0 short=0,0,0,0 dropped=0,0,0,0
495 latency : target=0, window=0, percentile=100.00%, depth=8
496
497Run status group 0 (all jobs):
498 READ: bw=516MiB/s (541MB/s), 516MiB/s-516MiB/s (541MB/s-541MB/s), io=60.5GiB (64.0GB), run=120008-120008msec
499
500Disk stats (read/write):
501 sdf: ios=247701/0, merge=0/0, ticks=1842961/0, in_queue=1842961, util=99.99%
502
503
504
505######## cloudcephosd1034: /dev/sdg
506#### Smartctl
507smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
508Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
509
510=== START OF INFORMATION SECTION ===
511Device Model: HFS1T9G32FEH-BA10A
512Serial Number: KJA8N5701I0808I4Q
513LU WWN Device Id: 5 ace42e 025353976
514Add. Product Id: DELL(tm)
515Firmware Version: DD02
516User Capacity: 1,920,383,410,176 bytes [1.92 TB]
517Sector Sizes: 512 bytes logical, 4096 bytes physical
518Rotation Rate: Solid State Device
519Form Factor: 2.5 inches
520TRIM Command: Available, deterministic, zeroed
521Device is: Not in smartctl database [for details use: -P showall]
522ATA Version is: ACS-4 (minor revision not indicated)
523SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
524Local Time is: Fri Mar 22 09:59:50 2024 UTC
525SMART support is: Available - device has SMART capability.
526SMART support is: Enabled
527
528=== START OF READ SMART DATA SECTION ===
529SMART overall-health self-assessment test result: PASSED
530
531General SMART Values:
532Offline data collection status: (0x02) Offline data collection activity
533 was completed without error.
534 Auto Offline Data Collection: Disabled.
535Self-test execution status: ( 0) The previous self-test routine completed
536 without error or no self-test has ever
537 been run.
538Total time to complete Offline
539data collection: ( 1730) seconds.
540Offline data collection
541capabilities: (0x19) SMART execute Offline immediate.
542 No Auto Offline data collection support.
543 Suspend Offline collection upon new
544 command.
545 Offline surface scan supported.
546 Self-test supported.
547 No Conveyance Self-test supported.
548 No Selective Self-test supported.
549SMART capabilities: (0x0002) Does not save SMART data before
550 entering power-saving mode.
551 Supports SMART auto save timer.
552Error logging capability: (0x01) Error logging supported.
553 General Purpose Logging supported.
554Short self-test routine
555recommended polling time: ( 2) minutes.
556Extended self-test routine
557recommended polling time: ( 30) minutes.
558SCT capabilities: (0x003d) SCT Status supported.
559 SCT Error Recovery Control supported.
560 SCT Feature Control supported.
561 SCT Data Table supported.
562
563SMART Attributes Data Structure revision number: 0
564Vendor Specific SMART Attributes with Thresholds:
565ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
566 1 Raw_Read_Error_Rate 0x000e 100 100 006 Old_age Always - 0
567 5 Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
568 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15173
569 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 40
570 13 Read_Soft_Error_Rate 0x002e 100 100 000 Old_age Always - 0
571173 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 252
572175 Program_Fail_Count_Chip 0x0032 099 099 000 Old_age Always - 238
573179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 002 Pre-fail Always - 0
574180 Unused_Rsvd_Blk_Cnt_Tot 0x0010 100 100 000 Old_age Offline - 4654
575181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always - 0
576182 Erase_Fail_Count_Total 0x0032 100 100 000 Old_age Always - 0
577194 Temperature_Celsius 0x0002 066 055 000 Old_age Always - 34 (Min/Max 19/45)
578195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
579198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 1408
580199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
581201 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
582202 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
583233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 210569
584235 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 210569
585241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 520374
586245 Unknown_Attribute 0x0033 099 099 001 Pre-fail Always - 99
587
588SMART Error Log not supported
589
590SMART Self-test Log not supported
591
592Selective Self-tests/Logging not supported
593
594
595
596
597#### FIO
598test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
599fio-3.25
600Starting 1 process
601
602test: (groupid=0, jobs=1): err= 0: pid=275925: Fri Mar 22 10:01:50 2024
603 read: IOPS=1034, BW=517MiB/s (542MB/s)(60.6GiB/120008msec)
604 slat (usec): min=4, max=943, avg=29.62, stdev=15.41
605 clat (usec): min=1230, max=13592, avg=7699.86, stdev=243.00
606 lat (usec): min=1982, max=13634, avg=7729.75, stdev=242.17
607 clat percentiles (usec):
608 | 1.00th=[ 7177], 5.00th=[ 7242], 10.00th=[ 7439], 20.00th=[ 7635],
609 | 30.00th=[ 7635], 40.00th=[ 7635], 50.00th=[ 7701], 60.00th=[ 7701],
610 | 70.00th=[ 7701], 80.00th=[ 7767], 90.00th=[ 7963], 95.00th=[ 8160],
611 | 99.00th=[ 8455], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9372],
612 | 99.99th=[10028]
613 bw ( KiB/s): min=527360, max=532480, per=100.00%, avg=530166.36, stdev=1040.89, samples=239
614 iops : min= 1030, max= 1040, avg=1035.48, stdev= 2.03, samples=239
615 lat (msec) : 2=0.01%, 4=0.01%, 10=99.99%, 20=0.01%
616 cpu : usr=0.57%, sys=4.09%, ctx=124175, majf=0, minf=531
617 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
618 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
619 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
620 issued rwts: total=124168,0,0,0 short=0,0,0,0 dropped=0,0,0,0
621 latency : target=0, window=0, percentile=100.00%, depth=8
622
623Run status group 0 (all jobs):
624 READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=60.6GiB (65.1GB), run=120008-120008msec
625
626Disk stats (read/write):
627 sdg: ios=248099/0, merge=0/0, ticks=1842144/0, in_queue=1842144, util=100.00%
628
629
630
631######## cloudcephosd1034: /dev/sdh
632#### This is the disk that was replaced
633#### Smartctl
634smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
635Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
636
637=== START OF INFORMATION SECTION ===
638Model Family: Dell Certified Intel S4x00/D3-S4x10 Series SSDs
639Device Model: SSDSC2KG019T8R
640Serial Number: PHYG225005VJ1P9DGN
641LU WWN Device Id: 5 5cd2e4 1556da5d6
642Add. Product Id: DELL(tm)
643Firmware Version: XCV1DL69
644User Capacity: 1,920,383,410,176 bytes [1.92 TB]
645Sector Sizes: 512 bytes logical, 4096 bytes physical
646Rotation Rate: Solid State Device
647Form Factor: 2.5 inches
648TRIM Command: Available, deterministic, zeroed
649Device is: In smartctl database [for details use: -P show]
650ATA Version is: ACS-3 T13/2161-D revision 5
651SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
652Local Time is: Fri Mar 22 10:01:50 2024 UTC
653SMART support is: Available - device has SMART capability.
654SMART support is: Enabled
655
656=== START OF READ SMART DATA SECTION ===
657SMART overall-health self-assessment test result: PASSED
658
659General SMART Values:
660Offline data collection status: (0x02) Offline data collection activity
661 was completed without error.
662 Auto Offline Data Collection: Disabled.
663Self-test execution status: ( 0) The previous self-test routine completed
664 without error or no self-test has ever
665 been run.
666Total time to complete Offline
667data collection: ( 2) seconds.
668Offline data collection
669capabilities: (0x79) SMART execute Offline immediate.
670 No Auto Offline data collection support.
671 Suspend Offline collection upon new
672 command.
673 Offline surface scan supported.
674 Self-test supported.
675 Conveyance Self-test supported.
676 Selective Self-test supported.
677SMART capabilities: (0x0003) Saves SMART data before entering
678 power-saving mode.
679 Supports SMART auto save timer.
680Error logging capability: (0x01) Error logging supported.
681 General Purpose Logging supported.
682Short self-test routine
683recommended polling time: ( 1) minutes.
684Extended self-test routine
685recommended polling time: ( 60) minutes.
686Conveyance self-test routine
687recommended polling time: ( 60) minutes.
688SCT capabilities: (0x003d) SCT Status supported.
689 SCT Error Recovery Control supported.
690 SCT Feature Control supported.
691 SCT Data Table supported.
692
693SMART Attributes Data Structure revision number: 1
694Vendor Specific SMART Attributes with Thresholds:
695ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
696 1 Raw_Read_Error_Rate 0x000e 130 130 039 Old_age Always - 4294967295
697 5 Reallocated_Sector_Ct 0x0033 100 100 001 Pre-fail Always - 0
698 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 13206
699 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 4
700 13 Read_Soft_Error_Rate 0x001e 080 080 000 Old_age Always - 281474976710655
701170 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
702173 Unknown_Attribute 0x0012 097 097 000 Old_age Always - 308
703174 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 3
704175 Program_Fail_Count_Chip 0x0012 100 100 010 Old_age Always - 305
705179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 010 Pre-fail Always - 0
706180 Unused_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 20590
707181 Program_Fail_Cnt_Total 0x003a 100 100 000 Old_age Always - 0
708182 Erase_Fail_Count_Total 0x003a 100 100 000 Old_age Always - 0
709184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0
710194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 25
711195 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0
712197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
713198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
714199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
715201 Power_Loss_Cap_Test 0x0033 100 100 010 Pre-fail Always - 2343 (6 65535)
716202 End_of_Life 0x0027 100 100 000 Pre-fail Always - 0
717233 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 6467662
718234 Thermal_Throttle_Status 0x0032 100 100 000 Old_age Always - 0/0
719235 Unknown_Attribute 0x000b 100 100 000 Pre-fail Always - 6467662
720241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 6467662
721242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 4410081
722245 Percent_Life_Remaining 0x0032 097 097 000 Old_age Always - 97
723
724SMART Error Log not supported
725
726SMART Self-test log structure revision number 1
727Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
728# 1 Short offline Completed without error 00% 9713 -
729
730Read SMART Selective Self-test Log failed: scsi error aborted command
731
732
733
734
735#### FIO
736test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
737fio-3.25
738Starting 1 process
739
740test: (groupid=0, jobs=1): err= 0: pid=276558: Fri Mar 22 10:03:51 2024
741 read: IOPS=910, BW=455MiB/s (477MB/s)(53.3GiB/120009msec)
742 slat (usec): min=3, max=670, avg=31.19, stdev=15.00
743 clat (usec): min=2737, max=15063, avg=8756.06, stdev=754.94
744 lat (usec): min=2937, max=15106, avg=8787.56, stdev=754.70
745 clat percentiles (usec):
746 | 1.00th=[ 7308], 5.00th=[ 7701], 10.00th=[ 7898], 20.00th=[ 8160],
747 | 30.00th=[ 8291], 40.00th=[ 8455], 50.00th=[ 8717], 60.00th=[ 8848],
748 | 70.00th=[ 9110], 80.00th=[ 9372], 90.00th=[ 9765], 95.00th=[10159],
749 | 99.00th=[10945], 99.50th=[11338], 99.90th=[11994], 99.95th=[12256],
750 | 99.99th=[13173]
751 bw ( KiB/s): min=424960, max=478208, per=100.00%, avg=466309.89, stdev=5122.73, samples=239
752 iops : min= 830, max= 934, avg=910.76, stdev=10.01, samples=239
753 lat (msec) : 4=0.01%, 10=93.47%, 20=6.53%
754 cpu : usr=0.60%, sys=3.88%, ctx=105292, majf=0, minf=660
755 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
756 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
757 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
758 issued rwts: total=109217,0,0,0 short=0,0,0,0 dropped=0,0,0,0
759 latency : target=0, window=0, percentile=100.00%, depth=8
760
761Run status group 0 (all jobs):
762 READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=53.3GiB (57.3GB), run=120009-120009msec
763
764Disk stats (read/write):
765 sdh: ios=218216/0, merge=0/0, ticks=1826064/0, in_queue=1826063, util=99.99%
766
767
768
769######## cloudcephosd1034: /dev/sdi
770#### Smartctl
771smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
772Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
773
774=== START OF INFORMATION SECTION ===
775Device Model: HFS1T9G32FEH-BA10A
776Serial Number: KJA8N5701I0808I60
777LU WWN Device Id: 5 ace42e 0253539a4
778Add. Product Id: DELL(tm)
779Firmware Version: DD02
780User Capacity: 1,920,383,410,176 bytes [1.92 TB]
781Sector Sizes: 512 bytes logical, 4096 bytes physical
782Rotation Rate: Solid State Device
783Form Factor: 2.5 inches
784TRIM Command: Available, deterministic, zeroed
785Device is: Not in smartctl database [for details use: -P showall]
786ATA Version is: ACS-4 (minor revision not indicated)
787SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
788Local Time is: Fri Mar 22 10:03:51 2024 UTC
789SMART support is: Available - device has SMART capability.
790SMART support is: Enabled
791
792=== START OF READ SMART DATA SECTION ===
793SMART overall-health self-assessment test result: PASSED
794
795General SMART Values:
796Offline data collection status: (0x02) Offline data collection activity
797 was completed without error.
798 Auto Offline Data Collection: Disabled.
799Self-test execution status: ( 0) The previous self-test routine completed
800 without error or no self-test has ever
801 been run.
802Total time to complete Offline
803data collection: ( 1730) seconds.
804Offline data collection
805capabilities: (0x19) SMART execute Offline immediate.
806 No Auto Offline data collection support.
807 Suspend Offline collection upon new
808 command.
809 Offline surface scan supported.
810 Self-test supported.
811 No Conveyance Self-test supported.
812 No Selective Self-test supported.
813SMART capabilities: (0x0002) Does not save SMART data before
814 entering power-saving mode.
815 Supports SMART auto save timer.
816Error logging capability: (0x01) Error logging supported.
817 General Purpose Logging supported.
818Short self-test routine
819recommended polling time: ( 2) minutes.
820Extended self-test routine
821recommended polling time: ( 30) minutes.
822SCT capabilities: (0x003d) SCT Status supported.
823 SCT Error Recovery Control supported.
824 SCT Feature Control supported.
825 SCT Data Table supported.
826
827SMART Attributes Data Structure revision number: 0
828Vendor Specific SMART Attributes with Thresholds:
829ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
830 1 Raw_Read_Error_Rate 0x000e 100 100 006 Old_age Always - 0
831 5 Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
832 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15173
833 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 41
834 13 Read_Soft_Error_Rate 0x002e 100 100 000 Old_age Always - 0
835173 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 245
836175 Program_Fail_Count_Chip 0x0032 099 099 000 Old_age Always - 230
837179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 002 Pre-fail Always - 0
838180 Unused_Rsvd_Blk_Cnt_Tot 0x0010 100 100 000 Old_age Offline - 4654
839181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always - 0
840182 Erase_Fail_Count_Total 0x0032 100 100 000 Old_age Always - 0
841194 Temperature_Celsius 0x0002 066 056 000 Old_age Always - 34 (Min/Max 21/44)
842195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
843198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 744
844199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
845201 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
846202 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
847233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 196687
848235 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 196687
849241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 503336
850245 Unknown_Attribute 0x0033 099 099 001 Pre-fail Always - 99
851
852SMART Error Log not supported
853
854SMART Self-test Log not supported
855
856Selective Self-tests/Logging not supported
857
858
859
860
861#### FIO
862test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
863fio-3.25
864Starting 1 process
865
866test: (groupid=0, jobs=1): err= 0: pid=277155: Fri Mar 22 10:05:51 2024
867 read: IOPS=1033, BW=517MiB/s (542MB/s)(60.6GiB/120008msec)
868 slat (usec): min=4, max=681, avg=29.42, stdev=14.06
869 clat (usec): min=1228, max=16211, avg=7706.56, stdev=253.15
870 lat (usec): min=1465, max=16254, avg=7736.24, stdev=252.41
871 clat percentiles (usec):
872 | 1.00th=[ 7177], 5.00th=[ 7242], 10.00th=[ 7439], 20.00th=[ 7635],
873 | 30.00th=[ 7635], 40.00th=[ 7701], 50.00th=[ 7701], 60.00th=[ 7701],
874 | 70.00th=[ 7767], 80.00th=[ 7767], 90.00th=[ 7963], 95.00th=[ 8160],
875 | 99.00th=[ 8455], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9372],
876 | 99.99th=[10683]
877 bw ( KiB/s): min=523264, max=532480, per=100.00%, avg=529720.77, stdev=1075.83, samples=239
878 iops : min= 1022, max= 1040, avg=1034.61, stdev= 2.10, samples=239
879 lat (msec) : 2=0.01%, 4=0.01%, 10=99.98%, 20=0.02%
880 cpu : usr=0.45%, sys=4.18%, ctx=124073, majf=0, minf=660
881 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
882 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
883 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
884 issued rwts: total=124065,0,0,0 short=0,0,0,0 dropped=0,0,0,0
885 latency : target=0, window=0, percentile=100.00%, depth=8
886
887Run status group 0 (all jobs):
888 READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=60.6GiB (65.0GB), run=120008-120008msec
889
890Disk stats (read/write):
891 sdi: ios=247894/0, merge=0/0, ticks=1841078/0, in_queue=1841077, util=100.00%
892
893
894
895######## cloudcephosd1034: /dev/sdj
896#### Smartctl
897smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-26-amd64] (local build)
898Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
899
900=== START OF INFORMATION SECTION ===
901Device Model: HFS1T9G32FEH-BA10A
902Serial Number: KJA8N5701I0808I6D
903LU WWN Device Id: 5 ace42e 0253539b1
904Add. Product Id: DELL(tm)
905Firmware Version: DD02
906User Capacity: 1,920,383,410,176 bytes [1.92 TB]
907Sector Sizes: 512 bytes logical, 4096 bytes physical
908Rotation Rate: Solid State Device
909Form Factor: 2.5 inches
910TRIM Command: Available, deterministic, zeroed
911Device is: Not in smartctl database [for details use: -P showall]
912ATA Version is: ACS-4 (minor revision not indicated)
913SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
914Local Time is: Fri Mar 22 10:05:51 2024 UTC
915SMART support is: Available - device has SMART capability.
916SMART support is: Enabled
917
918=== START OF READ SMART DATA SECTION ===
919SMART overall-health self-assessment test result: PASSED
920
921General SMART Values:
922Offline data collection status: (0x02) Offline data collection activity
923 was completed without error.
924 Auto Offline Data Collection: Disabled.
925Self-test execution status: ( 0) The previous self-test routine completed
926 without error or no self-test has ever
927 been run.
928Total time to complete Offline
929data collection: ( 1730) seconds.
930Offline data collection
931capabilities: (0x19) SMART execute Offline immediate.
932 No Auto Offline data collection support.
933 Suspend Offline collection upon new
934 command.
935 Offline surface scan supported.
936 Self-test supported.
937 No Conveyance Self-test supported.
938 No Selective Self-test supported.
939SMART capabilities: (0x0002) Does not save SMART data before
940 entering power-saving mode.
941 Supports SMART auto save timer.
942Error logging capability: (0x01) Error logging supported.
943 General Purpose Logging supported.
944Short self-test routine
945recommended polling time: ( 2) minutes.
946Extended self-test routine
947recommended polling time: ( 30) minutes.
948SCT capabilities: (0x003d) SCT Status supported.
949 SCT Error Recovery Control supported.
950 SCT Feature Control supported.
951 SCT Data Table supported.
952
953SMART Attributes Data Structure revision number: 0
954Vendor Specific SMART Attributes with Thresholds:
955ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
956 1 Raw_Read_Error_Rate 0x000e 100 100 006 Old_age Always - 0
957 5 Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
958 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15173
959 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 41
960 13 Read_Soft_Error_Rate 0x002e 100 100 000 Old_age Always - 0
961173 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 243
962175 Program_Fail_Count_Chip 0x0032 099 099 000 Old_age Always - 226
963179 Used_Rsvd_Blk_Cnt_Tot 0x0033 100 100 002 Pre-fail Always - 0
964180 Unused_Rsvd_Blk_Cnt_Tot 0x0010 100 100 000 Old_age Offline - 4660
965181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always - 0
966182 Erase_Fail_Count_Total 0x0032 100 100 000 Old_age Always - 0
967194 Temperature_Celsius 0x0002 065 056 000 Old_age Always - 35 (Min/Max 21/44)
968195 Hardware_ECC_Recovered 0x0032 100 100 000 Old_age Always - 0
969198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 560
970199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
971201 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
972202 Unknown_SSD_Attribute 0x0033 100 100 050 Pre-fail Always - 0
973233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 194191
974235 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 194191
975241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 493622
976245 Unknown_Attribute 0x0033 099 099 001 Pre-fail Always - 99
977
978SMART Error Log not supported
979
980SMART Self-test Log not supported
981
982Selective Self-tests/Logging not supported
983
984
985
986
987#### FIO
988test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=8
989fio-3.25
990Starting 1 process
991
992test: (groupid=0, jobs=1): err= 0: pid=277765: Fri Mar 22 10:07:51 2024
993 read: IOPS=1033, BW=517MiB/s (542MB/s)(60.6GiB/120007msec)
994 slat (usec): min=4, max=713, avg=29.55, stdev=13.98
995 clat (usec): min=1206, max=14535, avg=7705.93, stdev=253.42
996 lat (usec): min=1921, max=14553, avg=7735.73, stdev=252.68
997 clat percentiles (usec):
998 | 1.00th=[ 7177], 5.00th=[ 7242], 10.00th=[ 7439], 20.00th=[ 7635],
999 | 30.00th=[ 7635], 40.00th=[ 7635], 50.00th=[ 7701], 60.00th=[ 7701],
1000 | 70.00th=[ 7767], 80.00th=[ 7767], 90.00th=[ 8029], 95.00th=[ 8160],
1001 | 99.00th=[ 8455], 99.50th=[ 8586], 99.90th=[ 9241], 99.95th=[ 9503],
1002 | 99.99th=[10290]
1003 bw ( KiB/s): min=526336, max=531456, per=100.00%, avg=529755.05, stdev=1035.87, samples=239
1004 iops : min= 1028, max= 1038, avg=1034.68, stdev= 2.01, samples=239
1005 lat (msec) : 2=0.01%, 4=0.01%, 10=99.98%, 20=0.02%
1006 cpu : usr=0.56%, sys=4.09%, ctx=124079, majf=0, minf=662
1007 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
1008 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
1009 complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
1010 issued rwts: total=124070,0,0,0 short=0,0,0,0 dropped=0,0,0,0
1011 latency : target=0, window=0, percentile=100.00%, depth=8
1012
1013Run status group 0 (all jobs):
1014 READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=60.6GiB (65.0GB), run=120007-120007msec
1015
1016Disk stats (read/write):
1017 sdj: ios=247895/0, merge=0/0, ticks=1840712/0, in_queue=1840712, util=100.00%
1018
1019
1020

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fnegri edited projects, added cloud-services-team (FY2023/2024-Q1-Q2); removed cloud-services-team.Oct 30 2023, 4:13 PM
Stashbot added a comment.Oct 30 2023, 4:29 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:29:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node (T348643)

Stashbot added a comment.Oct 30 2023, 4:29 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:29:50Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)

Stashbot added a comment.Oct 30 2023, 4:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:30:01Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node (T348643)

Stashbot added a comment.Oct 30 2023, 4:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:30:05Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)

Stashbot added a comment.Oct 30 2023, 4:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:30:29Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node (T348643)

Stashbot added a comment.Oct 30 2023, 4:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:37:28Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node (T348643)

Stashbot added a comment.Oct 30 2023, 4:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:37:34Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)

Stashbot added a comment.Oct 30 2023, 4:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:37:39Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node (T348643)

Stashbot added a comment.Oct 30 2023, 4:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:37:48Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)

Stashbot added a comment.Oct 30 2023, 4:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:38:52Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node (T348643)

Stashbot added a comment.Oct 30 2023, 4:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2023-10-30T16:38:56Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)

Andrew updated the task description. (Show Details)Oct 31 2023, 1:59 AM
Andrew updated the task description. (Show Details)Oct 31 2023, 2:14 PM
Andrew updated the task description. (Show Details)Nov 1 2023, 2:26 AM
Andrew added a comment.Nov 1 2023, 3:10 AM

In order to see if things are degrading, here's a point in time slice:

P53122

Jclark-ctr updated the task description. (Show Details)Nov 1 2023, 2:43 PM
Andrew updated the task description. (Show Details)Nov 1 2023, 2:49 PM
Andrew added a comment.Nov 1 2023, 3:39 PM

slice from this morning:

P53124

(by the way, I'm generating that with

cumin1001:~$ sudo cumin "P{cloudcephosd*eqiad*}" "smartctl -a /dev/sdj | grep Offline_Uncorrectable" --force | phaste

)

Andrew updated the task description. (Show Details)Nov 2 2023, 1:48 PM
Jclark-ctr updated the task description. (Show Details)Nov 2 2023, 3:04 PM
Andrew updated the task description. (Show Details)Nov 3 2023, 1:45 AM
Jclark-ctr updated the task description. (Show Details)Nov 3 2023, 2:44 PM
Andrew updated the task description. (Show Details)Nov 3 2023, 4:29 PM
Andrew updated the task description. (Show Details)Nov 5 2023, 12:44 AM
Jclark-ctr updated the task description. (Show Details)Nov 6 2023, 3:40 PM
Andrew added a comment.Nov 6 2023, 9:23 PM

P53144

Andrew added a comment.Nov 6 2023, 9:38 PM

No new errors reported since the 1st. I'm not clear on if that means we've fixed the problem or not; there are still a ton of errors reporting. It does sort of fit Dell's implication that everything went wrong at once and this isn't an ongoing issue.

Andrew added a comment.Nov 17 2023, 1:13 AM

P53530

Andrew added a comment.EditedNov 17 2023, 1:24 AM

Here's the change in errors on /dev/sdj since the 31st.

4c4
< (1) cloudcephosd1024.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2840
---
> (1) cloudcephosd1024.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2856
6c6
< (1) cloudcephosd1026.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1152
---
> (1) cloudcephosd1026.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       984
8c8
< (1) cloudcephosd1028.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       3032
---
> (1) cloudcephosd1028.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1944
13c13
< (1) cloudcephosd1033.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2024
---
> (1) cloudcephosd1033.eqiad.wmnet 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2040

Fairly little degredation (and no change on the 12 over drives not listed), but still /some/ degredation. And of course there are still a ton of errors from before the firmware upgrades.

I'm not 100% confident in my interpretation of these numbers. It seems like things are much better since the firmware upgrade, but it's also possible that the issue is bursty and we just haven't had a burst in a while. The fact that this shows 'uncorrectable errors' being corrected on some drives is mysterious.

Andrew updated the task description. (Show Details)Nov 17 2023, 3:30 PM
taavi closed subtask T349425: CephSlowOps Ceph cluster in has slow ops, which might be blocking some writes as Resolved.Nov 27 2023, 5:10 PM
dcaro added a subtask: T352570: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes.Dec 7 2023, 11:03 AM
Jclark-ctr added a comment.Dec 11 2023, 5:05 PM

@Andrew Dell is requesting smartctl output showing what drives errors are coming from if you can send the full output

Andrew added a comment.Dec 12 2023, 4:51 PM

P54340

Andrew added a comment.Dec 12 2023, 5:23 PM

Here's a new diff. This compares outputs from Nov 17 with today. The < is from the 17th, the < is today.

< cloudcephosd1021.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1856
< cloudcephosd1022.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1296
---
> cloudcephosd1021.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1912
> cloudcephosd1022.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1312
7,8c7,8
< cloudcephosd1024.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2856
< cloudcephosd1025.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       944
---
> cloudcephosd1024.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2864
> cloudcephosd1025.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       960
11,13c11,13
< cloudcephosd1028.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1944
< cloudcephosd1029.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2600
< cloudcephosd1030.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1256
---
> cloudcephosd1028.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1952
> cloudcephosd1029.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2608
> cloudcephosd1030.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       1264
16,17c16,17
< cloudcephosd1033.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2040
< cloudcephosd1034.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       512
---
> cloudcephosd1033.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2064
> cloudcephosd1034.eqiad.wmnet ----- OUTPUT of 'smartctl -a /dev...ne_Uncorrectable' ----- 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       520
Andrew added a comment.Dec 12 2023, 5:39 PM
In T348643#9397062, @Jclark-ctr wrote:

@Andrew Dell is requesting smartctl output showing what drives errors are coming from if you can send the full output

Here is the full output for every drive on every host:

P54343

Note that the drives aren't individually tagged but the output is in order from sda through sdj.

dcaro added a comment.Dec 15 2023, 11:07 AM

I got some graphs here with the increases of the error count:
https://grafana-rw.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&from=now-7d&to=now

image.png (817×1 px, 111 KB)

There's many increases of small amount on many of the hosts (not on the old ones though it seems).

We also only have a few weeks of data, as it seems that we started collecting it only at the end of november.

I'm not sure that we are collecting it from all the hosts either, and the devices are shown as 'megaraid', looking

dcaro added a comment.Dec 15 2023, 11:11 AM

That might be all of them since nov 28th:

dcaro@urcuchillay$ head -n 1 Hard\ drives\ health\ -\ increase\ in\ offline\ uncorrectable-data-as-joinbyfield-2023-12-15\ 11\ 08\ 55.csv | grep -o 'cloudcephosd1...' | sort | uniq -c
      3 cloudcephosd1021
      4 cloudcephosd1022
      1 cloudcephosd1023
      5 cloudcephosd1024
      5 cloudcephosd1025
      3 cloudcephosd1026
      2 cloudcephosd1027
      5 cloudcephosd1028
      4 cloudcephosd1029
      6 cloudcephosd1030
      1 cloudcephosd1031
      3 cloudcephosd1032
      2 cloudcephosd1033
      4 cloudcephosd1034

Not all the drives though

dcaro added a comment.Dec 15 2023, 12:07 PM

Got all the increments since we have data in prometheus 29th of Nov (data dump from grafana + a bit of pandas xd):

cloudcephmon2005-dev:9100-sda 8
cloudcephmon2005-dev:9100-sdb 24
cloudcephmon2005-dev:9100-sdc 16
cloudcephmon2005-dev:9100-sdd 32
cloudcephmon2006-dev:9100-sdc 8
cloudcephosd1021:9100-sat+megaraid,5 8
cloudcephosd1021:9100-sat+megaraid,6 32
cloudcephosd1021:9100-sat+megaraid,9 56
cloudcephosd1022:9100-sat+megaraid,3 24
cloudcephosd1022:9100-sat+megaraid,4 8
cloudcephosd1022:9100-sat+megaraid,6 24
cloudcephosd1022:9100-sat+megaraid,9 16
cloudcephosd1023:9100-sat+megaraid,2 8
cloudcephosd1024:9100-sat+megaraid,3 16
cloudcephosd1024:9100-sat+megaraid,4 32
cloudcephosd1024:9100-sat+megaraid,5 8
cloudcephosd1024:9100-sat+megaraid,6 8
cloudcephosd1024:9100-sat+megaraid,9 8
cloudcephosd1025:9100-sat+megaraid,2 8
cloudcephosd1025:9100-sat+megaraid,3 16
cloudcephosd1025:9100-sat+megaraid,4 72
cloudcephosd1025:9100-sat+megaraid,6 16
cloudcephosd1025:9100-sat+megaraid,9 8
cloudcephosd1026:9100-sat+megaraid,3 8
cloudcephosd1026:9100-sat+megaraid,7 16
cloudcephosd1026:9100-sat+megaraid,8 16
cloudcephosd1027:9100-sat+megaraid,2 8
cloudcephosd1027:9100-sat+megaraid,8 8
cloudcephosd1028:9100-sat+megaraid,4 16
cloudcephosd1028:9100-sat+megaraid,5 40
cloudcephosd1028:9100-sat+megaraid,6 8
cloudcephosd1028:9100-sat+megaraid,7 32
cloudcephosd1028:9100-sat+megaraid,9 16
cloudcephosd1029:9100-sat+megaraid,2 40
cloudcephosd1029:9100-sat+megaraid,3 8
cloudcephosd1029:9100-sat+megaraid,5 8
cloudcephosd1029:9100-sat+megaraid,6 24
cloudcephosd1030:9100-sat+megaraid,3 16
cloudcephosd1030:9100-sat+megaraid,4 8
cloudcephosd1030:9100-sat+megaraid,5 40
cloudcephosd1030:9100-sat+megaraid,6 16
cloudcephosd1030:9100-sat+megaraid,8 8
cloudcephosd1030:9100-sat+megaraid,9 32
cloudcephosd1031:9100-sat+megaraid,3 16
cloudcephosd1032:9100-sat+megaraid,2 16
cloudcephosd1032:9100-sat+megaraid,4 8
cloudcephosd1032:9100-sat+megaraid,5 8
cloudcephosd1033:9100-sat+megaraid,4 16
cloudcephosd1033:9100-sat+megaraid,9 32
cloudcephosd1034:9100-sat+megaraid,2 8
cloudcephosd1034:9100-sat+megaraid,5 8
cloudcephosd1034:9100-sat+megaraid,6 8
cloudcephosd1034:9100-sat+megaraid,9 8
Jclark-ctr added a comment.Dec 20 2023, 4:47 PM

i have been back and forth with dell no answer yet of what is causing this I still believe it is still a Firmware issue of course dell will not admit if its a known issue

Jclark-ctr added a comment.Jan 8 2024, 3:38 PM

Self dispatched 8 new drives for cloudcephosd1028

dcaro added a comment.Jan 8 2024, 3:39 PM

Scheduled next thursday to do the swap of the drives, will get the host out of the cluster before that.

fnegri changed the task status from Open to In Progress.Jan 8 2024, 5:55 PM
fnegri triaged this task as High priority.
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.
dcaro added a comment.Jan 9 2024, 1:15 PM

They denied the new drives request, so will try to gather information in a per-host basis with:

  • Smartctl of each hard drive
  • A note on the increase of errors for each hard drive
dcaro added a comment.Jan 9 2024, 1:17 PM

Helper graph here https://grafana-rw.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?forceLogin&forceLogin&from=now-7d&orgId=1&to=now (the lower one, with the counts of errors increase in the last 30d, this might be helpful to get the data https://github.com/dcoles/prometheus-pandas)

dcaro added a comment.EditedJan 24 2024, 5:01 PM

Some 'raw' data on the last 30 days increase of errors per-host/drive:

cloudcephosd1021-sdc 88
cloudcephosd1021-sdd 24
cloudcephosd1021-sde 8
cloudcephosd1021-sdf 80
cloudcephosd1021-sdg 32
cloudcephosd1021-sdh 24
cloudcephosd1021-sdi 8
cloudcephosd1021-sdj 72
cloudcephosd1022-sdc 16
cloudcephosd1022-sdd 56
cloudcephosd1022-sde 16
cloudcephosd1022-sdf 24
cloudcephosd1022-sdg 48
cloudcephosd1022-sdh 40
cloudcephosd1022-sdi 40
cloudcephosd1022-sdj 56
cloudcephosd1023-sdc 16
cloudcephosd1023-sdd 40
cloudcephosd1023-sde 16
cloudcephosd1023-sdf 32
cloudcephosd1023-sdg 96
cloudcephosd1023-sdh 56
cloudcephosd1023-sdi 56
cloudcephosd1023-sdj 24
cloudcephosd1024-sdc 216
cloudcephosd1024-sdd 8
cloudcephosd1024-sde 32
cloudcephosd1024-sdf 136
cloudcephosd1024-sdg 64
cloudcephosd1024-sdh 40
cloudcephosd1024-sdi 40
cloudcephosd1024-sdj 144
cloudcephosd1025-sdc 16
cloudcephosd1025-sdd 104
cloudcephosd1025-sde 104
cloudcephosd1025-sdf 48
cloudcephosd1025-sdg 32
cloudcephosd1025-sdh 16
cloudcephosd1025-sdi 32
cloudcephosd1025-sdj 80
cloudcephosd1026-sdc 56
cloudcephosd1026-sdd 40
cloudcephosd1026-sde 32
cloudcephosd1026-sdf 40
cloudcephosd1026-sdg 64
cloudcephosd1026-sdh 64
cloudcephosd1026-sdi 96
cloudcephosd1026-sdj 40
cloudcephosd1027-sdc 88
cloudcephosd1027-sdd 16
cloudcephosd1027-sde 16
cloudcephosd1027-sdf 144
cloudcephosd1027-sdg 32
cloudcephosd1027-sdh 48
cloudcephosd1027-sdi 16
cloudcephosd1027-sdj 8
cloudcephosd1028-sdc 32
cloudcephosd1028-sdd 40
cloudcephosd1028-sde 48
cloudcephosd1028-sdf 392
cloudcephosd1028-sdg 80
cloudcephosd1028-sdh 24
cloudcephosd1028-sdi 96
cloudcephosd1028-sdj 136
cloudcephosd1029-sdc 48
cloudcephosd1029-sdd 88
cloudcephosd1029-sde 32
cloudcephosd1029-sdf 24
cloudcephosd1029-sdg 224
cloudcephosd1029-sdh 96
cloudcephosd1029-sdi 32
cloudcephosd1029-sdj 24
cloudcephosd1030-sdc 88
cloudcephosd1030-sdd 16
cloudcephosd1030-sde 96
cloudcephosd1030-sdf 200
cloudcephosd1030-sdg 144
cloudcephosd1030-sdh 128
cloudcephosd1030-sdi 32
cloudcephosd1030-sdj 112
cloudcephosd1031-sdc 64
cloudcephosd1031-sdd 80
cloudcephosd1031-sde 80
cloudcephosd1031-sdf 88
cloudcephosd1031-sdg 48
cloudcephosd1031-sdh 64
cloudcephosd1031-sdi 192
cloudcephosd1031-sdj 144
cloudcephosd1032-sdc 64
cloudcephosd1032-sdd 72
cloudcephosd1032-sde 40
cloudcephosd1032-sdf 72
cloudcephosd1032-sdg 272
cloudcephosd1032-sdh 40
cloudcephosd1032-sdi 88
cloudcephosd1032-sdj 48
cloudcephosd1033-sdc 48
cloudcephosd1033-sdd 40
cloudcephosd1033-sde 56
cloudcephosd1033-sdf 24
cloudcephosd1033-sdg 72
cloudcephosd1033-sdh 48
cloudcephosd1033-sdi 16
cloudcephosd1033-sdj 176
cloudcephosd1034-sdc 160
cloudcephosd1034-sdd 56
cloudcephosd1034-sde 24
cloudcephosd1034-sdf 48
cloudcephosd1034-sdg 48
cloudcephosd1034-sdi 40
cloudcephosd1034-sdj 24
dcaro added a comment.Jan 24 2024, 5:36 PM

I'm running a script now to gather nicer reports with smartctl included, will send it once it's finished.

dcaro added a comment.Jan 24 2024, 5:43 PM

Here you go, that has one directory per host, with one file per drive with the total increase of errors in the last 30d plus the smartctl output on it.
Let me know if you need more, sorry for the delay :S

ceph_errors_data.tgz13 KBDownload

Jclark-ctr added a comment.Jan 25 2024, 2:49 PM

Submitted all new tsr reports along with smartctl data

Jclark-ctr moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jan 29 2024, 7:14 PM
fnegri edited projects, added cloud-services-team (FY2023/2024-Q3-Q4); removed cloud-services-team (FY2023/2024-Q1-Q2).Feb 1 2024, 11:13 AM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q3-Q4) board.
Jclark-ctr added a comment.Feb 7 2024, 2:17 PM

@dcaro The same response was sent for each case please advise how you would like me to proceed.

Dell Case
184631498
184631254
184631152
184631061
184630986
184630873
184630754
184630654
184630512
184630409
184630359
184629686
184629625

Dells Response below. Phone numbers , service tags , And extensions removed

I have also reviewed smartctl logs; looking over the log, all of the drives have passed smart test, none have any reallocated_sector_ct going up, just the offline_uncorrectable counts showing, so until data is written to these sectors, at which time they will either re-allocate indicating a bad sector or be recovered instead.

If the number of re-allocated sectors goes up higher this can be an indication to a drive that could be pre-fail but would likely need other indicators as well or a really high number.

You can review Dell's online Knowledge Base Article Number: 000013062: Centera: How to use smartctl to determine drive health. (Centera: How to use smartctl to determine drive health. | Dell US ).

To view the above KB article, you may need to log into the dell.com site for access.

Per the KB article:
Reallocated_Sector_Ct

Count of reallocated sectors. When the hard drive finds a read/write/verification error, it marks this sector as "reallocated" and transfers data to a special reserved area (spare area). This process is also known as remapping and "reallocated" sectors are called remaps. This is why, on a modern hard disks, you cannot see "bad blocks" while testing the surface - all bad blocks are hidden in reallocated sectors. This value is usually less than 2500 but is vendor and drive dependent. Once this gets above the 2000 mark you will usually see the self assessment report the drive as "FAILING".

Having a high number of reallocated sectors does not mean that the drive is ?bad?. It simply means that there has been "X" number of sectors remapped. Having said that, a high number of reallocated sectors can indicate that there is some sort of defect on the disk surface and typically those defects can grow resulting in more sectors that need to be re-allocated.

Current_Pending_Sector

Current count of unstable sectors (waiting for remapping). The raw value of this attribute indicates the total number of sectors waiting for remapping. Later, when some of these sectors are read successfully, the value is decreased. If errors still occur when reading some sector, the hard drive will try to restore the data, transfer it to the reserved disk area (spare area) and mark this sector as remapped.

A "high" value might be a number above 10 or 20. Enhanced Data Management (EDM) will usually detect this and repair the partition unless the pending sectors are in a partition that EDM does not scrub (boot, upgrade, swap partitions, sector zero, or sectors on partition boundaries).

I have found that this value does NOT always indicate all of the unreadable sectors. Sometimes Linux will report Uncorrectable errors yet they do not show up in the SMART data.

When EDM repairs a drive it will usually result in this number decreasing. Remember, this number represents all of the pending sectors for the ENTIRE disk and not just the blob partitions on that drive.

Bottom line, even after discussing all of the logs with one of my senior colleagues, there does not appear to be any hard drive issues on the server at this time, either from the TSR or the smartctl logs.

What is the status of the server, is your team getting hard drive errors in the OS?

If you need additional assistance, you are welcome to respond to this email.

If immediate assistance is required, please call back into the support queue at ## and provide the triage agent with the Case# ; ask for Greg at extension #. If I am not available, you can work with the next available technician.

Please inform me of any issues or concerns prior to my closing out the case, so I may address them immediately, as your satisfaction with our support is wholly my responsibility.

dcaro added a comment.Feb 7 2024, 5:02 PM

@Jclark-ctr thanks!

Can you forward them the following questions and info?

  • What does offline_uncorrectable mean?
  • If I understood correctly, that offline_uncorrectable count should be going down eventually, when another write is done to the sectors, but for us it only goes up, is there anything we are not doing?
  • What would be a "high number" for offline_uncorrectable errors? (we have >2k on many disks already, and only growing)
  • What is the impact on performance for the disk when it has to flag a sector as offline_uncorrectable?
  • The effect that we see on the servers is that some operations take way longer than they should (in the order of minutes), and those being the storage servers underlying the foundation of our cloud, the disks misbehaving makes the whole cloud halt to the point of a lot of the VMs needing to get restarted and losing service.
  • Note that so far, we have not seen that issue on the old servers, only on the ones reporting offline_uncorrectable issues.
Jclark-ctr added a comment.Feb 14 2024, 8:07 PM

@dcaro here is an update from dell

I found a couple of online articles:
What Does Uncorrectable Sector Count Mean & How to Fix It - MiniTool
How to interpret following S.M.A.R.T values: “ATA Error Count”, Offline_Uncorrectable? - WD Internal Drives / Desktop & Mobile Drives - WD Community

My senior engineer provided the following information:

The offline uncorrectable counter will not go down, it increments for the life of the drive; it is not really a wear indicator and not used to monitor for issues.

That counter can typically be ignored; if this counter is incrementing at an alarming state we would look at the following counters.
Reallocated_Sector_Ct 0x0033 100 100 002 Pre-fail Always - 0
Unused_Rsvd_Blk_Cnt_Tot 0x0012 100 100 000 Old_age Always - 1012

If reallocated sectors get too high, the drive will first go to a predicted failure state and then after some time, it will fail; same will happen if the unused rsvd blk count gets too low.

As of now, none of the hard drives have hit the threshold for the predicted failure state to trip the Pred Fail flag on the hard drives.

If that Pred Fail flag does show up on your hard drives, we would replace the hard drive at that time.

You can check the Pred Fail state in the iDRAC ->Storage->Physical Disks and expand out the + sign to check advanced properties on the drives: image.png

What is the status of the server, has everything been taken care of, or do you still need additional assistance?

If you need additional assistance, you are welcome to respond to this email.

If your emails are not being responded to within 2 - 4 hours, and immediate assistance is required, please call back into the support queue at 800 945-3355 and provide the triage agent with the case number 177592506; ask for Greg Stewart at extension 628-8089. If I am not available, you can work with the next available technician.

I am your Dell EMC primary point of contact through resolution of this Support Request.

Please inform me of any issues or concerns prior to my closing out the case, so I may address them immediately, as your satisfaction with our support is wholly my responsibility.

Here are the best ways to contact me:

Jclark-ctr added a comment.Feb 14 2024, 8:09 PM

https://www.minitool.com/lib/uncorrectable-sector-count.html
https://community.wd.com/t/how-to-interpret-following-s-m-a-r-t-values-ata-error-count-offline-uncorrectable/241408

dcaro added a comment.Feb 15 2024, 12:48 PM

thanks @Jclark-ctr, unfortunately, that does not really help a lot, and does not answer any of the questions :/

The offline uncorrectable counter will not go down, it increments for the life of the drive; it is not really a wear indicator and not used to monitor for issues.

The link https://www.minitool.com/lib/uncorrectable-sector-count.html, seems to not agree at all with that:

"Simply put, offline uncorrectable pending sector count warning implies imminent disk failure and requires immediate backup and drive replacement. Here is a full guide to fix hard drive uncorrectable sector count warning."

The other link: https://community.wd.com/t/how-to-interpret-following-s-m-a-r-t-values-ata-error-count-offline-uncorrectable/241408/5
Seems to report that the user stopped seeing that number increase when they replaced the SATA cable:
"With the new SATA cable no changes in ATA Error Count, so I’ll use this disk. The cable seems to be broken."

Can we try changing one of the cables, see if that has an impact?

if this counter is incrementing at an alarming state we would look at the following counters.

I will need also an actual raw value for this, how many is "an alarming state"?

dcaro added a subtask: T348716: [ceph] export number of bad sectors per-disk.Feb 22 2024, 8:53 AM
taavi closed subtask T352570: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes as Resolved.Feb 27 2024, 8:38 AM
dcaro updated the task description. (Show Details)Feb 28 2024, 11:16 AM
Andrew added a comment.Mar 5 2024, 7:41 PM

Notes from today's (unproductive) meeting:

We met with several Dell reps including an engineer named Michael. Michael feels very strongly (and maybe angrily) that 'Offline uncorrectable sectors' is a meaningless number with SSDs; that it's well-defined with a spinny disk but is essentially an undefined/reserved value for SSDs. He thinks we should ignore it and look for other problem metrics.

At that point I was at a bit of a loss because I don't think I have secondary issues at hand to point at. I mentioned that we'd been seeing performance issues that we presume are related; he's going to send a request for OS logs to try to investigate what's happening performance-wise.

They were unwilling to send us additional drives, partly because of not believing there's a problem, and partly because new drives would be a different make/model so wouldn't necessarily teach us anything.

We scheduled another call for Thursday morning which hopefully David can attend.

My takeaway:

  • Maybe there really isn't a drive problem or maybe Dell is just trying to make us go away; I can't tell which. In any case we need to come up with a symptom to talk to them about outside of 'Offline uncorrectable'.
  • I still sort of want to know what that number /really is/, Michael says he's in the process of asking the drive manufacturer the same question so maybe he'll follow up.
  • Dealing with Dell tech support is very unpleasant! Big ups to @Jclark-ctr for keeping his cool throughout.
fnegri awarded a token.Mar 6 2024, 11:43 AM
dcaro added a comment.Mar 8 2024, 1:37 PM

Notes from today's (more productive) meeting:

About the number in specific, they got from the hard drive vendor that it's not indicative of a failure, just a "symptom", but they did not reply yet on what does it mean...

The engineer has gone through some of the logs, but did not find anything that popped out, or timeout error or similar (no slowops happend on the host during the logs timespan).

They proposed the following:

  • Send 4 drives to replace on one of our hosts, so they can get the failing drives and do some tests on them (we will see then if the new drives have the same issue too)
  • Take down one host (the one with the replaced drive), and run a series of tests (need to be done from the DC, in person). This will not wipe the OS, but will mess up with the osd drives, so we have to take the host off the cluster and bring it in the cluster after.
  • Gather a sosreport from a host that does not have issues (one of the old hosts for example)
  • Try to gather a sosreport from a host that had slowops (if I can pin-point one and catch it soon enough).

@Jclark-ctr @Andrew please fill in anything I might have missed.

Jclark-ctr added a comment.Mar 8 2024, 3:16 PM

@dcaro thanks for the notes much more productive meeting. although nothing popped out for the engineer he also admitted he did not know what to look for on the logs for Debian linux. I will spend some time researching today/tonight while traveling to see what I can find also.

We will need to request an ftp site from dell for any logs. There ftp sites expire after a few days from my history with dell I believe its 5 days but could be 10.

Stashbot added a comment.Mar 19 2024, 1:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-19T13:30:18Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_node (T348643)

Stashbot added a comment.Mar 19 2024, 3:21 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-19T15:21:51Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)

Stashbot added a comment.Mar 19 2024, 3:49 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-19T15:49:10Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_node (T348643)

dcaro updated the task description. (Show Details)Mar 19 2024, 3:52 PM
Stashbot added a comment.Mar 19 2024, 6:28 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-19T18:28:15Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)

Stashbot added a comment.Mar 20 2024, 2:31 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-20T14:31:04Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_node (T348643)

Stashbot added a comment.Mar 20 2024, 3:00 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-20T15:00:43Z] <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T348643)

Stashbot added a comment.Mar 20 2024, 3:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-20T15:02:55Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_node (T348643)

Stashbot added a comment.Mar 20 2024, 3:25 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-20T15:25:40Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)

Stashbot added a comment.Mar 20 2024, 3:27 PM

Mentioned in SAL (#wikimedia-cloud) [2024-03-20T15:27:23Z] <dcaro> turning off cloudcephosd1030 to swap some disks (T348643)

Stashbot added a comment.Mar 20 2024, 3:39 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-20T15:39:22Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.ceph.osd.drain_node (T348643)

Stashbot added a comment.Mar 20 2024, 5:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-03-20T17:02:12Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)

dcaro updated the task description. (Show Details)Mar 21 2024, 11:04 AM
dcaro added a comment.Mar 21 2024, 11:06 AM

New hard drives offline uncorrectable values (cloudcephosd1030) are all 0:

root@cloudcephosd1030:~# for i in /dev/sd{f,g,h,i,j}; do echo "## $i"; smartctl -a /dev/sdj | grep Offline_Uncorrectable; done
## /dev/sdf
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
## /dev/sdg
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
## /dev/sdh
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
## /dev/sdi
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
## /dev/sdj
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
dcaro updated the task description. (Show Details)Mar 21 2024, 11:28 AM
dcaro updated the task description. (Show Details)Mar 22 2024, 10:24 AM
Stashbot added a comment.Mar 22 2024, 10:25 AM

Mentioned in SAL (#wikimedia-cloud) [2024-03-22T10:25:55Z] <dcaro> adding back cloudcephosd1034 to the pool after doing the performance tests (T348643)

dcaro added a comment.Mar 22 2024, 10:46 AM

TSR and performance tests sent to DELL, bringing all the hosts back online.

dcaro added a comment.Thu, Apr 18, 10:02 AM

So far manual tests on the hard drive have been unable to trigger the issue and increase the counter, tested running fio directly on the drives with different setting with spans from minutes to several days.

MoritzMuehlenhoff subscribed.Thu, Apr 18, 10:07 AM
Andrew reassigned this task from Andrew to dcaro.Tue, Apr 23, 7:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL