Page MenuHomePhabricator
Paste P13735

cloudcephosd100* osd logs between 05:00 and 05:10
ActivePublic

Authored by dcaro on Jan 12 2021, 10:02 AM.
Tags
None
Referenced Files
F33996499: cloudcephosd100* osd logs between 05:00 and 05:10
Jan 12 2021, 10:02 AM
Subscribers
None
===== NODE GROUP =====
(1) cloudcephosd1007.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.6.log:2021-01-12 00:05:03.592 7f8480756700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.6.log:2021-01-12 00:05:03.592 7f8480756700 4 rocksdb: [db/db_impl.cc:778]
===== NODE GROUP =====
(1) cloudcephosd1014.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.073 7f027256d700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 24531 from recycle list
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.073 7f027256d700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #24538. Immutable memtables: 0.
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.073 7f027356f700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.076907) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.073 7f027356f700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 10201] Flushing memtable with next log file: 24538
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.073 7f027356f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409908076950, "job": 10201, "event": "flush_started", "num_memtables": 1, "num_entries": 221553, "num_deletes": 24893, "total_data_size": 255954532, "memory_usage": 260054456, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.073 7f027356f700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 10201] Level-0 flush table #24539: started
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.181 7f027356f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409908185881, "cf_name": "default", "job": 10201, "event": "table_file_creation", "file_number": 24539, "file_size": 10617668, "table_properties": {"data_size": 10258630, "index_size": 183255, "filter_size": 174917, "raw_key_size": 3393332, "raw_average_key_size": 48, "raw_value_size": 8965975, "raw_average_value_size": 128, "num_data_blocks": 2525, "num_entries": 69932, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.181 7f027356f700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 10201] Level-0 flush table #24539: 10617668 bytes OK
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.185 7f027356f700 4 rocksdb: [db/db_impl_files.cc:194] adding log 24533 to recycle list
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.185 7f027356f700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.185917) [db/memtable_list.cc:380] [default] Level-0 commit table #24539 started
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.185 7f027356f700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.186370) [db/memtable_list.cc:428] [default] Level-0 commit table #24539: memtable #1 done
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.185 7f027356f700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.186380) EVENT_LOG_v1 {"time_micros": 1610409908186375, "job": 10201, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 3, 20, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.104.log:2021-01-12 00:05:08.185 7f027356f700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.186403) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[1 3 20 0 0 0 0] max score 0.68
===== NODE GROUP =====
(1) cloudcephosd1010.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:00.248 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51789: 137596 keys, 69075866 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:00.248 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409900255534, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51789, "file_size": 69075866, "table_properties": {"data_size": 67111136, "index_size": 1619865, "filter_size": 344005, "raw_key_size": 12052479, "raw_average_key_size": 87, "raw_value_size": 62913987, "raw_average_value_size": 457, "num_data_blocks": 17256, "num_entries": 137596, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:00.572 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51790: 137591 keys, 69076934 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:00.572 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409900579857, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51790, "file_size": 69076934, "table_properties": {"data_size": 67110395, "index_size": 1621674, "filter_size": 344005, "raw_key_size": 12051992, "raw_average_key_size": 87, "raw_value_size": 62913491, "raw_average_value_size": 457, "num_data_blocks": 17251, "num_entries": 137591, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:00.924 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51791: 137582 keys, 69078430 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:00.924 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409900929361, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51791, "file_size": 69078430, "table_properties": {"data_size": 67110067, "index_size": 1623498, "filter_size": 344005, "raw_key_size": 12051247, "raw_average_key_size": 87, "raw_value_size": 62906583, "raw_average_value_size": 457, "num_data_blocks": 17263, "num_entries": 137582, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.300 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51792: 137400 keys, 69078664 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.300 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901306134, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51792, "file_size": 69078664, "table_properties": {"data_size": 67112970, "index_size": 1621213, "filter_size": 343621, "raw_key_size": 12035232, "raw_average_key_size": 87, "raw_value_size": 62919123, "raw_average_value_size": 457, "num_data_blocks": 17257, "num_entries": 137400, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.860 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51793: 309131 keys, 68598738 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.860 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901864367, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51793, "file_size": 68598738, "table_properties": {"data_size": 66725749, "index_size": 1099196, "filter_size": 772933, "raw_key_size": 16033238, "raw_average_key_size": 51, "raw_value_size": 60867732, "raw_average_value_size": 196, "num_data_blocks": 16613, "num_entries": 309131, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 20737] Compacted 1@1 + 10@2 files to L2 => 690312417 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: (Original Log Time 2021/01/12-00:05:01.878115) [db/compaction_job.cc:768] [default] compacted to: files[0 3 28 0 0 0 0] max score 0.68, MB/sec: 203.6 rd, 187.0 wr, level 2, files in(1, 10) out(10) MB in(59.3, 657.4) out(658.3), read-write-amplify(23.2) write-amplify(11.1) OK, records in: 2176074, records dropped: 629072 output_compression: NoCompression
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: (Original Log Time 2021/01/12-00:05:01.878139) EVENT_LOG_v1 {"time_micros": 1610409901878128, "job": 20737, "event": "compaction_finished", "compaction_time_micros": 3691750, "compaction_time_cpu_micros": 2213694, "output_level": 2, "num_output_files": 10, "total_output_size": 690312417, "num_input_records": 2176074, "num_output_records": 1547002, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 28, 0, 0, 0, 0]}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878272, "job": 20737, "event": "table_file_deletion", "file_number": 51782}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878311, "job": 20737, "event": "table_file_deletion", "file_number": 51723}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878356, "job": 20737, "event": "table_file_deletion", "file_number": 51722}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878399, "job": 20737, "event": "table_file_deletion", "file_number": 51721}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878441, "job": 20737, "event": "table_file_deletion", "file_number": 51720}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878481, "job": 20737, "event": "table_file_deletion", "file_number": 51719}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878532, "job": 20737, "event": "table_file_deletion", "file_number": 51718}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878575, "job": 20737, "event": "table_file_deletion", "file_number": 51717}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878622, "job": 20737, "event": "table_file_deletion", "file_number": 51716}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878688, "job": 20737, "event": "table_file_deletion", "file_number": 51715}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:05:01.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409901878734, "job": 20737, "event": "table_file_deletion", "file_number": 51714}
===== NODE GROUP =====
(1) cloudcephosd1005.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.34.log:2021-01-12 00:05:05.019 7f023c2d3700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.34.log:2021-01-12 00:05:05.019 7f023c2d3700 4 rocksdb: [db/db_impl.cc:778]
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.367 7f113827b700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 36489 from recycle list
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.367 7f113827b700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #36493. Immutable memtables: 0.
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.367 7f113927d700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.373957) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.367 7f113927d700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 14267] Flushing memtable with next log file: 36493
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.367 7f113927d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409908373992, "job": 14267, "event": "flush_started", "num_memtables": 1, "num_entries": 226949, "num_deletes": 26250, "total_data_size": 255836618, "memory_usage": 260053976, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.367 7f113927d700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 14267] Level-0 flush table #36494: started
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.471 7f113927d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409908476966, "cf_name": "default", "job": 14267, "event": "table_file_creation", "file_number": 36494, "file_size": 10730465, "table_properties": {"data_size": 10368526, "index_size": 181676, "filter_size": 179397, "raw_key_size": 3449300, "raw_average_key_size": 48, "raw_value_size": 9045536, "raw_average_value_size": 126, "num_data_blocks": 2530, "num_entries": 71713, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.471 7f113927d700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 14267] Level-0 flush table #36494: 10730465 bytes OK
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.471 7f113927d700 4 rocksdb: [db/db_impl_files.cc:194] adding log 36491 to recycle list
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.471 7f113927d700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.477009) [db/memtable_list.cc:380] [default] Level-0 commit table #36494 started
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.471 7f113927d700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.477418) [db/memtable_list.cc:428] [default] Level-0 commit table #36494: memtable #1 done
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.471 7f113927d700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.477427) EVENT_LOG_v1 {"time_micros": 1610409908477422, "job": 14267, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 5, 30, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.39.log:2021-01-12 00:05:08.471 7f113927d700 4 rocksdb: (Original Log Time 2021/01/12-00:05:08.477450) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[3 5 30 0 0 0 0] max score 0.98
===== NODE GROUP =====
(1) cloudcephosd1001.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.51.log:2021-01-12 00:05:04.911 7f7495d82700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.51.log:2021-01-12 00:05:04.911 7f7495d82700 4 rocksdb: [db/db_impl.cc:778]
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:05.191 7f6634f74700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:05.191 7f6634f74700 4 rocksdb: [db/db_impl.cc:778]
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.063 7f663b781700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 64325 from recycle list
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.063 7f663b781700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #64329. Immutable memtables: 0.
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.063 7f663c783700 4 rocksdb: (Original Log Time 2021/01/12-00:05:07.070088) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.063 7f663c783700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 27558] Flushing memtable with next log file: 64329
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.063 7f663c783700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409907070131, "job": 27558, "event": "flush_started", "num_memtables": 1, "num_entries": 254339, "num_deletes": 27499, "total_data_size": 255419262, "memory_usage": 260067440, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.063 7f663c783700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 27558] Level-0 flush table #64330: started
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.179 7f663c783700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409907182754, "cf_name": "default", "job": 27558, "event": "table_file_creation", "file_number": 64330, "file_size": 10771243, "table_properties": {"data_size": 10405162, "index_size": 180442, "filter_size": 184773, "raw_key_size": 3537659, "raw_average_key_size": 47, "raw_value_size": 9058483, "raw_average_value_size": 122, "num_data_blocks": 2545, "num_entries": 73858, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.179 7f663c783700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 27558] Level-0 flush table #64330: 10771243 bytes OK
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.179 7f663c783700 4 rocksdb: [db/db_impl_files.cc:194] adding log 64327 to recycle list
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.179 7f663c783700 4 rocksdb: (Original Log Time 2021/01/12-00:05:07.182789) [db/memtable_list.cc:380] [default] Level-0 commit table #64330 started
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.179 7f663c783700 4 rocksdb: (Original Log Time 2021/01/12-00:05:07.183171) [db/memtable_list.cc:428] [default] Level-0 commit table #64330: memtable #1 done
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.179 7f663c783700 4 rocksdb: (Original Log Time 2021/01/12-00:05:07.183180) EVENT_LOG_v1 {"time_micros": 1610409907183175, "job": 27558, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 3, 26, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.52.log:2021-01-12 00:05:07.179 7f663c783700 4 rocksdb: (Original Log Time 2021/01/12-00:05:07.183200) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[3 3 26 0 0 0 0] max score 0.75
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.795 7f88401c9700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 96426 from recycle list
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.795 7f88401c9700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #96430. Immutable memtables: 0.
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.795 7f88411cb700 4 rocksdb: (Original Log Time 2021/01/12-00:05:02.799773) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.795 7f88411cb700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 38618] Flushing memtable with next log file: 96430
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.795 7f88411cb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409902799806, "job": 38618, "event": "flush_started", "num_memtables": 1, "num_entries": 240999, "num_deletes": 27539, "total_data_size": 255593589, "memory_usage": 260058840, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.795 7f88411cb700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 38618] Level-0 flush table #96431: started
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88411cb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409902911476, "cf_name": "default", "job": 38618, "event": "table_file_creation", "file_number": 96431, "file_size": 11439156, "table_properties": {"data_size": 11058008, "index_size": 191029, "filter_size": 189253, "raw_key_size": 3634575, "raw_average_key_size": 48, "raw_value_size": 9665374, "raw_average_value_size": 127, "num_data_blocks": 2700, "num_entries": 75649, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88411cb700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 38618] Level-0 flush table #96431: 11439156 bytes OK
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88411cb700 4 rocksdb: [db/db_impl_files.cc:194] adding log 96428 to recycle list
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88411cb700 4 rocksdb: (Original Log Time 2021/01/12-00:05:02.911512) [db/memtable_list.cc:380] [default] Level-0 commit table #96431 started
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88411cb700 4 rocksdb: (Original Log Time 2021/01/12-00:05:02.911895) [db/memtable_list.cc:428] [default] Level-0 commit table #96431: memtable #1 done
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88411cb700 4 rocksdb: (Original Log Time 2021/01/12-00:05:02.911903) EVENT_LOG_v1 {"time_micros": 1610409902911899, "job": 38618, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 4, 30, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88411cb700 4 rocksdb: (Original Log Time 2021/01/12-00:05:02.911925) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[4 4 30 0 0 0 0] max score 1.00
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88421cd700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 38619] Compacting 4@0 + 4@1 files to L1, score 1.00
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88421cd700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 38618 Base level 0, inputs: [96431(10MB) 96429(11MB) 96427(8983KB) 96425(9587KB)], [96420(66MB) 96421(66MB) 96422(24MB) 96423(26MB)]
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:02.907 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409902912044, "job": 38619, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [96431, 96429, 96427, 96425], "files_L1": [96420, 96421, 96422, 96423], "score": 1, "input_data_size": 235444010}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:03.351 7f88421cd700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 38619] Generated table #96432: 264050 keys, 69305772 bytes
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:03.351 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409903356905, "cf_name": "default", "job": 38619, "event": "table_file_creation", "file_number": 96432, "file_size": 69305772, "table_properties": {"data_size": 67111443, "index_size": 1533302, "filter_size": 660165, "raw_key_size": 23917732, "raw_average_key_size": 90, "raw_value_size": 57604731, "raw_average_value_size": 218, "num_data_blocks": 16865, "num_entries": 264050, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:03.755 7f88421cd700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 38619] Generated table #96433: 224150 keys, 69251496 bytes
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:03.755 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409903762683, "cf_name": "default", "job": 38619, "event": "table_file_creation", "file_number": 96433, "file_size": 69251496, "table_properties": {"data_size": 67112254, "index_size": 1577927, "filter_size": 560453, "raw_key_size": 19636297, "raw_average_key_size": 87, "raw_value_size": 61031626, "raw_average_value_size": 272, "num_data_blocks": 16851, "num_entries": 224150, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:03.923 7f88421cd700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 38619] Generated table #96434: 98182 keys, 28313043 bytes
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:03.923 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409903930371, "cf_name": "default", "job": 38619, "event": "table_file_creation", "file_number": 96434, "file_size": 28313043, "table_properties": {"data_size": 27418791, "index_size": 647818, "filter_size": 245573, "raw_key_size": 8602727, "raw_average_key_size": 87, "raw_value_size": 24798329, "raw_average_value_size": 252, "num_data_blocks": 6894, "num_entries": 98182, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.471 7f88421cd700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 38619] Generated table #96435: 515117 keys, 47701702 bytes
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.471 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904478272, "cf_name": "default", "job": 38619, "event": "table_file_creation", "file_number": 96435, "file_size": 47701702, "table_properties": {"data_size": 45726963, "index_size": 685999, "filter_size": 1287877, "raw_key_size": 22136453, "raw_average_key_size": 42, "raw_value_size": 37503922, "raw_average_value_size": 72, "num_data_blocks": 11310, "num_entries": 515117, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.475 7f88421cd700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 38619] Compacted 4@0 + 4@1 files to L1 => 214572013 bytes
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.475 7f88421cd700 4 rocksdb: (Original Log Time 2021/01/12-00:05:04.482498) [db/compaction_job.cc:768] [default] compacted to: files[0 4 30 0 0 0 0] max score 0.80, MB/sec: 150.3 rd, 137.0 wr, level 1, files in(4, 4) out(4) MB in(40.8, 183.7) out(204.6), read-write-amplify(10.5) write-amplify(5.0) OK, records in: 1180049, records dropped: 42873 output_compression: NoCompression
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.475 7f88421cd700 4 rocksdb: (Original Log Time 2021/01/12-00:05:04.482517) EVENT_LOG_v1 {"time_micros": 1610409904482508, "job": 38619, "event": "compaction_finished", "compaction_time_micros": 1566444, "compaction_time_cpu_micros": 1064378, "output_level": 1, "num_output_files": 4, "total_output_size": 214572013, "num_input_records": 1144372, "num_output_records": 1101499, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 30, 0, 0, 0, 0]}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.475 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482614, "job": 38619, "event": "table_file_deletion", "file_number": 96431}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.475 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482653, "job": 38619, "event": "table_file_deletion", "file_number": 96429}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.475 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482713, "job": 38619, "event": "table_file_deletion", "file_number": 96427}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.475 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482752, "job": 38619, "event": "table_file_deletion", "file_number": 96425}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.479 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482806, "job": 38619, "event": "table_file_deletion", "file_number": 96423}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.479 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482840, "job": 38619, "event": "table_file_deletion", "file_number": 96422}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.479 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482877, "job": 38619, "event": "table_file_deletion", "file_number": 96421}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.479 7f88421cd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904482913, "job": 38619, "event": "table_file_deletion", "file_number": 96420}
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.991 7f88399bc700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.53.log:2021-01-12 00:05:04.991 7f88399bc700 4 rocksdb: [db/db_impl.cc:778]
/var/log/ceph/ceph-osd.54.log:2021-01-12 00:05:05.223 7ff71bc50700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.54.log:2021-01-12 00:05:05.223 7ff71bc50700 4 rocksdb: [db/db_impl.cc:778]
/var/log/ceph/ceph-osd.55.log:2021-01-12 00:05:05.155 7f8ff9cc7700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.55.log:2021-01-12 00:05:05.155 7f8ff9cc7700 4 rocksdb: [db/db_impl.cc:778]
===== NODE GROUP =====
(1) cloudcephosd1012.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.078 7fd226e09700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 38378 from recycle list
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.078 7fd226e09700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #38382. Immutable memtables: 0.
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.078 7fd227e0b700 4 rocksdb: (Original Log Time 2021/01/12-00:05:05.082747) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.078 7fd227e0b700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 15297] Flushing memtable with next log file: 38382
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.078 7fd227e0b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905082782, "job": 15297, "event": "flush_started", "num_memtables": 1, "num_entries": 210288, "num_deletes": 24904, "total_data_size": 256121163, "memory_usage": 260051472, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.078 7fd227e0b700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 15297] Level-0 flush table #38383: started
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.170 7fd227e0b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905176903, "cf_name": "default", "job": 15297, "event": "table_file_creation", "file_number": 38383, "file_size": 9636798, "table_properties": {"data_size": 9312606, "index_size": 162361, "filter_size": 160965, "raw_key_size": 3136683, "raw_average_key_size": 48, "raw_value_size": 8129152, "raw_average_value_size": 126, "num_data_blocks": 2278, "num_entries": 64362, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.170 7fd227e0b700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 15297] Level-0 flush table #38383: 9636798 bytes OK
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.170 7fd227e0b700 4 rocksdb: [db/db_impl_files.cc:194] adding log 38380 to recycle list
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.170 7fd227e0b700 4 rocksdb: (Original Log Time 2021/01/12-00:05:05.176966) [db/memtable_list.cc:380] [default] Level-0 commit table #38383 started
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.170 7fd227e0b700 4 rocksdb: (Original Log Time 2021/01/12-00:05:05.177413) [db/memtable_list.cc:428] [default] Level-0 commit table #38383: memtable #1 done
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.170 7fd227e0b700 4 rocksdb: (Original Log Time 2021/01/12-00:05:05.177422) EVENT_LOG_v1 {"time_micros": 1610409905177418, "job": 15297, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 4, 28, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.170 7fd227e0b700 4 rocksdb: (Original Log Time 2021/01/12-00:05:05.177448) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[4 4 28 0 0 0 0] max score 1.00
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.174 7fd22860c700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 15298] Compacting 4@0 + 4@1 files to L1, score 1.00
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.174 7fd22860c700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 15297 Base level 0, inputs: [38383(9410KB) 38381(10MB) 38379(8375KB) 38377(7985KB)], [38372(66MB) 38373(66MB) 38374(37MB) 38375(16MB)]
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.174 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905177649, "job": 15298, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [38383, 38381, 38379, 38377], "files_L1": [38372, 38373, 38374, 38375], "score": 1, "input_data_size": 232727825}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.638 7fd22860c700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 15298] Generated table #38384: 265080 keys, 69320788 bytes
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:05.638 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905645335, "cf_name": "default", "job": 15298, "event": "table_file_creation", "file_number": 38384, "file_size": 69320788, "table_properties": {"data_size": 67112449, "index_size": 1544752, "filter_size": 662725, "raw_key_size": 23950081, "raw_average_key_size": 90, "raw_value_size": 57718777, "raw_average_value_size": 217, "num_data_blocks": 16902, "num_entries": 265080, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.042 7fd22860c700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 15298] Generated table #38385: 204389 keys, 69208769 bytes
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.042 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906048199, "cf_name": "default", "job": 15298, "event": "table_file_creation", "file_number": 38385, "file_size": 69208769, "table_properties": {"data_size": 67111420, "index_size": 1585442, "filter_size": 511045, "raw_key_size": 17904048, "raw_average_key_size": 87, "raw_value_size": 61437112, "raw_average_value_size": 300, "num_data_blocks": 16945, "num_entries": 204389, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.522 7fd22860c700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 15298] Generated table #38386: 499193 keys, 53393387 bytes
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.522 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906525708, "cf_name": "default", "job": 15298, "event": "table_file_creation", "file_number": 38386, "file_size": 53393387, "table_properties": {"data_size": 51289325, "index_size": 855132, "filter_size": 1248069, "raw_key_size": 24958376, "raw_average_key_size": 49, "raw_value_size": 42971786, "raw_average_value_size": 86, "num_data_blocks": 12727, "num_entries": 499193, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.882 7fd22860c700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 15298] Generated table #38387: 595077 keys, 17476015 bytes
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.882 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906886810, "cf_name": "default", "job": 15298, "event": "table_file_creation", "file_number": 38387, "file_size": 17476015, "table_properties": {"data_size": 15870152, "index_size": 117187, "filter_size": 1487813, "raw_key_size": 10711386, "raw_average_key_size": 18, "raw_value_size": 7813209, "raw_average_value_size": 13, "num_data_blocks": 3894, "num_entries": 595077, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 15298] Compacted 4@0 + 4@1 files to L1 => 209398959 bytes
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: (Original Log Time 2021/01/12-00:05:06.891650) [db/compaction_job.cc:768] [default] compacted to: files[0 4 28 0 0 0 0] max score 0.83, MB/sec: 136.1 rd, 122.5 wr, level 1, files in(4, 4) out(4) MB in(35.7, 186.2) out(199.7), read-write-amplify(11.8) write-amplify(5.6) OK, records in: 1657006, records dropped: 67149 output_compression: NoCompression
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: (Original Log Time 2021/01/12-00:05:06.891694) EVENT_LOG_v1 {"time_micros": 1610409906891675, "job": 15298, "event": "compaction_finished", "compaction_time_micros": 1709659, "compaction_time_cpu_micros": 1178820, "output_level": 1, "num_output_files": 4, "total_output_size": 209398959, "num_input_records": 1630888, "num_output_records": 1563739, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 28, 0, 0, 0, 0]}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906891883, "job": 15298, "event": "table_file_deletion", "file_number": 38383}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906891955, "job": 15298, "event": "table_file_deletion", "file_number": 38381}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906892026, "job": 15298, "event": "table_file_deletion", "file_number": 38379}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906892060, "job": 15298, "event": "table_file_deletion", "file_number": 38377}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906892098, "job": 15298, "event": "table_file_deletion", "file_number": 38375}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906892135, "job": 15298, "event": "table_file_deletion", "file_number": 38374}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906892180, "job": 15298, "event": "table_file_deletion", "file_number": 38373}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:06.886 7fd22860c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409906892217, "job": 15298, "event": "table_file_deletion", "file_number": 38372}
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:08.982 7fd216de9700 0 log_channel(cluster) log [DBG] : 3.ec2 scrub starts
/var/log/ceph/ceph-osd.91.log:2021-01-12 00:05:09.530 7fd216de9700 0 log_channel(cluster) log [DBG] : 3.ec2 scrub ok
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.294 7fef5d2a0700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 45249 from recycle list
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.294 7fef5d2a0700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #45253. Immutable memtables: 0.
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.294 7fef5e2a2700 4 rocksdb: (Original Log Time 2021/01/12-00:05:04.298159) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.294 7fef5e2a2700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 16830] Flushing memtable with next log file: 45253
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.294 7fef5e2a2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904298194, "job": 16830, "event": "flush_started", "num_memtables": 1, "num_entries": 220206, "num_deletes": 26200, "total_data_size": 255939542, "memory_usage": 260056240, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.294 7fef5e2a2700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 16830] Level-0 flush table #45254: started
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5e2a2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904400399, "cf_name": "default", "job": 16830, "event": "table_file_creation", "file_number": 45254, "file_size": 11366586, "table_properties": {"data_size": 10997959, "index_size": 191436, "filter_size": 176325, "raw_key_size": 3510122, "raw_average_key_size": 49, "raw_value_size": 9672042, "raw_average_value_size": 137, "num_data_blocks": 2664, "num_entries": 70520, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5e2a2700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 16830] Level-0 flush table #45254: 11366586 bytes OK
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5e2a2700 4 rocksdb: [db/db_impl_files.cc:194] adding log 45251 to recycle list
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5e2a2700 4 rocksdb: (Original Log Time 2021/01/12-00:05:04.400434) [db/memtable_list.cc:380] [default] Level-0 commit table #45254 started
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5e2a2700 4 rocksdb: (Original Log Time 2021/01/12-00:05:04.400862) [db/memtable_list.cc:428] [default] Level-0 commit table #45254: memtable #1 done
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5e2a2700 4 rocksdb: (Original Log Time 2021/01/12-00:05:04.400882) EVENT_LOG_v1 {"time_micros": 1610409904400873, "job": 16830, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 3, 35, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5e2a2700 4 rocksdb: (Original Log Time 2021/01/12-00:05:04.400928) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[4 3 35 0 0 0 0] max score 1.00
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5eaa3700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 16831] Compacting 4@0 + 3@1 files to L1, score 1.00
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5eaa3700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 16830 Base level 0, inputs: [45254(10MB) 45252(11MB) 45250(10MB) 45248(9918KB)], [45232(66MB) 45233(66MB) 45234(38MB)]
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.394 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904401165, "job": 16831, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [45254, 45252, 45250, 45248], "files_L1": [45232, 45233, 45234], "score": 1, "input_data_size": 223045614}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.854 7fef5eaa3700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 16831] Generated table #45255: 219795 keys, 69234849 bytes
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:04.854 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409904861199, "cf_name": "default", "job": 16831, "event": "table_file_creation", "file_number": 45255, "file_size": 69234849, "table_properties": {"data_size": 67111940, "index_size": 1572474, "filter_size": 549573, "raw_key_size": 19414266, "raw_average_key_size": 88, "raw_value_size": 60679902, "raw_average_value_size": 276, "num_data_blocks": 16909, "num_entries": 219795, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.278 7fef5eaa3700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 16831] Generated table #45256: 223748 keys, 69270365 bytes
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.278 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905282837, "cf_name": "default", "job": 16831, "event": "table_file_creation", "file_number": 45256, "file_size": 69270365, "table_properties": {"data_size": 67109644, "index_size": 1600430, "filter_size": 559429, "raw_key_size": 19604303, "raw_average_key_size": 87, "raw_value_size": 61074234, "raw_average_value_size": 272, "num_data_blocks": 16956, "num_entries": 223748, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.546 7fef5eaa3700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 16831] Generated table #45257: 133699 keys, 42549073 bytes
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.546 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905550332, "cf_name": "default", "job": 16831, "event": "table_file_creation", "file_number": 45257, "file_size": 42549073, "table_properties": {"data_size": 41231293, "index_size": 982642, "filter_size": 334277, "raw_key_size": 11713062, "raw_average_key_size": 87, "raw_value_size": 37591680, "raw_average_value_size": 281, "num_data_blocks": 10396, "num_entries": 133699, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.758 7fef5eaa3700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 16831] Generated table #45258: 234560 keys, 25510469 bytes
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.758 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905763130, "cf_name": "default", "job": 16831, "event": "table_file_creation", "file_number": 45258, "file_size": 25510469, "table_properties": {"data_size": 24537111, "index_size": 386058, "filter_size": 586437, "raw_key_size": 10805921, "raw_average_key_size": 46, "raw_value_size": 20652568, "raw_average_value_size": 88, "num_data_blocks": 6075, "num_entries": 234560, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 16831] Compacted 4@0 + 3@1 files to L1 => 206564756 bytes
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: (Original Log Time 2021/01/12-00:05:05.767894) [db/compaction_job.cc:768] [default] compacted to: files[0 4 35 0 0 0 0] max score 0.85, MB/sec: 163.7 rd, 151.6 wr, level 1, files in(4, 3) out(4) MB in(41.7, 171.0) out(197.0), read-write-amplify(9.8) write-amplify(4.7) OK, records in: 848001, records dropped: 27552 output_compression: NoCompression
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: (Original Log Time 2021/01/12-00:05:05.767916) EVENT_LOG_v1 {"time_micros": 1610409905767906, "job": 16831, "event": "compaction_finished", "compaction_time_micros": 1362198, "compaction_time_cpu_micros": 819912, "output_level": 1, "num_output_files": 4, "total_output_size": 206564756, "num_input_records": 839354, "num_output_records": 811802, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 35, 0, 0, 0, 0]}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905768043, "job": 16831, "event": "table_file_deletion", "file_number": 45254}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905768078, "job": 16831, "event": "table_file_deletion", "file_number": 45252}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905768117, "job": 16831, "event": "table_file_deletion", "file_number": 45250}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905768183, "job": 16831, "event": "table_file_deletion", "file_number": 45248}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905768228, "job": 16831, "event": "table_file_deletion", "file_number": 45234}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905768282, "job": 16831, "event": "table_file_deletion", "file_number": 45233}
/var/log/ceph/ceph-osd.95.log:2021-01-12 00:05:05.762 7fef5eaa3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409905768338, "job": 16831, "event": "table_file_deletion", "file_number": 45232}
===== NODE GROUP =====
(1) cloudcephosd1003.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.66.log:2021-01-12 00:05:05.627 7f16e05f3700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.66.log:2021-01-12 00:05:05.627 7f16e05f3700 4 rocksdb: [db/db_impl.cc:778]
===== NODE GROUP =====
(8) cloudcephosd[1002,1004,1006,1008-1009,1011,1013,1015].eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
not found
================