Page MenuHomePhabricator
Paste P13734

cloudcephosd100* osd logs between 04:50 and 05:00
ActivePublic

Authored by dcaro on Jan 12 2021, 10:01 AM.
Tags
None
Referenced Files
F33996498: cloudcephosd100* osd logs between 04:55 and 05:00
Jan 12 2021, 10:01 AM
Subscribers
None
===== NODE GROUP =====
(1) cloudcephosd1007.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.5.log:2021-01-12 00:04:55.011 7fa7a0dbf700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.5.log:2021-01-12 00:04:55.011 7fa7a0dbf700 4 rocksdb: [db/db_impl.cc:778]
===== NODE GROUP =====
(1) cloudcephosd1006.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.40.log:2021-01-12 00:04:50.310 7f6518768700 0 log_channel(cluster) log [DBG] : 3.788 deep-scrub ok
===== NODE GROUP =====
(1) cloudcephosd1005.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.32.log:2021-01-12 00:04:50.735 7fbdce4e3700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.32.log:2021-01-12 00:04:50.735 7fbdce4e3700 4 rocksdb: [db/db_impl.cc:778]
/var/log/ceph/ceph-osd.33.log:2021-01-12 00:04:57.527 7fccf98ed700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.33.log:2021-01-12 00:04:57.527 7fccf98ed700 4 rocksdb: [db/db_impl.cc:778]
===== NODE GROUP =====
(1) cloudcephosd1014.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.109.log:2021-01-12 00:04:51.072 7fe46db48700 0 log_channel(cluster) log [DBG] : 6.266 deep-scrub starts
/var/log/ceph/ceph-osd.109.log:2021-01-12 00:04:51.424 7fe469b40700 0 log_channel(cluster) log [DBG] : 6.266 deep-scrub ok
===== NODE GROUP =====
(1) cloudcephosd1011.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.85.log:2021-01-12 00:04:51.327 7fdf95269700 0 log_channel(cluster) log [DBG] : 3.96 deep-scrub ok
===== NODE GROUP =====
(1) cloudcephosd1010.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.712 7f08d03ce700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 30819 from recycle list
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.712 7f08d03ce700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #30836. Immutable memtables: 0.
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.712 7f08d13d0700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.718958) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.712 7f08d13d0700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 12281] Flushing memtable with next log file: 30836
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.712 7f08d13d0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409896719003, "job": 12281, "event": "flush_started", "num_memtables": 1, "num_entries": 362505, "num_deletes": 32205, "total_data_size": 253525901, "memory_usage": 260049448, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.712 7f08d13d0700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 12281] Level-0 flush table #30837: started
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.860 7f08d13d0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409896866258, "cf_name": "default", "job": 12281, "event": "table_file_creation", "file_number": 30837, "file_size": 12591615, "table_properties": {"data_size": 12161414, "index_size": 214226, "filter_size": 215109, "raw_key_size": 4110376, "raw_average_key_size": 47, "raw_value_size": 10590518, "raw_average_value_size": 123, "num_data_blocks": 2985, "num_entries": 86002, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.860 7f08d13d0700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 12281] Level-0 flush table #30837: 12591615 bytes OK
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.860 7f08d13d0700 4 rocksdb: [db/db_impl_files.cc:194] adding log 30821 to recycle list
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.860 7f08d13d0700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.866291) [db/memtable_list.cc:380] [default] Level-0 commit table #30837 started
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.860 7f08d13d0700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.866705) [db/memtable_list.cc:428] [default] Level-0 commit table #30837: memtable #1 done
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.860 7f08d13d0700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.866714) EVENT_LOG_v1 {"time_micros": 1610409896866709, "job": 12281, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 3, 31, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.72.log:2021-01-12 00:04:56.860 7f08d13d0700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.866739) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[1 3 31 0 0 0 0] max score 0.70
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.316 7fd120c2d700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 51774 from recycle list
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.316 7fd120c2d700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #51778. Immutable memtables: 0.
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.316 7fd121c2f700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.321546) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.316 7fd121c2f700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 20735] Flushing memtable with next log file: 51778
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.316 7fd121c2f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409896321578, "job": 20735, "event": "flush_started", "num_memtables": 1, "num_entries": 174056, "num_deletes": 22545, "total_data_size": 256764789, "memory_usage": 260053832, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.316 7fd121c2f700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 20735] Level-0 flush table #51779: started
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd121c2f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409896405772, "cf_name": "default", "job": 20735, "event": "table_file_creation", "file_number": 51779, "file_size": 7944637, "table_properties": {"data_size": 7676123, "index_size": 132795, "filter_size": 134853, "raw_key_size": 2680922, "raw_average_key_size": 49, "raw_value_size": 6685290, "raw_average_value_size": 124, "num_data_blocks": 1873, "num_entries": 53910, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd121c2f700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 20735] Level-0 flush table #51779: 7944637 bytes OK
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd121c2f700 4 rocksdb: [db/db_impl_files.cc:194] adding log 51776 to recycle list
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd121c2f700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.405861) [db/memtable_list.cc:380] [default] Level-0 commit table #51779 started
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd121c2f700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.406330) [db/memtable_list.cc:428] [default] Level-0 commit table #51779: memtable #1 done
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd121c2f700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.406339) EVENT_LOG_v1 {"time_micros": 1610409896406334, "job": 20735, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 4, 28, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd121c2f700 4 rocksdb: (Original Log Time 2021/01/12-00:04:56.406371) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[4 4 28 0 0 0 0] max score 1.00
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd122430700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 20736] Compacting 4@0 + 4@1 files to L1, score 1.00
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd122430700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 20735 Base level 0, inputs: [51779(7758KB) 51777(9785KB) 51775(7465KB) 51773(5879KB)], [51768(66MB) 51769(66MB) 51770(51MB) 51771(14MB)]
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.400 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409896406610, "job": 20736, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [51779, 51777, 51775, 51773], "files_L1": [51768, 51769, 51770, 51771], "score": 1, "input_data_size": 238769001}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.852 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20736] Generated table #51780: 253099 keys, 69299775 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:56.852 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409896858518, "cf_name": "default", "job": 20736, "event": "table_file_creation", "file_number": 51780, "file_size": 69299775, "table_properties": {"data_size": 67112688, "index_size": 1553452, "filter_size": 632773, "raw_key_size": 22778162, "raw_average_key_size": 89, "raw_value_size": 58327364, "raw_average_value_size": 230, "num_data_blocks": 16932, "num_entries": 253099, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:57.308 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20736] Generated table #51781: 243302 keys, 69316340 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:57.308 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409897314712, "cf_name": "default", "job": 20736, "event": "table_file_creation", "file_number": 51781, "file_size": 69316340, "table_properties": {"data_size": 67111830, "index_size": 1595323, "filter_size": 608325, "raw_key_size": 21314540, "raw_average_key_size": 87, "raw_value_size": 60622100, "raw_average_value_size": 249, "num_data_blocks": 16963, "num_entries": 243302, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:57.860 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20736] Generated table #51782: 630954 keys, 62217928 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:57.860 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409897866243, "cf_name": "default", "job": 20736, "event": "table_file_creation", "file_number": 51782, "file_size": 62217928, "table_properties": {"data_size": 59621812, "index_size": 1017842, "filter_size": 1577413, "raw_key_size": 32521767, "raw_average_key_size": 51, "raw_value_size": 49106700, "raw_average_value_size": 77, "num_data_blocks": 14810, "num_entries": 630954, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.160 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20736] Generated table #51783: 452371 keys, 15256808 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.160 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898165665, "cf_name": "default", "job": 20736, "event": "table_file_creation", "file_number": 51783, "file_size": 15256808, "table_properties": {"data_size": 13994108, "index_size": 130888, "filter_size": 1130949, "raw_key_size": 10525367, "raw_average_key_size": 23, "raw_value_size": 7670569, "raw_average_value_size": 16, "num_data_blocks": 3441, "num_entries": 452371, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.164 7fd122430700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 20736] Compacted 4@0 + 4@1 files to L1 => 216090851 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: (Original Log Time 2021/01/12-00:04:58.172140) [db/compaction_job.cc:768] [default] compacted to: files[0 4 28 0 0 0 0] max score 1.02, MB/sec: 135.7 rd, 122.8 wr, level 1, files in(4, 4) out(4) MB in(30.2, 197.5) out(206.1), read-write-amplify(14.4) write-amplify(6.8) OK, records in: 1670427, records dropped: 69202 output_compression: NoCompression
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: (Original Log Time 2021/01/12-00:04:58.172162) EVENT_LOG_v1 {"time_micros": 1610409898172153, "job": 20736, "event": "compaction_finished", "compaction_time_micros": 1759338, "compaction_time_cpu_micros": 1293045, "output_level": 1, "num_output_files": 4, "total_output_size": 216090851, "num_input_records": 1648928, "num_output_records": 1579726, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 28, 0, 0, 0, 0]}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172276, "job": 20736, "event": "table_file_deletion", "file_number": 51779}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172316, "job": 20736, "event": "table_file_deletion", "file_number": 51777}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172352, "job": 20736, "event": "table_file_deletion", "file_number": 51775}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172395, "job": 20736, "event": "table_file_deletion", "file_number": 51773}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172435, "job": 20736, "event": "table_file_deletion", "file_number": 51771}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172483, "job": 20736, "event": "table_file_deletion", "file_number": 51770}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172528, "job": 20736, "event": "table_file_deletion", "file_number": 51769}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172580, "job": 20736, "event": "table_file_deletion", "file_number": 51768}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 20737] Compacting 1@1 + 10@2 files to L2, score 1.02
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 20736 Base level 1, inputs: [51782(59MB)], [51714(65MB) 51715(65MB) 51716(65MB) 51717(65MB) 51718(65MB) 51719(65MB) 51720(65MB) 51721(65MB) 51722(65MB) 51723(64MB)]
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.168 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898172645, "job": 20737, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [51782], "files_L2": [51714, 51715, 51716, 51717, 51718, 51719, 51720, 51721, 51722, 51723], "score": 1.0151, "input_data_size": 751561848}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.532 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51784: 137695 keys, 69079381 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.532 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898538226, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51784, "file_size": 69079381, "table_properties": {"data_size": 67111196, "index_size": 1623064, "filter_size": 344261, "raw_key_size": 12060929, "raw_average_key_size": 87, "raw_value_size": 62903539, "raw_average_value_size": 456, "num_data_blocks": 17281, "num_entries": 137695, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.872 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51785: 137313 keys, 69080798 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:58.872 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898879222, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51785, "file_size": 69080798, "table_properties": {"data_size": 67110802, "index_size": 1625771, "filter_size": 343365, "raw_key_size": 12027604, "raw_average_key_size": 87, "raw_value_size": 62912216, "raw_average_value_size": 458, "num_data_blocks": 17259, "num_entries": 137313, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:59.232 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51786: 137762 keys, 69083460 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:59.232 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409899237215, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51786, "file_size": 69083460, "table_properties": {"data_size": 67112936, "index_size": 1625147, "filter_size": 344517, "raw_key_size": 12066906, "raw_average_key_size": 87, "raw_value_size": 62904697, "raw_average_value_size": 456, "num_data_blocks": 17255, "num_entries": 137762, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:59.560 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51787: 137407 keys, 69078762 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:59.560 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409899564798, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51787, "file_size": 69078762, "table_properties": {"data_size": 67111326, "index_size": 1622955, "filter_size": 343621, "raw_key_size": 12035889, "raw_average_key_size": 87, "raw_value_size": 62912795, "raw_average_value_size": 457, "num_data_blocks": 17246, "num_entries": 137407, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:59.884 7fd122430700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 20737] Generated table #51788: 137525 keys, 69081384 bytes
/var/log/ceph/ceph-osd.73.log:2021-01-12 00:04:59.884 7fd122430700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409899891029, "cf_name": "default", "job": 20737, "event": "table_file_creation", "file_number": 51788, "file_size": 69081384, "table_properties": {"data_size": 67111355, "index_size": 1625292, "filter_size": 343877, "raw_key_size": 12046127, "raw_average_key_size": 87, "raw_value_size": 62906153, "raw_average_value_size": 457, "num_data_blocks": 17265, "num_entries": 137525, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
===== NODE GROUP =====
(1) cloudcephosd1003.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.64.log:2021-01-12 00:04:50.695 7fc27351c700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.64.log:2021-01-12 00:04:50.695 7fc27351c700 4 rocksdb: [db/db_impl.cc:778]
/var/log/ceph/ceph-osd.65.log:2021-01-12 00:04:58.259 7f11bba34700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
/var/log/ceph/ceph-osd.65.log:2021-01-12 00:04:58.259 7f11bba34700 4 rocksdb: [db/db_impl.cc:778]
===== NODE GROUP =====
(6) cloudcephosd[1001-1002,1004,1009,1013,1015].eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
not found
===== NODE GROUP =====
(1) cloudcephosd1008.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.486 7fd268a4b700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 45796 from recycle list
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.486 7fd268a4b700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #45800. Immutable memtables: 0.
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.486 7fd269a4d700 4 rocksdb: (Original Log Time 2021/01/12-00:04:58.489131) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.486 7fd269a4d700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 17361] Flushing memtable with next log file: 45800
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.486 7fd269a4d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898489168, "job": 17361, "event": "flush_started", "num_memtables": 1, "num_entries": 224239, "num_deletes": 26921, "total_data_size": 255857463, "memory_usage": 260052744, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.486 7fd269a4d700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 17361] Level-0 flush table #45801: started
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.598 7fd269a4d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409898601857, "cf_name": "default", "job": 17361, "event": "table_file_creation", "file_number": 45801, "file_size": 10585439, "table_properties": {"data_size": 10232906, "index_size": 175214, "filter_size": 176453, "raw_key_size": 3436998, "raw_average_key_size": 48, "raw_value_size": 8937322, "raw_average_value_size": 126, "num_data_blocks": 2489, "num_entries": 70570, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.598 7fd269a4d700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 17361] Level-0 flush table #45801: 10585439 bytes OK
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.598 7fd269a4d700 4 rocksdb: [db/db_impl_files.cc:194] adding log 45798 to recycle list
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.598 7fd269a4d700 4 rocksdb: (Original Log Time 2021/01/12-00:04:58.601894) [db/memtable_list.cc:380] [default] Level-0 commit table #45801 started
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.598 7fd269a4d700 4 rocksdb: (Original Log Time 2021/01/12-00:04:58.602379) [db/memtable_list.cc:428] [default] Level-0 commit table #45801: memtable #1 done
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.598 7fd269a4d700 4 rocksdb: (Original Log Time 2021/01/12-00:04:58.602401) EVENT_LOG_v1 {"time_micros": 1610409898602390, "job": 17361, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 4, 33, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.11.log:2021-01-12 00:04:58.598 7fd269a4d700 4 rocksdb: (Original Log Time 2021/01/12-00:04:58.602468) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[3 4 33 0 0 0 0] max score 0.76
===== NODE GROUP =====
(1) cloudcephosd1012.eqiad.wmnet
----- OUTPUT of 'grep -i "2021-01...echo "not found"' -----
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.666 7f2f99802700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 44520 from recycle list
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.666 7f2f99802700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #44528. Immutable memtables: 0.
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.666 7f2f9a804700 4 rocksdb: (Original Log Time 2021/01/12-00:04:53.671814) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.666 7f2f9a804700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 17121] Flushing memtable with next log file: 44528
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.666 7f2f9a804700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409893671849, "job": 17121, "event": "flush_started", "num_memtables": 1, "num_entries": 206683, "num_deletes": 24815, "total_data_size": 256192638, "memory_usage": 260051968, "flush_reason": "Write Buffer Full"}
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.666 7f2f9a804700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 17121] Level-0 flush table #44529: started
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.762 7f2f9a804700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1610409893768975, "cf_name": "default", "job": 17121, "event": "table_file_creation", "file_number": 44529, "file_size": 9911737, "table_properties": {"data_size": 9577009, "index_size": 166113, "filter_size": 167749, "raw_key_size": 3227189, "raw_average_key_size": 48, "raw_value_size": 8346526, "raw_average_value_size": 124, "num_data_blocks": 2343, "num_entries": 67077, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.762 7f2f9a804700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 17121] Level-0 flush table #44529: 9911737 bytes OK
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.766 7f2f9a804700 4 rocksdb: [db/db_impl_files.cc:194] adding log 44526 to recycle list
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.766 7f2f9a804700 4 rocksdb: (Original Log Time 2021/01/12-00:04:53.769008) [db/memtable_list.cc:380] [default] Level-0 commit table #44529 started
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.766 7f2f9a804700 4 rocksdb: (Original Log Time 2021/01/12-00:04:53.770015) [db/memtable_list.cc:428] [default] Level-0 commit table #44529: memtable #1 done
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.766 7f2f9a804700 4 rocksdb: (Original Log Time 2021/01/12-00:04:53.770024) EVENT_LOG_v1 {"time_micros": 1610409893770020, "job": 17121, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 4, 32, 0, 0, 0, 0], "immutable_memtables": 0}
/var/log/ceph/ceph-osd.89.log:2021-01-12 00:04:53.766 7f2f9a804700 4 rocksdb: (Original Log Time 2021/01/12-00:04:53.770052) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[2 4 32 0 0 0 0] max score 0.79
================

Event Timeline

dcaro changed the title of this paste from cloudcephosd100* osd logs between 04:55 and 05:00 to cloudcephosd100* osd logs between 04:50 and 05:00.