Page MenuHomePhabricator
Create Task
Maniphest T180714

s5 primary master db1063 crashed
Closed, ResolvedPublic
Actions

Description

At Thu Nov 16 17:15 server db1063, master of the s5 shard of databases (dewiki and wikidata) started to have storage crashes (T180714#3767308)

Update Fri Nov 17 07:55:06 UTC 2017:

Pending things:

  • Revert schema change on codfw [DONE on master - replication will bring it to the slaves]
  • move dbstore1001 under the new master (db1070) [DONE]
  • revert schema change on dbstore1001 [DONE]
  • move dbstore1002 under the new master (db1070) [DONE]
  • revert schema change on dbstore1002 [ONGOING]
  • decide what to do with db1063 (definitely not a master for anything else, we might rebuild it as vslow from db1071 for example) as it is broken: https://phabricator.wikimedia.org/P6337
  • db1071 moved under db1070

IR document: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171116-s5-dewiki-wikidata

Details

SubjectRepoBranchLines +/-
install_server: Reimage db1063operations/puppetproduction+1 -1
db-eqiad.php: Restore original weight for s5 hostsoperations/mediawiki-configmaster+2 -2
db-eqiad.php: Increase weight for some hostsoperations/mediawiki-configmaster+3 -3
db-eqiad.php: Pool db1100 as vslowoperations/mediawiki-configmaster+2 -2
db-eqiad.php: Start repooling all the hostsoperations/mediawiki-configmaster+21 -15
s5.hosts: db1070 is now the masteroperations/softwaremaster+1 -1
mariadb: Depool fully all delayed serversoperations/mediawiki-configmaster+4 -4
mariadb: Pool only the servers with working replicationoperations/mediawiki-configmaster+10 -16
mariadb: Failover db1063 to db1070operations/mediawiki-configmaster+2 -2
db1070: New master in s5operations/puppetproduction+4 -5
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAddshoreT173818 [Epic] Kill the Wikidata build step
 ResolvedRelease• demonT178635 1.31.0-wmf.8 deployment blockers
 ResolvedNoneT180714 s5 primary master db1063 crashed

Event Timeline

jcrespo created this task.Nov 16 2017, 5:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2017, 5:32 PM
gerritbot subscribed.Nov 16 2017, 5:32 PM

Change 391869 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Failover db1063 to db1070

https://gerrit.wikimedia.org/r/391869

gerritbot added a project: Patch-For-Review.Nov 16 2017, 5:32 PM
fgiunchedi subscribed.Nov 16 2017, 5:36 PM

I retrieved the kernel logs from syslog servers at the time of the incident:

P6332 Masterwork From Distant Lands
1Nov 16 16:51:26 db1063 dbus[1366]: [system] Successfully activated service 'org.freedesktop.timedate1'
2Nov 16 16:51:26 db1063 systemd-timedated[40971]: /etc/localtime should be a symbolic link to a time zone data file in /usr/share/zoneinfo/.
3Nov 16 16:51:26 db1063 systemd[1]: Started Time & Date Service.
4Nov 16 17:14:11 db1063 dbus[1366]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
5Nov 16 17:14:11 db1063 systemd[1]: Starting Time & Date Service...
6Nov 16 17:14:11 db1063 dbus[1366]: [system] Successfully activated service 'org.freedesktop.timedate1'
7Nov 16 17:14:11 db1063 systemd-timedated[46875]: /etc/localtime should be a symbolic link to a time zone data file in /usr/share/zoneinfo/.
8Nov 16 17:14:11 db1063 systemd[1]: Started Time & Date Service.
9Nov 16 17:15:23 db1063 kernel: [17198753.219192] megaraid_sas 0000:03:00.0: Iop2SysDoorbellIntfor scsi0
10Nov 16 17:15:24 db1063 kernel: [17198754.220718] megaraid_sas 0000:03:00.0: Found FW in FAULT state, will reset adapter scsi0.
11Nov 16 17:15:24 db1063 kernel: [17198754.220724] megaraid_sas 0000:03:00.0: resetting fusion adapter scsi0.
12Nov 16 17:15:59 db1063 kernel: [17198789.356976] TCP: request_sock_TCP: Possible SYN flooding on port 5666. Sending cookies. Check SNMP counters.
13Nov 16 17:17:12 db1063 kernel: [17198862.019872] megaraid_sas 0000:03:00.0: Diag reset adapter never cleared megasas_adp_reset_fusion 2548
14Nov 16 17:18:34 db1063 kernel: [17198943.817263] INFO: task systemd:1 blocked for more than 120 seconds.
15Nov 16 17:18:34 db1063 kernel: [17198943.824566] Not tainted 4.4.0-3-amd64 #1
16Nov 16 17:18:34 db1063 kernel: [17198943.829828] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
17Nov 16 17:18:34 db1063 kernel: [17198943.838866] systemd D 0000000000000004 0 1 0 0x00000000
18Nov 16 17:18:34 db1063 kernel: [17198943.838874] ffff880ff9356d80 ffff880ff500f180 ffff880ff9378000 ffff880ff9377860
19Nov 16 17:18:34 db1063 kernel: [17198943.838879] 7fffffffffffffff ffffffff81594060 ffff880ff93778e0 ffff88090bf9e1c8
20Nov 16 17:18:34 db1063 kernel: [17198943.838882] ffffffff815937e1 0000000000000000 ffffffff81596795 7fffffffffffffff
21Nov 16 17:18:34 db1063 kernel: [17198943.838886] Call Trace:
22Nov 16 17:18:34 db1063 kernel: [17198943.838901] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
23Nov 16 17:18:34 db1063 kernel: [17198943.838905] [<ffffffff815937e1>] ? schedule+0x31/0x80
24Nov 16 17:18:34 db1063 kernel: [17198943.838921] [<ffffffff81596795>] ? schedule_timeout+0x235/0x2d0
25Nov 16 17:18:34 db1063 kernel: [17198943.838941] [<ffffffff812bf4db>] ? queue_unplugged+0x9b/0xa0
26Nov 16 17:18:34 db1063 kernel: [17198943.838948] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
27Nov 16 17:18:34 db1063 kernel: [17198943.838952] [<ffffffff81592dd4>] ? io_schedule_timeout+0xb4/0x130
28Nov 16 17:18:34 db1063 kernel: [17198943.838960] [<ffffffff810b8305>] ? prepare_to_wait+0x55/0x80
29Nov 16 17:18:34 db1063 kernel: [17198943.838972] [<ffffffff81594077>] ? bit_wait_io+0x17/0x60
30Nov 16 17:18:34 db1063 kernel: [17198943.838975] [<ffffffff81593b5a>] ? __wait_on_bit+0x5a/0x90
31Nov 16 17:18:34 db1063 kernel: [17198943.838979] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
32Nov 16 17:18:34 db1063 kernel: [17198943.838982] [<ffffffff81593c0e>] ? out_of_line_wait_on_bit+0x7e/0xa0
33Nov 16 17:18:34 db1063 kernel: [17198943.838986] [<ffffffff810b8670>] ? autoremove_wake_function+0x40/0x40
34Nov 16 17:18:34 db1063 kernel: [17198943.838995] [<ffffffff81213553>] ? bh_submit_read+0x73/0x80
35Nov 16 17:18:34 db1063 kernel: [17198943.839033] [<ffffffffa02751d5>] ? ext4_get_branch+0xc5/0x130 [ext4]
36Nov 16 17:18:34 db1063 kernel: [17198943.839060] [<ffffffffa0275c97>] ? ext4_ind_map_blocks+0xb7/0xaf0 [ext4]
37Nov 16 17:18:34 db1063 kernel: [17198943.839076] [<ffffffff81188fc5>] ? zone_statistics+0x75/0x80
38Nov 16 17:18:34 db1063 kernel: [17198943.839082] [<ffffffff81172596>] ? get_page_from_freelist+0x4a6/0xa10
39Nov 16 17:18:34 db1063 kernel: [17198943.839094] [<ffffffff81188fc5>] ? zone_statistics+0x75/0x80
40Nov 16 17:18:34 db1063 kernel: [17198943.839114] [<ffffffffa0236c04>] ? ext4_map_blocks+0x1f4/0x4e0 [ext4]
41Nov 16 17:18:34 db1063 kernel: [17198943.839122] [<ffffffff8116aaab>] ? __add_to_page_cache_locked+0x16b/0x1e0
42Nov 16 17:18:34 db1063 kernel: [17198943.839128] [<ffffffff811937a4>] ? workingset_refault+0x54/0xa0
43Nov 16 17:18:34 db1063 kernel: [17198943.839148] [<ffffffffa027fe78>] ? ext4_mpage_readpages+0x4a8/0x810 [ext4]
44Nov 16 17:18:34 db1063 kernel: [17198943.839156] [<ffffffff811b85fa>] ? alloc_pages_current+0x8a/0x110
45Nov 16 17:18:34 db1063 kernel: [17198943.839161] [<ffffffff81177c60>] ? __do_page_cache_readahead+0x1a0/0x220
46Nov 16 17:18:34 db1063 kernel: [17198943.839165] [<ffffffff8116abe7>] ? pagecache_get_page+0x27/0x1a0
47Nov 16 17:18:34 db1063 kernel: [17198943.839168] [<ffffffff8116c601>] ? filemap_fault+0x381/0x400
48Nov 16 17:18:34 db1063 kernel: [17198943.839185] [<ffffffffa023d7a1>] ? ext4_filemap_fault+0x31/0x50 [ext4]
49Nov 16 17:18:34 db1063 kernel: [17198943.839190] [<ffffffff811958ff>] ? __do_fault+0x4f/0xe0
50Nov 16 17:18:34 db1063 kernel: [17198943.839195] [<ffffffff8119a2f3>] ? handle_mm_fault+0x1053/0x1840
51Nov 16 17:18:34 db1063 kernel: [17198943.839201] [<ffffffff811dcf81>] ? __fput+0x161/0x1d0
52Nov 16 17:18:34 db1063 kernel: [17198943.839208] [<ffffffff81063711>] ? __do_page_fault+0x1a1/0x440
53Nov 16 17:18:34 db1063 kernel: [17198943.839213] [<ffffffff81599778>] ? page_fault+0x28/0x30
54Nov 16 17:18:34 db1063 kernel: [17198943.839238] INFO: task jbd2/sda1-8:740 blocked for more than 120 seconds.
55Nov 16 17:18:34 db1063 kernel: [17198943.847114] Not tainted 4.4.0-3-amd64 #1
56Nov 16 17:18:34 db1063 kernel: [17198943.852365] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
57Nov 16 17:18:34 db1063 kernel: [17198943.861403] jbd2/sda1-8 D ffff881fff215d80 0 740 2 0x00000000
58Nov 16 17:18:34 db1063 kernel: [17198943.861408] ffff881ff562cfc0 ffff880ff8f18180 ffff881ff579c000 ffff881ff579bb00
59Nov 16 17:18:34 db1063 kernel: [17198943.861411] 7fffffffffffffff ffffffff81594060 ffff881ff579bb80 0000000000000908
60Nov 16 17:18:34 db1063 kernel: [17198943.861415] ffffffff815937e1 0000000000000000 ffffffff81596795 7fffffffffffffff
61Nov 16 17:18:34 db1063 kernel: [17198943.861418] Call Trace:
62Nov 16 17:18:34 db1063 kernel: [17198943.861422] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
63Nov 16 17:18:34 db1063 kernel: [17198943.861425] [<ffffffff815937e1>] ? schedule+0x31/0x80
64Nov 16 17:18:34 db1063 kernel: [17198943.861442] [<ffffffff81596795>] ? schedule_timeout+0x235/0x2d0
65Nov 16 17:18:34 db1063 kernel: [17198943.861445] [<ffffffff812bf4db>] ? queue_unplugged+0x9b/0xa0
66Nov 16 17:18:34 db1063 kernel: [17198943.861449] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
67Nov 16 17:18:34 db1063 kernel: [17198943.861452] [<ffffffff81592dd4>] ? io_schedule_timeout+0xb4/0x130
68Nov 16 17:18:34 db1063 kernel: [17198943.861455] [<ffffffff810b8305>] ? prepare_to_wait+0x55/0x80
69Nov 16 17:18:34 db1063 kernel: [17198943.861458] [<ffffffff81594077>] ? bit_wait_io+0x17/0x60
70Nov 16 17:18:34 db1063 kernel: [17198943.861461] [<ffffffff81593b5a>] ? __wait_on_bit+0x5a/0x90
71Nov 16 17:18:34 db1063 kernel: [17198943.861469] [<ffffffff81169601>] ? wait_on_page_bit+0xc1/0xe0
72Nov 16 17:18:34 db1063 kernel: [17198943.861472] [<ffffffff810b8670>] ? autoremove_wake_function+0x40/0x40
73Nov 16 17:18:34 db1063 kernel: [17198943.861476] [<ffffffff811696f7>] ? __filemap_fdatawait_range+0xd7/0x150
74Nov 16 17:18:34 db1063 kernel: [17198943.861481] [<ffffffff812c397f>] ? submit_bio+0x6f/0x170
75Nov 16 17:18:34 db1063 kernel: [17198943.861485] [<ffffffff8116977f>] ? filemap_fdatawait_range+0xf/0x30
76Nov 16 17:18:34 db1063 kernel: [17198943.861498] [<ffffffffa018ed45>] ? jbd2_journal_commit_transaction+0xd15/0x1900 [jbd2]
77Nov 16 17:18:34 db1063 kernel: [17198943.861506] [<ffffffff810acb42>] ? dequeue_entity+0x3f2/0x920
78Nov 16 17:18:34 db1063 kernel: [17198943.861510] [<ffffffff810ad243>] ? put_prev_entity+0x33/0x710
79Nov 16 17:18:34 db1063 kernel: [17198943.861518] [<ffffffff810dd749>] ? try_to_del_timer_sync+0x59/0x80
80Nov 16 17:18:34 db1063 kernel: [17198943.861525] [<ffffffffa01931cd>] ? kjournald2+0xdd/0x280 [jbd2]
81Nov 16 17:18:34 db1063 kernel: [17198943.861528] [<ffffffff810b8630>] ? wait_woken+0x90/0x90
82Nov 16 17:18:34 db1063 kernel: [17198943.861534] [<ffffffffa01930f0>] ? commit_timeout+0x10/0x10 [jbd2]
83Nov 16 17:18:34 db1063 kernel: [17198943.861540] [<ffffffff81096eff>] ? kthread+0xdf/0x100
84Nov 16 17:18:34 db1063 kernel: [17198943.861544] [<ffffffff81096e20>] ? kthread_park+0x50/0x50
85Nov 16 17:18:34 db1063 kernel: [17198943.861548] [<ffffffff8159795f>] ? ret_from_fork+0x3f/0x70
86Nov 16 17:18:34 db1063 kernel: [17198943.861551] [<ffffffff81096e20>] ? kthread_park+0x50/0x50
87Nov 16 17:18:34 db1063 kernel: [17198943.861558] INFO: task systemd-journal:801 blocked for more than 120 seconds.
88Nov 16 17:18:34 db1063 kernel: [17198943.869816] Not tainted 4.4.0-3-amd64 #1
89Nov 16 17:18:34 db1063 kernel: [17198943.875071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
90Nov 16 17:18:34 db1063 kernel: [17198943.884107] systemd-journal D 0000000000000007 0 801 1 0x00000000
91Nov 16 17:18:34 db1063 kernel: [17198943.884111] ffff880ff56c4280 ffff881ff55b6f40 ffff880ff6274000 ffff880ff6273860
92Nov 16 17:18:34 db1063 kernel: [17198943.884118] 7fffffffffffffff ffffffff81594060 ffff880ff62738e0 ffff88162c7c9778
93Nov 16 17:18:34 db1063 kernel: [17198943.884121] ffffffff815937e1 0000000000000000 ffffffff81596795 7fffffffffffffff
94Nov 16 17:18:34 db1063 kernel: [17198943.884124] Call Trace:
95Nov 16 17:18:34 db1063 kernel: [17198943.884127] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
96Nov 16 17:18:34 db1063 kernel: [17198943.884138] [<ffffffff815937e1>] ? schedule+0x31/0x80
97Nov 16 17:18:34 db1063 kernel: [17198943.884146] [<ffffffff81596795>] ? schedule_timeout+0x235/0x2d0
98Nov 16 17:18:34 db1063 kernel: [17198943.884149] [<ffffffff812bf4db>] ? queue_unplugged+0x9b/0xa0
99Nov 16 17:18:34 db1063 kernel: [17198943.884152] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
100Nov 16 17:18:34 db1063 kernel: [17198943.884155] [<ffffffff81592dd4>] ? io_schedule_timeout+0xb4/0x130
101Nov 16 17:18:34 db1063 kernel: [17198943.884158] [<ffffffff810b8305>] ? prepare_to_wait+0x55/0x80
102Nov 16 17:18:34 db1063 kernel: [17198943.884171] [<ffffffff81594077>] ? bit_wait_io+0x17/0x60
103Nov 16 17:18:34 db1063 kernel: [17198943.884177] [<ffffffff81593b5a>] ? __wait_on_bit+0x5a/0x90
104Nov 16 17:18:34 db1063 kernel: [17198943.884180] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
105Nov 16 17:18:34 db1063 kernel: [17198943.884183] [<ffffffff81593c0e>] ? out_of_line_wait_on_bit+0x7e/0xa0
106Nov 16 17:18:34 db1063 kernel: [17198943.884186] [<ffffffff810b8670>] ? autoremove_wake_function+0x40/0x40
107Nov 16 17:18:34 db1063 kernel: [17198943.884199] [<ffffffff81213553>] ? bh_submit_read+0x73/0x80
108Nov 16 17:18:34 db1063 kernel: [17198943.884217] [<ffffffffa02751d5>] ? ext4_get_branch+0xc5/0x130 [ext4]
109Nov 16 17:18:34 db1063 kernel: [17198943.884237] [<ffffffffa0275c97>] ? ext4_ind_map_blocks+0xb7/0xaf0 [ext4]
110Nov 16 17:18:34 db1063 kernel: [17198943.884241] [<ffffffff81188fc5>] ? zone_statistics+0x75/0x80
111Nov 16 17:18:34 db1063 kernel: [17198943.884244] [<ffffffff81172596>] ? get_page_from_freelist+0x4a6/0xa10
112Nov 16 17:18:34 db1063 kernel: [17198943.884248] [<ffffffff81188fc5>] ? zone_statistics+0x75/0x80
113Nov 16 17:18:34 db1063 kernel: [17198943.884262] [<ffffffffa0236c04>] ? ext4_map_blocks+0x1f4/0x4e0 [ext4]
114Nov 16 17:18:34 db1063 kernel: [17198943.884264] [<ffffffff8116aaab>] ? __add_to_page_cache_locked+0x16b/0x1e0
115Nov 16 17:18:34 db1063 kernel: [17198943.884268] [<ffffffff811937a4>] ? workingset_refault+0x54/0xa0
116Nov 16 17:18:34 db1063 kernel: [17198943.884283] [<ffffffffa027fe78>] ? ext4_mpage_readpages+0x4a8/0x810 [ext4]
117Nov 16 17:18:34 db1063 kernel: [17198943.884288] [<ffffffff811b85fa>] ? alloc_pages_current+0x8a/0x110
118Nov 16 17:18:34 db1063 kernel: [17198943.884292] [<ffffffff81177c60>] ? __do_page_cache_readahead+0x1a0/0x220
119Nov 16 17:18:34 db1063 kernel: [17198943.884294] [<ffffffff8116abe7>] ? pagecache_get_page+0x27/0x1a0
120Nov 16 17:18:34 db1063 kernel: [17198943.884297] [<ffffffff8116c601>] ? filemap_fault+0x381/0x400
121Nov 16 17:18:34 db1063 kernel: [17198943.884311] [<ffffffffa023d7a1>] ? ext4_filemap_fault+0x31/0x50 [ext4]
122Nov 16 17:18:34 db1063 kernel: [17198943.884314] [<ffffffff811958ff>] ? __do_fault+0x4f/0xe0
123Nov 16 17:18:34 db1063 kernel: [17198943.884318] [<ffffffff8119a2f3>] ? handle_mm_fault+0x1053/0x1840
124Nov 16 17:18:34 db1063 kernel: [17198943.884322] [<ffffffff81063711>] ? __do_page_fault+0x1a1/0x440
125Nov 16 17:18:34 db1063 kernel: [17198943.884325] [<ffffffff81599778>] ? page_fault+0x28/0x30
126Nov 16 17:18:34 db1063 kernel: [17198943.884333] INFO: task xfsaild/dm-0:1034 blocked for more than 120 seconds.
127Nov 16 17:18:34 db1063 kernel: [17198943.892399] Not tainted 4.4.0-3-amd64 #1
128Nov 16 17:18:34 db1063 kernel: [17198943.897651] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
129Nov 16 17:18:34 db1063 kernel: [17198943.906687] xfsaild/dm-0 D ffff880fffaf5d80 0 1034 2 0x00000000
130Nov 16 17:18:34 db1063 kernel: [17198943.906691] ffff881ff7210e00 ffff880ff8f5f080 ffff881ff7928000 0000000000000000
131Nov 16 17:18:34 db1063 kernel: [17198943.906693] ffff881ff7b10900 ffff881ff5e6b000 ffff881ff5e6b128 ffff880ff7a44000
132Nov 16 17:18:34 db1063 kernel: [17198943.906696] ffffffff815937e1 0000000000000001 ffffffffa0632f84 000000020042938d
133Nov 16 17:18:34 db1063 kernel: [17198943.906699] Call Trace:
134Nov 16 17:18:34 db1063 kernel: [17198943.906702] [<ffffffff815937e1>] ? schedule+0x31/0x80
135Nov 16 17:18:34 db1063 kernel: [17198943.906795] [<ffffffffa0632f84>] ? _xfs_log_force+0x164/0x2d0 [xfs]
136Nov 16 17:18:34 db1063 kernel: [17198943.906798] [<ffffffff810a2150>] ? wake_up_q+0x60/0x60
137Nov 16 17:18:34 db1063 kernel: [17198943.906824] [<ffffffffa0633111>] ? xfs_log_force+0x21/0x90 [xfs]
138Nov 16 17:18:34 db1063 kernel: [17198943.906850] [<ffffffffa063dc27>] ? xfsaild+0x197/0x740 [xfs]
139Nov 16 17:18:34 db1063 kernel: [17198943.906877] [<ffffffffa063da90>] ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
140Nov 16 17:18:34 db1063 kernel: [17198943.906881] [<ffffffff81096eff>] ? kthread+0xdf/0x100
141Nov 16 17:18:34 db1063 kernel: [17198943.906884] [<ffffffff81096e20>] ? kthread_park+0x50/0x50
142Nov 16 17:18:34 db1063 kernel: [17198943.906887] [<ffffffff8159795f>] ? ret_from_fork+0x3f/0x70
143Nov 16 17:18:34 db1063 kernel: [17198943.906891] [<ffffffff81096e20>] ? kthread_park+0x50/0x50
144Nov 16 17:18:34 db1063 kernel: [17198943.906932] INFO: task mysqld:40572 blocked for more than 120 seconds.
145Nov 16 17:18:34 db1063 kernel: [17198943.914511] Not tainted 4.4.0-3-amd64 #1
146Nov 16 17:18:34 db1063 kernel: [17198943.919765] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
147Nov 16 17:18:34 db1063 kernel: [17198943.928798] mysqld D ffff880fffa75d80 0 40572 4385 0x00000000
148Nov 16 17:18:34 db1063 kernel: [17198943.928802] ffff8805aaed6280 ffff880ff8f3af80 ffff881f8e550000 ffff881f8e54fd00
149Nov 16 17:18:34 db1063 kernel: [17198943.928804] 7fffffffffffffff ffffffff81594060 ffff881f8e54fd80 0007ffffffffffff
150Nov 16 17:18:34 db1063 kernel: [17198943.928807] ffffffff815937e1 0000000000000000 ffffffff81596795 7fffffffffffffff
151Nov 16 17:18:34 db1063 kernel: [17198943.928809] Call Trace:
152Nov 16 17:18:34 db1063 kernel: [17198943.928812] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
153Nov 16 17:18:34 db1063 kernel: [17198943.928814] [<ffffffff815937e1>] ? schedule+0x31/0x80
154Nov 16 17:18:34 db1063 kernel: [17198943.928817] [<ffffffff81596795>] ? schedule_timeout+0x235/0x2d0
155Nov 16 17:18:34 db1063 kernel: [17198943.928820] [<ffffffff812c57c8>] ? blk_queue_end_tag+0x28/0xc0
156Nov 16 17:18:34 db1063 kernel: [17198943.928847] [<ffffffffa00963d5>] ? scsi_request_fn+0x1c5/0x5f0 [scsi_mod]
157Nov 16 17:18:34 db1063 kernel: [17198943.928849] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
158Nov 16 17:18:34 db1063 kernel: [17198943.928852] [<ffffffff81592dd4>] ? io_schedule_timeout+0xb4/0x130
159Nov 16 17:18:34 db1063 kernel: [17198943.928854] [<ffffffff810b8305>] ? prepare_to_wait+0x55/0x80
160Nov 16 17:18:34 db1063 kernel: [17198943.928856] [<ffffffff81594077>] ? bit_wait_io+0x17/0x60
161Nov 16 17:18:34 db1063 kernel: [17198943.928859] [<ffffffff81593b5a>] ? __wait_on_bit+0x5a/0x90
162Nov 16 17:18:34 db1063 kernel: [17198943.928862] [<ffffffff81169601>] ? wait_on_page_bit+0xc1/0xe0
163Nov 16 17:18:34 db1063 kernel: [17198943.928865] [<ffffffff810b8670>] ? autoremove_wake_function+0x40/0x40
164Nov 16 17:18:34 db1063 kernel: [17198943.928868] [<ffffffff811696f7>] ? __filemap_fdatawait_range+0xd7/0x150
165Nov 16 17:18:34 db1063 kernel: [17198943.928870] [<ffffffff8116b48f>] ? __filemap_fdatawrite_range+0xcf/0x100
166Nov 16 17:18:34 db1063 kernel: [17198943.928874] [<ffffffff8116977f>] ? filemap_fdatawait_range+0xf/0x30
167Nov 16 17:18:34 db1063 kernel: [17198943.928876] [<ffffffff8116b59b>] ? filemap_write_and_wait_range+0x3b/0x60
168Nov 16 17:18:34 db1063 kernel: [17198943.928901] [<ffffffffa0616301>] ? xfs_file_fsync+0x61/0x220 [xfs]
169Nov 16 17:18:34 db1063 kernel: [17198943.928905] [<ffffffff8120d4e8>] ? do_fsync+0x38/0x60
170Nov 16 17:18:34 db1063 kernel: [17198943.928907] [<ffffffff8120d75c>] ? SyS_fsync+0xc/0x10
171Nov 16 17:18:34 db1063 kernel: [17198943.928910] [<ffffffff815975b6>] ? system_call_fast_compare_end+0xc/0x6b
172Nov 16 17:18:34 db1063 kernel: [17198943.928913] INFO: task mysqld:40701 blocked for more than 120 seconds.
173Nov 16 17:18:34 db1063 kernel: [17198943.936494] Not tainted 4.4.0-3-amd64 #1
174Nov 16 17:18:34 db1063 kernel: [17198943.941764] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
175Nov 16 17:18:34 db1063 kernel: [17198943.950800] mysqld D ffff880fffa35d80 0 40701 4385 0x00000000
176Nov 16 17:18:34 db1063 kernel: [17198943.950807] ffff88081913c4c0 ffff880ff8f24f00 ffff881fa1c50000 ffff881fa1c4fab8
177Nov 16 17:18:34 db1063 kernel: [17198943.950810] 7fffffffffffffff ffff88081913c4c0 ffff88081913c4c0 ffff880902e9f340
178Nov 16 17:18:34 db1063 kernel: [17198943.950812] ffffffff815937e1 0000000000000000 ffffffff81596795 7fffffffffffffff
179Nov 16 17:18:34 db1063 kernel: [17198943.950814] Call Trace:
180Nov 16 17:18:34 db1063 kernel: [17198943.950817] [<ffffffff815937e1>] ? schedule+0x31/0x80
181Nov 16 17:18:34 db1063 kernel: [17198943.950842] [<ffffffff81596795>] ? schedule_timeout+0x235/0x2d0
182Nov 16 17:18:34 db1063 kernel: [17198943.950851] [<ffffffff812bf21f>] ? __blk_run_queue+0x2f/0x40
183Nov 16 17:18:34 db1063 kernel: [17198943.950854] [<ffffffff812bf465>] ? queue_unplugged+0x25/0xa0
184Nov 16 17:18:34 db1063 kernel: [17198943.950856] [<ffffffff81592dd4>] ? io_schedule_timeout+0xb4/0x130
185Nov 16 17:18:34 db1063 kernel: [17198943.950860] [<ffffffff81218a88>] ? do_blockdev_direct_IO+0x1b38/0x2bf0
186Nov 16 17:18:34 db1063 kernel: [17198943.950884] [<ffffffffa060b070>] ? xfs_get_blocks+0x10/0x10 [xfs]
187Nov 16 17:18:34 db1063 kernel: [17198943.950911] [<ffffffffa060a56c>] ? xfs_vm_direct_IO+0x6c/0xe0 [xfs]
188Nov 16 17:18:34 db1063 kernel: [17198943.950919] [<ffffffff8116ba09>] ? generic_file_read_iter+0x449/0x600
189Nov 16 17:18:34 db1063 kernel: [17198943.950942] [<ffffffffa06160a1>] ? xfs_file_read_iter+0x111/0x310 [xfs]
190Nov 16 17:18:34 db1063 kernel: [17198943.950950] [<ffffffff811dadca>] ? __vfs_read+0xca/0x100
191Nov 16 17:18:34 db1063 kernel: [17198943.950956] [<ffffffff811db3e1>] ? vfs_read+0x81/0x120
192Nov 16 17:18:34 db1063 kernel: [17198943.950959] [<ffffffff811dc396>] ? SyS_pread64+0x86/0xb0
193Nov 16 17:18:34 db1063 kernel: [17198943.950962] [<ffffffff81222894>] ? SyS_epoll_wait+0x94/0xd0
194Nov 16 17:18:34 db1063 kernel: [17198943.950964] [<ffffffff815975b6>] ? system_call_fast_compare_end+0xc/0x6b
195Nov 16 17:18:34 db1063 kernel: [17198943.950978] INFO: task mysqld:45784 blocked for more than 120 seconds.
196Nov 16 17:18:34 db1063 kernel: [17198943.958565] Not tainted 4.4.0-3-amd64 #1
197Nov 16 17:18:34 db1063 kernel: [17198943.963817] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
198Nov 16 17:18:34 db1063 kernel: [17198943.972850] mysqld D 0000000000000002 0 45784 4385 0x00000000
199Nov 16 17:18:34 db1063 kernel: [17198943.972853] ffff880000176140 ffff880ff51ba080 ffff881046f40000 ffff881046f3fd00
200Nov 16 17:18:34 db1063 kernel: [17198943.972860] 7fffffffffffffff ffffffff81594060 ffff881046f3fd80 0007ffffffffffff
201Nov 16 17:18:34 db1063 kernel: [17198943.972862] ffffffff815937e1 0000000000000000 ffffffff81596795 7fffffffffffffff
202Nov 16 17:18:34 db1063 kernel: [17198943.972864] Call Trace:
203Nov 16 17:18:34 db1063 kernel: [17198943.972867] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
204Nov 16 17:18:34 db1063 kernel: [17198943.972869] [<ffffffff815937e1>] ? schedule+0x31/0x80
205Nov 16 17:18:34 db1063 kernel: [17198943.972892] [<ffffffff81596795>] ? schedule_timeout+0x235/0x2d0
206Nov 16 17:18:34 db1063 kernel: [17198943.972902] [<ffffffff812bfa99>] ? kblockd_schedule_work+0x19/0x20
207Nov 16 17:18:34 db1063 kernel: [17198943.972913] [<ffffffffa009608b>] ? __scsi_queue_insert+0xbb/0x100 [scsi_mod]
208Nov 16 17:18:34 db1063 kernel: [17198943.972919] [<ffffffffa00967bd>] ? scsi_request_fn+0x5ad/0x5f0 [scsi_mod]
209Nov 16 17:18:34 db1063 kernel: [17198943.972921] [<ffffffff81594060>] ? bit_wait_timeout+0xa0/0xa0
210Nov 16 17:18:34 db1063 kernel: [17198943.972924] [<ffffffff81592dd4>] ? io_schedule_timeout+0xb4/0x130
211Nov 16 17:18:34 db1063 kernel: [17198943.972933] [<ffffffff810b8305>] ? prepare_to_wait+0x55/0x80

gerritbot added a comment.Nov 16 2017, 5:41 PM

Change 391870 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1070: New master in s5

https://gerrit.wikimedia.org/r/391870

gerritbot added a comment.Nov 16 2017, 5:51 PM

Change 391870 merged by Jcrespo:
[operations/puppet@production] db1070: New master in s5

https://gerrit.wikimedia.org/r/391870

gerritbot added a comment.Nov 16 2017, 5:53 PM

Change 391869 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Failover db1063 to db1070

https://gerrit.wikimedia.org/r/391869

gerritbot added a comment.Nov 16 2017, 6:04 PM

Change 391873 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1100 as vslow

https://gerrit.wikimedia.org/r/391873

Aklapper mentioned this in T180704: "Add links" under "Languages" for newly created pages only shows "An unknown error occured".Nov 16 2017, 6:27 PM
gerritbot added a comment.Nov 16 2017, 6:40 PM

Change 391876 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool only the servers with working replication

https://gerrit.wikimedia.org/r/391876

gerritbot added a comment.Nov 16 2017, 6:42 PM

Change 391876 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Pool only the servers with working replication

https://gerrit.wikimedia.org/r/391876

• Marostegui mentioned this in T174569: Schema change for refactored comment storage.Nov 16 2017, 7:02 PM
gerritbot added a comment.Nov 16 2017, 8:01 PM

Change 391889 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool full all delayed servers

https://gerrit.wikimedia.org/r/391889

gerritbot added a comment.Nov 16 2017, 8:08 PM

Change 391889 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Depool fully all delayed servers

https://gerrit.wikimedia.org/r/391889

Robby subscribed.Nov 16 2017, 10:24 PM
greg added a parent task: T178635: 1.31.0-wmf.8 deployment blockers.Nov 16 2017, 11:21 PM
greg mentioned this in T178635: 1.31.0-wmf.8 deployment blockers.
Jdforrester-WMF subscribed.Nov 16 2017, 11:28 PM
greg triaged this task as Unbreak Now! priority.Nov 16 2017, 11:37 PM
greg added a project: Wikimedia-Incident.
greg moved this task from Active investigation to Active Situation on the Wikimedia-Incident board.
Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptNov 16 2017, 11:37 PM
• Marostegui subscribed.Nov 17 2017, 4:48 AM

reverts for logging and recentchanges on dewiki and wikidatawiki are finished for most of the hosts (still running for db1071 as its hardware isn't as powerful).
Also finishing reverting archive table (very small table, 5G in dewiki and 2G in wikidata)

• Marostegui added a comment.Nov 17 2017, 4:51 AM

archive done
doing ipblocks now

• Marostegui added a comment.Nov 17 2017, 4:56 AM

ipblocks done
filearchive done
oldimage done
protected_titles done

servers are now catching up

• Marostegui added a comment.Nov 17 2017, 4:59 AM

Reminder: we have to enable GTID on the slaves.

gerritbot added a comment.Nov 17 2017, 5:07 AM

Change 391995 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Start repooling all the hosts

https://gerrit.wikimedia.org/r/391995

• Marostegui added a comment.Nov 17 2017, 5:09 AM

All the hosts (apart from db1071) are now up to date and ready to be pooled.
I have done: https://gerrit.wikimedia.org/r/#/c/391995/2 but I will wait for another pair of eyes to take a look at that patch before deploying it.

gerritbot added a comment.Nov 17 2017, 5:11 AM

Change 391996 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5.hosts: db1070 is now the master

https://gerrit.wikimedia.org/r/391996

gerritbot added a comment.Nov 17 2017, 5:14 AM

Change 391996 merged by jenkins-bot:
[operations/software@master] s5.hosts: db1070 is now the master

https://gerrit.wikimedia.org/r/391996

• Marostegui renamed this task from db1063 crashed to s5 primary master db1063 crashed.Nov 17 2017, 6:31 AM
Stashbot subscribed.Nov 17 2017, 6:34 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-17T06:34:49Z] <marostegui> Revert schema changes on s5 codfw master with replication enabled, lag will be generated on codfw s5 - T180714

• Marostegui moved this task from Triage to In progress on the DBA board.Nov 17 2017, 6:36 AM
gerritbot added a comment.Nov 17 2017, 7:15 AM

Change 391995 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Start repooling all the hosts

https://gerrit.wikimedia.org/r/391995

Stashbot added a comment.Nov 17 2017, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-17T07:17:18Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool s5 eqiad hosts after reverting schema change - T180714 (duration: 00m 50s)

gerritbot added a comment.Nov 17 2017, 7:30 AM

Change 391873 abandoned by Marostegui:
db-eqiad.php: Pool db1100 as vslow

Reason:
not needed

https://gerrit.wikimedia.org/r/391873

Stashbot added a comment.Nov 17 2017, 7:33 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-17T07:33:19Z] <marostegui> Enable GTID on all the eqiad up-to-date hosts, only pending db1071 - T180714

• Marostegui lowered the priority of this task from Unbreak Now! to Medium.EditedNov 17 2017, 7:34 AM

Setting back priority to normal as we are back to a normal state now.
Pending things:

  • Revert schema change on codfw (ONGOING)
  • move dbstore1001 under the new master (db1070)
  • revert schema change on dbstore1001
  • move dbstore1002 under the new master (db1070) [DONE]
  • revert schema change on dbstore1002 [ONGOING]
  • decide what to do with db1063 (definitely not a master for anything else, we might rebuild it as vslow from db1071 for example)
gerritbot added a comment.Nov 17 2017, 7:39 AM

Change 391997 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for some hosts

https://gerrit.wikimedia.org/r/391997

Stashbot added a comment.Nov 17 2017, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-17T07:50:12Z] <marostegui> Move dbstore1002 under db1070 - T180714

Stashbot added a comment.Nov 17 2017, 7:51 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-17T07:51:04Z] <marostegui> Revert schema change on dbstore1002 - T180714

• Marostegui updated the task description. (Show Details)Nov 17 2017, 8:00 AM
• Marostegui updated the task description. (Show Details)Nov 17 2017, 8:26 AM
gerritbot added a comment.Nov 17 2017, 8:28 AM

Change 391997 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for some hosts

https://gerrit.wikimedia.org/r/391997

• Marostegui added a comment.EditedNov 17 2017, 8:44 AM

db1063 is totally broken and won't start

P6337 (An Untitled Masterwork)
1171117 08:42:07 mysqld_safe Starting mysqld daemon with databases from /srv/sqldata
2171117 8:42:07 [Note] /opt/wmf-mariadb10/bin/mysqld (mysqld 10.0.29-MariaDB) starting as process 45668 ...
3171117 8:42:07 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used; This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=db1063' or '--log-bin=db1063-bin' to avoid this problem.
4171117 8:42:07 [Note] InnoDB: Using mutexes to ref count buffer pool pages
5171117 8:42:07 [Note] InnoDB: The InnoDB memory heap is disabled
6171117 8:42:07 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
7171117 8:42:07 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
8171117 8:42:07 [Note] InnoDB: Compressed tables use zlib 1.2.3
9171117 8:42:07 [Note] InnoDB: Using Linux native AIO
10171117 8:42:07 [Note] InnoDB: Using CPU crc32 instructions
11171117 8:42:07 [Note] InnoDB: Initializing buffer pool, size = 94.0G
12171117 8:42:10 [Note] InnoDB: Completed initialization of buffer pool
13171117 8:42:11 [Note] InnoDB: Highest supported file format is Barracuda.
14171117 8:42:11 [Note] InnoDB: Log scan progressed past the checkpoint lsn 15803672999980
15171117 8:42:11 [Note] InnoDB: Database was not shutdown normally!
16171117 8:42:11 [Note] InnoDB: Starting crash recovery.
17171117 8:42:11 [Note] InnoDB: Reading tablespace information from the .ibd files...
18171117 8:42:12 [Note] InnoDB: Restoring possible half-written data pages
19171117 8:42:12 [Note] InnoDB: from the doublewrite buffer...
20InnoDB: Doing recovery: scanned up to log sequence number 15803678242816
21InnoDB: Doing recovery: scanned up to log sequence number 15803683485696
22InnoDB: Doing recovery: scanned up to log sequence number 15803688728576
23InnoDB: Doing recovery: scanned up to log sequence number 15803693971456
24InnoDB: Doing recovery: scanned up to log sequence number 15803699214336
25InnoDB: Doing recovery: scanned up to log sequence number 15803704457216
26InnoDB: Doing recovery: scanned up to log sequence number 15803709700096
27InnoDB: Doing recovery: scanned up to log sequence number 15803714942976
28InnoDB: Doing recovery: scanned up to log sequence number 15803720185856
29InnoDB: Doing recovery: scanned up to log sequence number 15803725428736
30InnoDB: Doing recovery: scanned up to log sequence number 15803730671616
31InnoDB: Doing recovery: scanned up to log sequence number 15803735914496
32InnoDB: Doing recovery: scanned up to log sequence number 15803741157376
33InnoDB: Doing recovery: scanned up to log sequence number 15803746400256
34InnoDB: Doing recovery: scanned up to log sequence number 15803751643136
35InnoDB: Doing recovery: scanned up to log sequence number 15803756886016
36InnoDB: Doing recovery: scanned up to log sequence number 15803762128896
37InnoDB: Doing recovery: scanned up to log sequence number 15803767371776
38InnoDB: Doing recovery: scanned up to log sequence number 15803772614656
39InnoDB: Doing recovery: scanned up to log sequence number 15803777857536
40InnoDB: Doing recovery: scanned up to log sequence number 15803783100416
41InnoDB: Doing recovery: scanned up to log sequence number 15803788343296
42InnoDB: Doing recovery: scanned up to log sequence number 15803793586176
43InnoDB: Doing recovery: scanned up to log sequence number 15803798829056
44InnoDB: Doing recovery: scanned up to log sequence number 15803804071936
45InnoDB: Doing recovery: scanned up to log sequence number 15803809314816
46InnoDB: Doing recovery: scanned up to log sequence number 15803814557696
47InnoDB: Doing recovery: scanned up to log sequence number 15803819800576
48InnoDB: Doing recovery: scanned up to log sequence number 15803825043456
49InnoDB: Doing recovery: scanned up to log sequence number 15803830286336
50InnoDB: Doing recovery: scanned up to log sequence number 15803835529216
51InnoDB: Doing recovery: scanned up to log sequence number 15803840772096
52InnoDB: Doing recovery: scanned up to log sequence number 15803846014976
53InnoDB: Doing recovery: scanned up to log sequence number 15803851257856
54InnoDB: Doing recovery: scanned up to log sequence number 15803856500736
55InnoDB: Doing recovery: scanned up to log sequence number 15803861743616
56InnoDB: Doing recovery: scanned up to log sequence number 15803866986496
57InnoDB: Doing recovery: scanned up to log sequence number 15803872229376
58InnoDB: Doing recovery: scanned up to log sequence number 15803877472256
59InnoDB: Doing recovery: scanned up to log sequence number 15803882715136
60InnoDB: Doing recovery: scanned up to log sequence number 15803887958016
61InnoDB: Doing recovery: scanned up to log sequence number 15803893200896
62InnoDB: Doing recovery: scanned up to log sequence number 15803898443776
63InnoDB: Doing recovery: scanned up to log sequence number 15803903686656
64InnoDB: Doing recovery: scanned up to log sequence number 15803908929536
65InnoDB: Doing recovery: scanned up to log sequence number 15803914172416
66InnoDB: Doing recovery: scanned up to log sequence number 15803919415296
67InnoDB: Doing recovery: scanned up to log sequence number 15803924658176
68InnoDB: Doing recovery: scanned up to log sequence number 15803929901056
69InnoDB: Doing recovery: scanned up to log sequence number 15803935143936
70InnoDB: Doing recovery: scanned up to log sequence number 15803940386816
71InnoDB: Doing recovery: scanned up to log sequence number 15803945629696
72InnoDB: Doing recovery: scanned up to log sequence number 15803950872576
73InnoDB: Doing recovery: scanned up to log sequence number 15803956115456
74InnoDB: Doing recovery: scanned up to log sequence number 15803961358336
75InnoDB: Doing recovery: scanned up to log sequence number 15803966601216
76InnoDB: Doing recovery: scanned up to log sequence number 15803971844096
77InnoDB: Doing recovery: scanned up to log sequence number 15803977086976
78InnoDB: Doing recovery: scanned up to log sequence number 15803982329856
79InnoDB: Doing recovery: scanned up to log sequence number 15803987572736
80InnoDB: Doing recovery: scanned up to log sequence number 15803992815616
81InnoDB: Doing recovery: scanned up to log sequence number 15803998058496
82InnoDB: Doing recovery: scanned up to log sequence number 15804003301376
83InnoDB: Doing recovery: scanned up to log sequence number 15804008544256
84InnoDB: Doing recovery: scanned up to log sequence number 15804013787136
85InnoDB: Doing recovery: scanned up to log sequence number 15804019030016
86InnoDB: Doing recovery: scanned up to log sequence number 15804024272896
87InnoDB: Doing recovery: scanned up to log sequence number 15804029515776
88InnoDB: Doing recovery: scanned up to log sequence number 15804034758656
89InnoDB: Doing recovery: scanned up to log sequence number 15804040001536
90InnoDB: Doing recovery: scanned up to log sequence number 15804045244416
91InnoDB: Doing recovery: scanned up to log sequence number 15804050487296
92InnoDB: Doing recovery: scanned up to log sequence number 15804055730176
93InnoDB: Doing recovery: scanned up to log sequence number 15804060973056
94InnoDB: Doing recovery: scanned up to log sequence number 15804066215936
95InnoDB: Doing recovery: scanned up to log sequence number 15804071458816
96InnoDB: Doing recovery: scanned up to log sequence number 15804076701696
97InnoDB: Doing recovery: scanned up to log sequence number 15804081944576
98InnoDB: Doing recovery: scanned up to log sequence number 15804087187456
99InnoDB: Doing recovery: scanned up to log sequence number 15804092430336
100InnoDB: Doing recovery: scanned up to log sequence number 15804097673216
101InnoDB: Doing recovery: scanned up to log sequence number 15804102916096
102InnoDB: Doing recovery: scanned up to log sequence number 15804108158976
103InnoDB: Doing recovery: scanned up to log sequence number 15804113401856
104InnoDB: Doing recovery: scanned up to log sequence number 15804118644736
105InnoDB: Doing recovery: scanned up to log sequence number 15804123887616
106InnoDB: Doing recovery: scanned up to log sequence number 15804129130496
107InnoDB: Doing recovery: scanned up to log sequence number 15804134373376
108InnoDB: Doing recovery: scanned up to log sequence number 15804139616256
109InnoDB: Doing recovery: scanned up to log sequence number 15804144859136
110InnoDB: Doing recovery: scanned up to log sequence number 15804150102016
111InnoDB: Doing recovery: scanned up to log sequence number 15804155344896
112InnoDB: Doing recovery: scanned up to log sequence number 15804160587776
113InnoDB: Doing recovery: scanned up to log sequence number 15804165830656
114InnoDB: Doing recovery: scanned up to log sequence number 15804171073536
115InnoDB: Doing recovery: scanned up to log sequence number 15804176316416
116InnoDB: Doing recovery: scanned up to log sequence number 15804181559296
117InnoDB: Doing recovery: scanned up to log sequence number 15804186802176
118InnoDB: Doing recovery: scanned up to log sequence number 15804192045056
119InnoDB: Doing recovery: scanned up to log sequence number 15804197287936
120InnoDB: Doing recovery: scanned up to log sequence number 15804202530816
121InnoDB: Doing recovery: scanned up to log sequence number 15804207773696
122InnoDB: Doing recovery: scanned up to log sequence number 15804213016576
123InnoDB: Doing recovery: scanned up to log sequence number 15804218259456
124InnoDB: Doing recovery: scanned up to log sequence number 15804223502336
125InnoDB: Doing recovery: scanned up to log sequence number 15804228745216
126InnoDB: Doing recovery: scanned up to log sequence number 15804233988096
127InnoDB: Doing recovery: scanned up to log sequence number 15804239230976
128InnoDB: Doing recovery: scanned up to log sequence number 15804244473856
129InnoDB: Doing recovery: scanned up to log sequence number 15804249716736
130InnoDB: Doing recovery: scanned up to log sequence number 15804254959616
131InnoDB: Doing recovery: scanned up to log sequence number 15804260202496
132InnoDB: Doing recovery: scanned up to log sequence number 15804265445376
133InnoDB: Doing recovery: scanned up to log sequence number 15804270688256
134InnoDB: Doing recovery: scanned up to log sequence number 15804275931136
135InnoDB: Doing recovery: scanned up to log sequence number 15804281174016
136InnoDB: Doing recovery: scanned up to log sequence number 15804286416896
137InnoDB: Doing recovery: scanned up to log sequence number 15804291659776
138InnoDB: Doing recovery: scanned up to log sequence number 15804296902656
139InnoDB: Doing recovery: scanned up to log sequence number 15804302145536
140InnoDB: Doing recovery: scanned up to log sequence number 15804307388416
141InnoDB: Doing recovery: scanned up to log sequence number 15804312631296
142InnoDB: Doing recovery: scanned up to log sequence number 15804317874176
143InnoDB: Doing recovery: scanned up to log sequence number 15804323117056
144InnoDB: Doing recovery: scanned up to log sequence number 15804328359936
145InnoDB: Doing recovery: scanned up to log sequence number 15804333602816
146InnoDB: Doing recovery: scanned up to log sequence number 15804338845696
147InnoDB: Doing recovery: scanned up to log sequence number 15804344088576
148InnoDB: Doing recovery: scanned up to log sequence number 15804349331456
149InnoDB: Doing recovery: scanned up to log sequence number 15804354574336
150InnoDB: Doing recovery: scanned up to log sequence number 15804359817216
151InnoDB: Doing recovery: scanned up to log sequence number 15804365060096
152InnoDB: Doing recovery: scanned up to log sequence number 15804370302976
153InnoDB: Doing recovery: scanned up to log sequence number 15804375545856
154InnoDB: Doing recovery: scanned up to log sequence number 15804380788736
155InnoDB: Doing recovery: scanned up to log sequence number 15804386031616
156InnoDB: Doing recovery: scanned up to log sequence number 15804391274496
157InnoDB: Doing recovery: scanned up to log sequence number 15804396517376
158InnoDB: Doing recovery: scanned up to log sequence number 15804401760256
159InnoDB: Doing recovery: scanned up to log sequence number 15804407003136
160InnoDB: Doing recovery: scanned up to log sequence number 15804412246016
161InnoDB: Doing recovery: scanned up to log sequence number 15804417488896
162InnoDB: Doing recovery: scanned up to log sequence number 15804422731776
163InnoDB: Doing recovery: scanned up to log sequence number 15804427974656
164InnoDB: Doing recovery: scanned up to log sequence number 15804433217536
165InnoDB: Doing recovery: scanned up to log sequence number 15804438460416
166InnoDB: Doing recovery: scanned up to log sequence number 15804443703296
167InnoDB: Doing recovery: scanned up to log sequence number 15804448946176
168InnoDB: Doing recovery: scanned up to log sequence number 15804454189056
169InnoDB: Doing recovery: scanned up to log sequence number 15804459431936
170InnoDB: Doing recovery: scanned up to log sequence number 15804464674816
171InnoDB: Doing recovery: scanned up to log sequence number 15804469917696
172InnoDB: Doing recovery: scanned up to log sequence number 15804475160576
173InnoDB: Doing recovery: scanned up to log sequence number 15804480403456
174InnoDB: Doing recovery: scanned up to log sequence number 15804485646336
175InnoDB: Doing recovery: scanned up to log sequence number 15804490889216
176InnoDB: Doing recovery: scanned up to log sequence number 15804496132096
177InnoDB: Doing recovery: scanned up to log sequence number 15804501374976
178InnoDB: Doing recovery: scanned up to log sequence number 15804506617856
179InnoDB: Doing recovery: scanned up to log sequence number 15804511860736
180InnoDB: Doing recovery: scanned up to log sequence number 15804517103616
181InnoDB: Doing recovery: scanned up to log sequence number 15804522346496
182InnoDB: Doing recovery: scanned up to log sequence number 15804527589376
183InnoDB: Doing recovery: scanned up to log sequence number 15804532832256
184InnoDB: Doing recovery: scanned up to log sequence number 15804538075136
185InnoDB: Doing recovery: scanned up to log sequence number 15804543318016
186InnoDB: Doing recovery: scanned up to log sequence number 15804548560896
187InnoDB: Doing recovery: scanned up to log sequence number 15804553803776
188InnoDB: Doing recovery: scanned up to log sequence number 15804558845440
1892017-11-17 08:42:32 7fada9c37780 InnoDB: Error: page 122931 log sequence number 15804558908733
190InnoDB: is in the future! Current system log sequence number 15804558844894.
191InnoDB: Your database may be corrupt or you may have copied the InnoDB
192InnoDB: tablespace but not the InnoDB log files. See
193InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
194InnoDB: for more information.
1952017-11-17 08:42:32 7fada9c37780 InnoDB: Error: page 126238 log sequence number 15804558913406
196InnoDB: is in the future! Current system log sequence number 15804558844894.
197InnoDB: Your database may be corrupt or you may have copied the InnoDB
198InnoDB: tablespace but not the InnoDB log files. See
199InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
200InnoDB: for more information.
201InnoDB: 1 transaction(s) which must be rolled back or cleaned up
202InnoDB: in total 2 row operations to undo
203InnoDB: Trx id counter is 102107098624
204171117 8:42:33 [Note] InnoDB: Starting an apply batch of log records to the database...
205InnoDB: Progress in percent: 0 InnoDB: Database page corruption on disk or a failed
206InnoDB: file read of page 17083931.
207InnoDB: You may have to recover from a backup.
2082017-11-17 08:42:34 7f949f3fd700 InnoDB: Page dump in ascii and hex (16384 bytes):
209 len 16384; hex d50146280104ae1b00527d700050b15100000e5f95025d3545bf0000000000000000000000b20040243d811b00000000242300050000011900000017c5f5fa37000000000000000002090000000000000000000000000000000000000000010002001b696e66696d756d0006000b000073757072656d756d090000100021513131333330303839323031363031323730353130343111adbffb0900001800215131313333303038393230313630323235303832383030125012b00900002000215131313333303038393230313630323235303832383031125012b6090400280021513131333330303839323031363131303231383031323217d4eb4a090000300021513131333330303839323031363132303630333339323418ad0b4c090000380021513131333330303839323031363132303630333339323518ad0b4e090000400021513131333330303839323031363132303630363138353018ad67c1090400480021513131333330303839323031363132303630363138353118ad67c509000050002151313133333030383932303137303231323038343030361a9ba3de09000058002151313133333030383932303137303232373139343731391b4219b90900006021b751313133333030383932303137303232373230333434301b427313080600680020513131333330303932303132313231383031343832310015997f0800007000205131313333303039323031333033323231383232323400d408390800007800205131313333303039323031333035303230323038313101f862a60800008000205131313333303039323031333035303230323038313201f862a8080400880020513131333330303932303133303530333036323832360202c7b1080000900020513131333330303932303133303530333036323832370202c7bb080000980020513131333330303932303133303730373033323430330335f12e080000a0002051313133333030393230313330373039303133363133033aac67080400a800205131313333303039323031333039303230343533303103ff4996080000b0002051313133333030393230313330393038313033323531040f011c080000b8002051313133333030393230313330393038313033323533040f011f080000c00020513131333330303932303133313232303039303633300599ddc5080400c80020513131333330303932303133313232303039303634350599ddef080000d00020513131333330303932303133313232303039303635370599de14080000d80020513131333330303932303133313232303039303732350599de5b080000e000205131313333303039323031333132323130373436303305a04c98080400e800205131313333303039323031333132323130373436303405a04c9a080000f0002051313133333030393230313430323231323032303031069206af080000f800205131313333303039323031343033303730303031303406c535bf0800010000205131313333303039323031343033313632303432323906e5ae230804010800205131313333303039323031343035323331303236323807dd7488080001100020513131333330303932303134313130343233303235300a223184080001180020513131333330303932303135303331303134353235310c11259f080001200020513131333330303932303135303830383131313830380e3f9b63080401280020513131333330303932303135303830383131313830380e3f9b65080001300020513131333330303932303135313032333038323934340f95ea2b08000138002051313133333030393230313531313135313834373039103f1f2e08000140002051313133333030393230313531313135313834373130103f1f3008040148002051313133333030393230313630353135313932353037141403f50800015000205131313333303039323031363036323430303432353314de50c70800015800205131313333303039323031363132303331383332343718a34d040800016000205131313333303039323031363132303331383332343918a34d130804016800205131313333303039323031363132313930383530313518e584b00800017000205131313333303039323031363132313930383530313618e584b20800017800205131313333303039323031373031323532313038323519fa199d0800018000205131313333303039323031373031323532313237353819fa410a080401880020513131333330303932303137303330363139323031301b8eedbb080001901b0b513131333330303932303137303330363139323031311b8eedc10900019800215131313333303039303230313330343138303432353435018e570e090001a000215131313333303039303230313330353133313433323136026ed51b090601a800215131313333303039303230313330353133313433323236026ed558090001b000215131313333303039303230313430363031313830363034080d80da090001b800215131313333303039303230313430363031313830363330080d8130090001c000215131313333303039303230313430363031313831303431080d8457090401c800215131313333303039303230313430363031313831343231080d872c090001d000215131313333303039303230313430363031313831353435080d8856090001d80021513131333330303930323031343037323930353339313008c892d3090001e0002151313133333030393032303135303831333032343331340e5449c4090401e8002151313133333030393032303135303831333032343332340e5449da090001f0002151313133333030393032303136303330353132333332391280dcc4090001f80021513131333330303930323031363033333130363539353412ec165f090002000021513131333330303930323031363033333130363539353412ec16630904020800215131313333303039303230313630373134303832333333154a96a709000210002151313133333030393032303137303331323133333235341bb7ac6309000218002151313133333030393032303137303331323133333235351bb7ac660900022000215131313333303039313230313330343138303432353436018e57110904022800215131313333303039313230313430373135303535303438089531fe090002300021513131333330303931323031343130313830323530333309cec912090002380021513131333330303931323031343130313830323530333409cec91f090002400021513131333330303931323031343130323030353133323309debbbd09040248002151313133333030393132303135303930383030303331350ed4f5f20900025000215131313333303039313230313630323235313032303033125058c30900025800215131313333303039313230313630323235313032303033125058c60900026000215131313333303039313230313630343037303333323138130bd4a50904026800215131313333303039313230313630343037303333323230130bd4b10900027000215131313333303039313230313630343138303531353332133acd320900027800215131313333303039323230313330343138303432353437018e571609000280183451313133333030393232303134303832343132313530330911b5b20906028800215131313333303039333230313330343138303432353438018e57180900029000215131313333303039343230313330343138303432353439018e571d090002980021513131333330303934323031343038303731353030313908e46117090002a016ea513131333330303934323031363035303430323034343213b5b4bb090502a800215131313333303039353230313330343138303432353530018e5722090002b00021513131333330303935323031343038303731353030333108e4612c090002b816a8513131333330303935323031363035303430323034353613b5b520090002c000215131313333303039363230313330343138303432353531018e5724090502c80021513131333330303936323031343038303731353030343308e46140090002d01666513131333330303936323031363035303430323034343213b5b4b9090002d800215131313333303039373230313330343138303432353532018e5729090002e00021513131333330303937323031333130323930393230333004d98be3090502e80021513131333330303937323031333130323930393230333104d98be7090002f000215131313333303039373230313630313039323030313230114e908b090002f800215131313333303039373230313630313039323030313231114e908c0900030000215131313333303039373230313630313131313035313537115a928e0904030800215131313333303039373230313630313131313035313537115a92900900031000215131313333303039383230313330343138303432353532018e572e090003180021513131333330303938323031333035323831333434303602cdcfe7090003200021513131333330303938323031343036333030323430343108650c030904032800215131313333303039383230313430373031313735353035086c28dd0900033000215131313333303039383230313430373137303630383230089d115109000338002151313133333030393832303135313031393135333434330f82986b09000340002151313133333030393832303135313031393135333434340f82987009040348002151313133333030393832303135313031393135333435310f82989109000350002151313133333030393832303135313032333033313133340f94df9409000358002151313133333030393832303135313032333033313134340f94dfb009000360153b5131313333303039383230313531313132323035353334102871ca0906036800215131313333303039393230313330343138303432353533018e5732090003700021513131333330303939323031333130323930393233353804d98e9e090003780021513131333330303939323031333130323930393234303004d98ea30900038000215131313333303039393230313630313130313931363534115634050904038800215131313333303039393230313630313131313035333330115a954607000390001f51313133333031323031323131323531303530343400034f0407000398001f513131333330313230313330343139313434373437019a4ff3070003a0001f51313133333031323031333038323831353336303803f59936070403a8001f51313133333031323031343031313031303237353305e4fcf6070003b0001f51313133333031323031343031313031303237353405e4fcf9070003b8001f51313133333031323031343033333131323134323307031fb6070003c0001f513131333330313230313430343233313333363433073ca617070403c8001f513131333330313230313430353132313831363330079f54ba070003d0001f5131313333303132303135303731313038303130390d9eb22f070003d8001f5131313333303132303135303733303135323630360e2c056f070003e0001f5131313333303132303135303931303032343334300eeeaa22070403e8001f5131313333303132303135303931303032343334310eeeaa31070003f0001f5131313333303132303135313031393138333235390f837b9a070003f8001f5131313333303132303135313032373138313634340fb4e0f407000400001f51313133333031323031353132323032323437333810fb0e3407040408001f513131333330313230313631323238313933363439191de4ed0700041012665131313333303132303137303330333138303630301b6fd96308000418002051313133333031303230313231323138303134383232001599810800042000205131313333303130323031333034323031363131353201a70e0708050428002051313133333031303230313330393139303034373436042e69d608000430002051313133333031303230313330393139303034373438042e69da0800043800205131313333303130323031333039333030343338323904529bb30800044000205131313333303130323031333039333030343338323904529bb508040448002051313133333031303230313331303032303834303533045a338508000450002051313133333031303230313331303032303834303534045a33870800045800205131313333303130323031333132303331323135333105618852080004600020513131333330313032303135303731333130303132350db3facb080404680020513131333330313032303135303733303231323035320e2d1f90080004700020513131333330313032303135313031303033353235300f48ec6c08000478002051313133333031303230313630313039323133323338114ee6ce0800048000205131313333303130323031373031323130303530323119d48635080404880020513131333330313032303137303330373233313034381b97dfac080004900020513131333330313032303137303331333233303232361bc18a08080004980d53513131333330313032303137303331333233303232371bc18a0d090004a000215131313333303130303230313330343138303432353534018e5735090604a8002151313133333031303032303134303632353039333632300850b4db090004b00021513131333330313030323031343037313030323138323508826106090004b8002151313133333031303032303134303731313136353431350887c5a6090004c00021513131333330313030323031343037313630393430353208995d09090404c8002151313133333031303032303134313132373130343732390a867a34090004d0002151313133333031303032303135303832333232323835350e967758090004d8002151313133333031303032303135303832333232323835360e96775c090004e0002151313133333031303032303135313030383133323433360f3c8cd0090404e8002151313133333031303032303135313030383133323433370f3c8cd2090004f0002151313133333031303032303135313032323035323632310f8e960d090004f8002151313133333031303032303135313032323035323632320f8e9612090005000021513131333330313030323031363036313630303436353614ba0bb6090405080b2351313133333031303032303136303732333133343933341565b3f50900051000215131313333303130313230313330343138303432353535018e573b0900051800215131313333303130313230313330353035313530333132021a978d0900052000215131313333303130313230313330353035313530333132021a979309060528002151313133333031303132303133303531333032323430390268a9b40900053000215131313333303130313230313331313136313034373234051be2100900053800215131313333303130313230313331313136313034373235051be2160900054000215131313333303130313230313331313136313634343032051da38c09040548002151313133333031303132303133313132343231323031320540839509000550002151313133333031303132303135303531373031303434360cda328709000558002151313133333031303132303135303730313135323031300d6e444a09000560002151313133333031303132303135303732303031333433300ddf897d09040568002151313133333031303132303135303931313230303233310efce56e090005700021513131333330313031323031363031313031303330353911529c910900057800215131313333303130313230313630333031313435393534126ae7220900058000215131313333303130313230313630353137303735313334142183dd09040588002151313133333031303132303137303231383136323734381ae0c94c09000590093451313133333031303132303137303231393031323031301ae617260900059800215131313333303130323230313330343138303432353536018e5740090005a00021513131333330313032323031343037313530353531313508953225090705a800215131313333303130323230313430373135303733353335089564b4090005b000215131313333303130323230313430373135303733353437089564cc090005b8002151313133333031303232303136313031363039323530331736db61090005c00c4751313133333031303232303136313031363039323530351736db71090505c807ea5131313333303130333230313330343138303432353537018e5745090005d000215131313333303130343230313330343138303432353537018e574d090005d8002151313133333031303432303134303731353131333831320895fb6f090005e0002151313133333031303432303135313032333033313135340f94dfcd090605e8002151313133333031303432303135313032333033313230340f94dfe9090005f00021513131333330313034323031363033323530343436313512cbe0c1090005f800215131313333303130353230313330343138303432353538018e57500900060000215131313333303130363230313330343138303432353539018e5752090406080021513131333330313036323031343038323032323237353909053abf09000610002151313133333031303632303135303232363030343233380bdcedbd09000618002151313133333031303632303135303232363133303434370bde5cc70900062000215131313333303130363230313630313139313032353130117e0c900904062800215131313333303130363230313630313139313032353330117e0cb80900063000215131313333303130373230313330343138303432363030018e57570900063800215131313333303130373230313430343237303730303334074a29980900064000215131313333303130373230313630393234303530313136169344450904064800215131313333303130373230313630393234303530313231169344620900065000215131313333303130383230313330343138303432363031018e575c09000658002151313133333031303832303135303630393033303532360d3234f1090006600021513131333330313038323031353132303830393536313510ca1548090406680021513131333330313038323031353132303831353434343010cd1dc5090006700021513131333330313038323031363034323832303037323513750e6a090006780021513131333330313038323031363035303731353135353613dddf98090006800021513131333330313038323031363132323230323330333718ef48c009040688002151313133333031303832303137303331303030333534321ba657bc09000690059751313133333031303832303137303331313232333133391bb39c710900069800215131313333303130393230313330343138303432363032018e575f090006a0002151313133333031303932303133303530353133313630330219a47a090706a8002151313133333031303932303133303530353133313630340219a486090006b0002151313133333031303932303133303531333032323434310268aae5090006b800215131313333303130393230313331313136313034373236051be219090006c000215131313333303130393230313331313136313034373237051be21d090406c800215131313333303130393230313331313136313634343230051da3f0090006d00021513131333330313039323031333131323432313330353705408f7c090006d8002151313133333031303932303135303531323231353231360cce0460090006e0002151313133333031303932303135303730313135323031310d6e444e090406e8002151313133333031303932303135303732303031333433310ddf89a5090006f0002151313133333031303932303135303931303136333933380ef4367c090006f80021513131333330313039323031363031303832333230323011485b8a0900070000215131313333303130393230313630333031313530303034126ae76a09040708002151313133333031303932303136303531373037353133331421838809000710034551313133333031303932303137303230343231343633371a53d875080007180020513131333330313132303132313231383031343832320015998208000720002051313133333031313230313330313037303632373334003340f908060728002051313133333031313230313330333130313434373135009ecd6b08000730002051313133333031313230313330333130313434373136009ecd6e0800073800205131313333303131323031333033323631313439353300e2eae408000740002051313133333031313230313330343138303534313031018ea7c30804074800205131313333303131323031333034323230313532333401b7eaae0800075000205131313333303131323031333034323531393039353301d33ab40800075800205131313333303131323031333035323530303334333702b717710800076000205131313333303131323031333035323530303334343902b717ac0804076800205131313333303131323031333035323530303334353102b717bb0800077000205131313333303131323031333132313931303536343605960dcc0800077800205131313333303131323031333132313931303536343705960dd00800078000205131313333303131323031333132313931303536343805960dd40804078800205131313333303131323031343035313531343133343707ac79af0800079000205131313333303131323031343035313531343133343807ac79b20800079800205131313333303131323031343035323231303035323607d6f5b6080007a0002051313133333031313230313430363138313935353433083a3b2a080007a800205131313333303131323031343037323030393338303208a9b5c3080007b0e0ea51313133333031313230313430383238303130333132091ec6ff090007b801cd51313133333031303332303137303331383138353533331bdc59c8090007c0002151313133333031303032303137303332383137303831391c0b5803090007c8f4dd51313133333031303032303137303332383137303831391c0b5804090007d0006251313133333031303132303137303432353039323034351c80d149080007d803d9513131333330313032303137303530323233323035391ca2baae090007e0008451313133333031303932303137303530363130323234301cba9a16090007e8004251313133333031303132303137303531333131333635351ce967c1090007f001ed51313133333031303832303137303533313036333832311d57eaeb090407f8004251313133333031303132303137303630383131343234331da11e2009000800004251313133333031303932303137303631313033323233341db9ec5309000808004251313133333031303132303137303632313130333531391e05790d09040810004251313133333031303932303137303632313130333532301e05791009000818010651313133333031303132303137303632353231313430331e2e2f6f09000820010651313133333031303932303137303632393033353333381e56fb0f09000828f66a51313133333031303332303137303730323135303534381e7afa82080008300020513131333330303932303137303731303230343735391ecf67dd08000838e4f5513131333330303932303137303731303230343830301ecf67e509000840e93751313133333030393432303137303732303035313232331f3440d109000848e97951313133333030393532303137303732303035313232331f3440d509000850e9bb51313133333030393632303137303732303035313232341f3440d80900085800a551313133333031303132303137303732393038343735351f725c260900086000a551313133333031303932303137303732393038343735371f725c3909000868014851313133333031303832303137303830313031333330381f8cadc809000870002151313133333030393232303137303831303133313534321fdbda4f09000878e7cc51313133333030393232303137303831303133313534331fdbda5b09000880f41a51313133333031303132303137303831323139303035341ff2ced209000888fa295131313333303130393230313730383134323332393236200a038107000890edb9513131333330313230313730383235313234353535208386ea0900089800215131313333303039383230313730383236313633363232208f2162090008a0eac55131313333303039383230313730383236313633363332208f21b1090008a8002151313133333030383932303137303930393034323035322105a71c090008b0de4951313133333030383932303137303932333038323833352197e710090008b8f755513131333330313038323031373039333030343431333621e9b604090008c0f3da513131333330313032323031373130323330333132323722b52b04080008c8eef45131313333303130323031373131303131393437343822fc396b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000701ee61e661de61d6621101ce41c601bdc1b581ad41a5019cc194818c4184017bc173820ad16b4163015ac152814a41420139c131812971217119711171099101d0fa10f250ea70e230d9f0d1b0c970c130b8f0b0b0a870a03097f08fb087707f3076f06eb066905e9056904e9046903e9036902e9026901e9016500e10063aa0dce7c4a08a598; asc F( R}p P Q _ ]5E @$= $# 7 infimum supremum !Q1133008920160127051041 !Q1133008920160225082800 P !Q1133008920160225082801 P ( !Q1133008920161102180122 J 0 !Q1133008920161206033924 L 8 !Q1133008920161206033925 N @ !Q1133008920161206061850 g H !Q1133008920161206061851 g P !Q1133008920170212084006 X !Q1133008920170227194719 B `! Q1133008920170227203440 Bs h Q113300920121218014821 p Q113300920130322182224 9 x Q113300920130502020811 b Q113300920130502020812 b Q113300920130503062826 Q113300920130503062827 Q113300920130707032403 5 . Q113300920130709013613 : g Q113300920130902045301 I Q113300920130908103251 Q113300920130908103253 Q113300920131220090630 Q113300920131220090645 Q113300920131220090657 Q113300920131220090725 [ Q113300920131221074603 L Q113300920131221074604 L Q113300920140221202001 Q113300920140307000104 5 Q113300920140316204229 # Q113300920140523102628 t Q113300920141104230250 "1 Q113300920150310145251 % Q113300920150808111808 ? c ( Q113300920150808111808 ? e 0 Q113300920151023082944 + 8 Q113300920151115184709 ? . @ Q113300920151115184710 ? 0 H Q113300920160515192507 P Q113300920160624004253 P X Q113300920161203183247 M ` Q113300920161203183249 M h Q113300920161219085015 p Q113300920161219085016 x Q113300920170125210825 Q113300920170125212758 A Q113300920170306192010 Q113300920170306192011 !Q1133009020130418042545 W !Q1133009020130513143216 n !Q1133009020130513143226 n X !Q1133009020140601180604 !Q1133009020140601180630 0 !Q1133009020140601181041 W !Q1133009020140601181421 , !Q1133009020140601181545 V !Q1133009020140729053910 !Q1133009020150813024314 TI !Q1133009020150813024324 TI !Q1133009020160305123329 !Q1133009020160331065954 _ !Q1133009020160331065954 c !Q1133009020160714082333 J !Q1133009020170312133254 c !Q1133009020170312133255 f !Q1133009120130418042546 W ( !Q1133009120140715055048 1 0 !Q1133009120141018025033 8 !Q1133009120141018025034 @ !Q1133009120141020051323 H !Q1133009120150908000315 P !Q1133009120160225102003 PX X !Q1133009120160225102003 PX ` !Q1133009120160407033218 h !Q1133009120160407033220 p !Q1133009120160418051532 : 2 x !Q1133009220130418042547 W 4Q1133009220140824121503 !Q1133009320130418042548 W !Q1133009420130418042549 W !Q1133009420140807150019 a Q1133009420160504020442 !Q1133009520130418042550 W" !Q1133009520140807150031 a, Q1133009520160504020456 !Q1133009620130418042551 W$ !Q1133009620140807150043 a@ fQ1133009620160504020442 !Q1133009720130418042552 W) !Q1133009720131029092030 !Q1133009720131029092031 !Q1133009720160109200120 N !Q1133009720160109200121 N !Q1133009720160111105157 Z !Q1133009720160111105157 Z !Q1133009820130418042552 W. !Q1133009820130528134406 !Q1133009820140630024041 e ( !Q1133009820140701175505 l( 0 !Q1133009820140717060820 Q 8 !Q1133009820151019153443 k @ !Q1133009820151019153444 p H !Q1133009820151019153451 P !Q1133009820151023031134 X !Q1133009820151023031144 ` ;Q1133009820151112205534 (q h !Q1133009920130418042553 W2 p !Q1133009920131029092358 x !Q1133009920131029092400 !Q1133009920160110191654 V4 !Q1133009920160111105330 Z F Q11330120121125105044 O Q11330120130419144747 O Q11330120130828153608 6 Q11330120140110102753 Q11330120140110102754 Q11330120140331121423 Q11330120140423133643 < Q11330120140512181630 T Q11330120150711080109 / Q11330120150730152606 , o Q11330120150910024340 " Q11330120150910024341 1 Q11330120151019183259 { Q11330120151027181644 Q11330120151220224738 4 Q11330120161228193649 fQ11330120170303180600 o c Q113301020121218014822 Q113301020130420161152 ( Q113301020130919004746 .i 0 Q113301020130919004748 .i 8 Q113301020130930043829 R @ Q113301020130930043829 R H Q113301020131002084053 Z3 P Q113301020131002084054 Z3 X Q113301020131203121531 a R ` Q113301020150713100125 h Q113301020150730212052 - p Q113301020151010035250 H l x Q113301020160109213238 N Q113301020170121005021 5 Q113301020170307231048 Q113301020170313230226 SQ113301020170313230227 !Q1133010020130418042554 W5 !Q1133010020140625093620 P !Q1133010020140710021825 a !Q1133010020140711165415 !Q1133010020140716094052 ] !Q1133010020141127104729 z4 !Q1133010020150823222855 wX !Q1133010020150823222856 w\ !Q1133010020151008132436 < !Q1133010020151008132437 < !Q1133010020151022052621 !Q1133010020151022052622 !Q1133010020160616004656 #Q1133010020160723134934 e !Q1133010120130418042555 W; !Q1133010120130505150312 !Q1133010120130505150312 ( !Q1133010120130513022409 h 0 !Q1133010120131116104724 8 !Q1133010120131116104725 @ !Q1133010120131116164402 H !Q1133010120131124212012 @ P !Q1133010120150517010446 2 X !Q1133010120150701152010 nDJ ` !Q1133010120150720013430 } h !Q1133010120150911200231 n p !Q1133010120160110103059 R x !Q1133010120160301145954 j " !Q1133010120160517075134 ! !Q1133010120170218162748 L 4Q1133010120170219012010 & !Q1133010220130418042556 W@ !Q1133010220140715055115 2% !Q1133010220140715073535 d !Q1133010220140715073547 d !Q1133010220161016092503 6 a GQ1133010220161016092505 6 q Q1133010320130418042557 WE !Q1133010420130418042557 WM !Q1133010420140715113812 o !Q1133010420151023031154 !Q1133010420151023031204 !Q1133010420160325044615 !Q1133010520130418042558 WP !Q1133010620130418042559 WR !Q1133010620140820222759 : !Q1133010620150226004238 !Q1133010620150226130447 \ !Q1133010620160119102510 ~ ( !Q1133010620160119102530 ~ 0 !Q1133010720130418042600 WW 8 !Q1133010720140427070034 J) @ !Q1133010720160924050116 DE H !Q1133010720160924050121 Db P !Q1133010820130418042601 W\ X !Q1133010820150609030526 24 ` !Q1133010820151208095615 H h !Q1133010820151208154440 p !Q1133010820160428200725 u j x !Q1133010820160507151556 !Q1133010820161222023037 H !Q1133010820170310003542 W Q1133010820170311223139 q !Q1133010920130418042602 W_ !Q1133010920130505131603 z !Q1133010920130505131604 !Q1133010920130513022441 h !Q1133010920131116104726 !Q1133010920131116104727 !Q1133010920131116164420 !Q1133010920131124213057 @ | !Q1133010920150512215216 ` !Q1133010920150701152011 nDN !Q1133010920150720013431 !Q1133010920150910163938 6| !Q1133010920160108232020 H[ !Q1133010920160301150004 j j !Q1133010920160517075133 ! EQ1133010920170204214637 S u Q113301120121218014822 Q113301120130107062734 3@ ( Q113301120130310144715 k 0 Q113301120130310144716 n 8 Q113301120130326114953 @ Q113301120130418054101 H Q113301120130422015234 P Q113301120130425190953 : X Q113301120130525003437 q ` Q113301120130525003449 h Q113301120130525003451 p Q113301120131219105646 x Q113301120131219105647 Q113301120131219105648 Q113301120140515141347 y Q113301120140515141348 y Q113301120140522100526 Q113301120140618195543 :;* Q113301120140720093802 Q113301120140828010312 Q1133010320170318185533 Y !Q1133010020170328170819 X Q1133010020170328170819 X bQ1133010120170425092045 I Q113301020170502232059 Q1133010920170506102240 BQ1133010120170513113655 g Q1133010820170531063821 W BQ1133010120170608114243 BQ1133010920170611032234 S BQ1133010120170621103519 y BQ1133010920170621103520 y Q1133010120170625211403 ./o Q1133010920170629035338 V ( jQ1133010320170702150548 z 0 Q113300920170710204759 g 8 Q113300920170710204800 g @ 7Q1133009420170720051223 4@ H yQ1133009520170720051223 4@ P Q1133009620170720051224 4@ X Q1133010120170729084755 r\& ` Q1133010920170729084757 r\9 h HQ1133010820170801013308 p !Q1133009220170810131542 O x Q1133009220170810131543 [ Q1133010120170812190054 )Q1133010920170814232926 Q11330120170825124555 !Q1133009820170826163622 !b Q1133009820170826163632 ! !Q1133008920170909042052! IQ1133008920170923082835! UQ1133010820170930044136! Q1133010220171023031227" + Q113301020171101194748" 9k p f f! ` X P H @ 8 0 ( % # w o i i i i i e c |J ;
210InnoDB: End of page dump
2112017-11-17 08:42:34 7f949f3fd700 InnoDB: uncompressed page, stored checksum in field1 3573630504, calculated checksums for field1: crc32 4141865058, innodb 306254111, none 3735928559, stored checksum in field2 2853031548, calculated checksums for field2: crc32 4141865058, innodb 3543893859, none 3735928559, page LSN 3679 2499960117, low 4 bytes of LSN at page end 1242080664, page number (if stored to page already) 17083931, space id (if created with >= MySQL-4.1.1 and stored already) 178
212InnoDB: Page may be an index page where index id is 521
213InnoDB: Database page corruption on disk or a failed
214InnoDB: file read of page 17083931.
215InnoDB: You may have to recover from a backup.
216InnoDB: It is also possible that your operating
217InnoDB: system has corrupted its own file cache
218InnoDB: and rebooting your computer removes the
219InnoDB: error.
220InnoDB: If the corrupt page is an index page
221InnoDB: you can also try to fix the corruption
222InnoDB: by dumping, dropping, and reimporting
223InnoDB: the corrupt table. You can use CHECK
224InnoDB: TABLE to scan your table for corruption.
225InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
226InnoDB: about forcing recovery.
227InnoDB: Ending processing because of a corrupt database page.
2282017-11-17 08:42:34 7f949f3fd700 InnoDB: Assertion failure in thread 140276303648512 in file buf0buf.cc line 4527
229InnoDB: We intentionally generate a memory trap.
230InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
231InnoDB: If you get repeated assertion failures or crashes, even
232InnoDB: immediately after the mysqld startup, there may be
233InnoDB: corruption in the InnoDB tablespace. Please refer to
234InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
235InnoDB: about forcing recovery.
236171117 8:42:34 [ERROR] mysqld got signal 6 ;
237This could be because you hit a bug. It is also possible that this binary
238or one of the libraries it was linked against is corrupt, improperly built,
239or misconfigured. This error can also be caused by malfunctioning hardware.
240
241To report this bug, see https://mariadb.com/kb/en/reporting-bugs
242
243We will try our best to scrape up some info that will hopefully help
244diagnose the problem, but since we have already crashed,
245something is definitely wrong and this may fail.
246
247Server version: 10.0.29-MariaDB
248key_buffer_size=134217728
249read_buffer_size=131072
250max_used_connections=0
251max_threads=2010
252thread_count=0
253It is possible that mysqld could use up to
254key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4703233 K bytes of memory
255Hope that's ok; if not, decrease some variables in the equation.
256
257Thread pointer: 0x0
258Attempting backtrace. You can use the following information to find out
259where mysqld died. If you see no messages after this, something went
260terribly wrong...
261stack_bottom = 0x0 thread_stack 0x30000
262mysys/stacktrace.c:268(my_print_stacktrace)[0xbc3ebe]
263sql/signal_handler.cc:159(handle_fatal_signal)[0x7378ef]
264/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7fada9818890]
265/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fada8175067]
266/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fada8176448]
267buf/buf0buf.cc:4527(buf_page_io_complete(buf_page_t*))[0x9d6f4b]
268fil/fil0fil.cc:5825(fil_aio_wait(unsigned long))[0xa21a36]
269srv/srv0start.cc:530(io_handler_thread)[0x971075]
270/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7fada9811064]
271/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fada822862d]
272The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
273information that should help you find out what is causing the crash.
274171117 08:42:35 mysqld_safe mysqld from pid file /srv/sqldata/db1063.pid ended

• Marostegui updated the task description. (Show Details)Nov 17 2017, 8:45 AM
• Marostegui updated the task description. (Show Details)Nov 17 2017, 9:23 AM
• Marostegui updated the task description. (Show Details)Nov 17 2017, 9:36 AM
gerritbot added a comment.Nov 17 2017, 9:56 AM

Change 392017 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Restore original weight for s5 hosts

https://gerrit.wikimedia.org/r/392017

gerritbot added a comment.Nov 17 2017, 9:58 AM

Change 392017 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Restore original weight for s5 hosts

https://gerrit.wikimedia.org/r/392017

• Marostegui added a comment.Nov 17 2017, 10:41 AM

I have copied db1063's binlogs over to:

root@dbstore1001:/srv/tmp/T180714# ls -lh
total 21G
-rw-r--r-- 1 root root 21G Nov 17 10:39 db1063_binlogs.tar.gz

MySQL is unusable, but at least we can keep the binlogs for a while just in case.
I suggest this host gets reimaged and used for something like vslow but never as master again.
Also checking for badblocks on it, to see if something gets reported.

• Marostegui updated the task description. (Show Details)Nov 17 2017, 10:41 AM
Chicocvenancio subscribed.Nov 17 2017, 3:04 PM
Stashbot added a comment.Nov 17 2017, 5:12 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-17T17:12:51Z] <marostegui> Move dbstore1001 under db1070 - T180714

Stashbot added a comment.Nov 17 2017, 5:14 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-17T17:14:51Z] <marostegui> Revert schema change on dbstore1001 - T180714

• Marostegui updated the task description. (Show Details)Nov 17 2017, 5:22 PM
• Marostegui updated the task description. (Show Details)Nov 17 2017, 5:40 PM
• Marostegui added a comment.Nov 18 2017, 8:34 AM

The schema change revert has finished on codfw master and replication has been started correctly.
The slaves are now getting the alters, and they should be done in the next 24h or so and replication will also resume once they are done.

• Marostegui updated the task description. (Show Details)Nov 18 2017, 8:35 AM
• Marostegui added a comment.Nov 18 2017, 8:15 PM

The badblocks run on db1063 has been finished with no badblocks reported.
I will reimage db1063 on Monday and rebuild it as vslow (reminder, binlogs have been saved: T180714#3769251)

• MZMcBride subscribed.Nov 19 2017, 5:24 PM
ops-monitoring-bot subscribed.Nov 20 2017, 6:29 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db1063.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201711200629_marostegui_24943_db1063_eqiad_wmnet.log.

gerritbot added a comment.Nov 20 2017, 7:24 AM

Change 392374 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db1063

https://gerrit.wikimedia.org/r/392374

gerritbot added a comment.Nov 20 2017, 7:26 AM

Change 392374 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db1063

https://gerrit.wikimedia.org/r/392374

ops-monitoring-bot added a comment.Nov 20 2017, 7:27 AM

Completed auto-reimage of hosts:

['db1063.eqiad.wmnet']

Of which those FAILED:

['db1063.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 20 2017, 7:27 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db1063.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201711200727_marostegui_4647_db1063_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 20 2017, 7:27 AM

Completed auto-reimage of hosts:

['db1063.eqiad.wmnet']

Of which those FAILED:

['db1063.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 20 2017, 7:28 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db1063.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201711200727_marostegui_4743_db1063_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 20 2017, 7:52 AM

Completed auto-reimage of hosts:

['db1063.eqiad.wmnet']

and were ALL successful.

• Marostegui mentioned this in T109179: Migrate MySQLs to use ROW-based replication.Nov 20 2017, 8:28 AM
• Marostegui added a comment.Nov 20 2017, 12:34 PM

db1063 has been rebuilt and it is now catching up.
Before putting it back as vslow, I am going to optimize wb_terms table as we have been doing with the rest of the hosts.

• Marostegui updated the task description. (Show Details)Nov 20 2017, 12:35 PM
• Marostegui added a comment.Nov 20 2017, 12:53 PM

The scope of this ticket is actually all done.
So I would suggest we close it and do any amends or follow ups taken from the IR: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171116-s5-dewiki-wikidata in different tickets.

• Marostegui closed this task as Resolved.Nov 21 2017, 7:27 AM
Liuxinyu970226 unsubscribed.Nov 21 2017, 9:53 AM
RandomDSdevel awarded a token.Nov 22 2017, 11:10 PM
• Marostegui mentioned this in T186321: Prepare and indicate proper master db failover candidates for all eqiad database sections (s1-s8, x1).Feb 5 2018, 9:39 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL