⚓ T149526 es2019 crashed again

	Project	Branch	Lines +/-	Subject
	operations/mediawiki-config	master	+1 -1	db-codfw.php: Repool es2019

		Status	Subtype	Assigned	Task
		Resolved		None	T130702 Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March
		Resolved		jcrespo	T149526 es2019 crashed again

jcrespo created this task.Oct 30 2016, 4:32 PM

Restricted Application added a project: SRE. · View Herald TranscriptOct 30 2016, 4:32 PM

Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-operations) [2016-10-30T16:35:25Z] <jynus> powercycle es2019 after crash T149526

No trace of HW logs I assume? :(

Fri May 27 2016 17:31:34	Correctable memory error rate exceeded for DIMM_A1.
Fri May 27 2016 18:42:52	Correctable memory error rate exceeded for DIMM_A1.
Tue Jun 07 2016 17:21:45	Correctable memory error rate exceeded for DIMM_A1.	
Tue Jun 07 2016 17:27:14	Correctable memory error rate exceeded for DIMM_A1.	
Tue Jun 07 2016 17:27:31	Correctable memory error rate exceeded for DIMM_A1.	
Sun Oct 30 2016 15:14:10	Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
Sun Oct 30 2016 15:14:13	CPU 2 has an internal error (IERR).

MEM0001: Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
 2016-10-30T15:14:10-0500
Log Sequence Number: 362
Detailed Description:
The memory has encountered a uncorrectable error. System performance may be degraded. The operating system and/or applications may fail as a result.
Recommended Action:
Re-install the memory component. If the problem persists, contact technical support. Refer to the product documentation to choose a convenient contact method.
Comment: root

CPU0000: CPU 2 has an internal error (IERR).
 2016-10-30T15:14:14-0500
Log Sequence Number: 366
Detailed Description:
System event log and OS logs may indicate that the exception is external to the processor.
Recommended Action:
Review System Event Log and Operating System Logs. If the issue persists, contact technical support. Refer to the product documentation to choose a convenient contact method.
Comment: root

This is from the hardware logs

The last time we replaced the memory on DIMM A1 on this system. What i will do is to swap DIMM B2 with DIMM B1 and clean the logs. If the message show on DIMM B2 then I will request new memory.

Open to schedule it when you have the time. Should we update the BIOS (again?). CC'ing @RobH here.

I would upgrade the BIOS if we need to reboot the server anyways.

MoritzMuehlenhoff assigned this task to Papaul.Nov 3 2016, 3:53 PM

Papaul triaged this task as Medium priority.Nov 16 2016, 3:46 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-21T15:58:12Z] <marostegui> Shutting down MySQL es2019 for HW maintenance - T149526

Mentioned in SAL (#wikimedia-operations) [2016-11-21T15:59:31Z] <marostegui> Powering off es2019 for HW maintenance - T149526

1- Swapped CPU 2 to CPU1
2 - Update BIOS from 2.1.6 to 2.2.5
3- Clear syslog

Leaving this task open for now .

Thanks. I have started MySQL and replication again - we will see how it behaves!

This ask has been open for almost 2 months. closing it. it can be reopen anytime we have the issue again.

jcrespo reopened this task as Open.Apr 22 2017, 7:38 AM

Server crashed again as per: T130702#3203266

Normal","Mon Nov 21 2016 16:22:07","Log cleared.
Critical","Sat Apr 22 2017 07:52:10","CPU 2 has an internal error (IERR).
Normal","Sat Apr 22 2017 06:56:24","A problem was detected related to the previous server boot.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_B4.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.

elukey added a subscriber: elukey.Apr 22 2017, 7:39 AM

This comment was removed by elukey.

Severity Date and Time Message ID Summary Comment
2017-04-22T07:37:48-0500 USR0032
The session for root from 10.64.32.20 using SSH is logged off.

2017-04-22T07:37:41-0500 USR0030
Successfully logged in using root, from 10.192.16.172 and GUI.

2017-04-22T07:36:16-0500 USR0030
Successfully logged in using root, from 10.64.32.20 and SSH.

2017-04-22T07:36:08-0500 USR0032
The session for root from 208.80.154.149 using SSH is logged off.

2017-04-22T07:30:25-0500 USR0032
The session for root from 10.64.32.20 using SSH is logged off.

2017-04-22T07:27:55-0500 USR0030
Successfully logged in using root, from 208.80.154.149 and SSH.

2017-04-22T07:22:50-0500 USR0030
Successfully logged in using root, from 10.64.32.20 and SSH.

2017-04-22T06:56:31-0500 MEM0001
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.

2017-04-22T06:56:30-0500 MEM0001
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.

2017-04-22T06:56:28-0500 MEM0001
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.

2017-04-22T06:56:27-0500 MEM0001
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.

2017-04-22T06:56:26-0500 MEM0001
Multi-bit memory errors detected on a memory device at location(s) DIMM_B4.

2017-04-22T06:56:24-0500 UEFI0079
One or more Uncorrectable Memory errors occurred in the previous boot.

2017-04-22T06:56:24-0500 PST0090
A problem was detected related to the previous server boot.

2017-04-22T07:52:10-0500 CPU0000
CPU 2 has an internal error (IERR).

2017-04-22T07:52:10-0500 SYS1003
System CPU Resetting.

2017-04-22T07:52:09-0500 RAC0703
Requested system hardreset.

2017-04-22T07:52:09-0500 SYS1003
System CPU Resetting.

2017-04-22T03:58:35-0500 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-04-15T08:15:00-0500 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-04-15T03:58:37-0500 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-04-08T08:10:41-0500 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-04-08T03:58:44-0500 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-04-01T08:10:57-0500 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-04-01T03:58:49-0500 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-03-25T08:12:35-0500 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-03-25T03:58:52-0500 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-03-18T08:10:54-0500 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-03-18T03:58:55-0500 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-03-11T07:10:17-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-03-11T02:59:02-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-03-04T07:20:34-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-03-04T02:59:05-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-02-25T07:14:41-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-02-25T02:59:06-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-02-18T07:18:58-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-02-18T02:59:12-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-02-11T07:18:01-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-02-11T02:59:14-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-02-04T07:11:28-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-02-04T02:59:20-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-01-28T07:19:12-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-01-28T02:59:22-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-01-21T07:11:32-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-01-21T02:59:31-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-01-14T07:15:38-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-01-14T02:59:34-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2017-01-07T07:14:37-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2017-01-07T02:59:35-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2016-12-31T07:10:41-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2016-12-31T02:59:43-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2016-12-24T07:14:53-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2016-12-24T02:59:47-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2016-12-17T07:14:41-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2016-12-17T02:59:50-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2016-12-10T07:13:03-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2016-12-10T02:59:51-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2016-12-03T07:09:55-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2016-12-03T02:59:55-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2016-11-26T07:14:52-0600 CTL38
The Patrol Read operation completed for Integrated RAID Controller 1.

2016-11-26T03:00:00-0600 CTL37
A Patrol Read operation started for Integrated RAID Controller 1.

2016-11-22T19:30:50-0600 USR0032
The session for root from 208.80.154.149 using SSH is logged off.

2016-11-22T19:30:45-0600 JCP036
The (installation or configuration) job JID_798645851255 is successfully deleted.

2016-11-22T19:29:45-0600 JCP027
Job created successfully.

2016-11-22T19:29:37-0600 USR0030
Successfully logged in using root, from 208.80.154.149 and SSH.

2016-11-22T19:29:37-0600 LOG007
The previous log entry was repeated 1 times.

2016-11-22T19:28:31-0600 SYS1003
System CPU Resetting.

2016-11-22T19:24:13-0600 JCP036
The (installation or configuration) job JID_798641868867 is successfully deleted.

2016-11-22T19:23:51-0600 USR0032
The session for root from 208.80.154.149 using SSH is logged off.

2016-11-22T19:23:06-0600 JCP027
Job created successfully.

2016-11-22T19:23:02-0600 USR0030
Successfully logged in using root, from 208.80.154.149 and SSH.

2016-11-22T19:23:02-0600 LOG007
The previous log entry was repeated 1 times.

2016-11-22T19:21:52-0600 SYS1003
System CPU Resetting.

2016-11-21T17:11:23-0600 USR0032
The session for root from 208.80.153.5 using SSH is logged off.

2016-11-21T17:11:03-0600 USR0032
The session for root from 10.193.0.7 using GUI is logged off.

2016-11-21T17:09:41-0600 USR0174
The Front Panel USB device is removed from the operating system.

2016-11-21T17:09:07-0600 NIC101
The NIC Integrated 1 Port 1 network link is started.

2016-11-21T17:08:26-0600 USR0030
Successfully logged in using root, from 208.80.153.5 and SSH.

2016-11-21T17:08:15-0600 USR0032
The session for root from 91.198.174.112 using SSH is logged off.

2016-11-21T17:08:13-0600 USR0032
The session for root from 208.80.153.5 using SSH is logged off.

2016-11-21T17:08:13-0600 LOG007
The previous log entry was repeated 1 times.

2016-11-21T17:02:30-0600 USR0030
Successfully logged in using root, from 208.80.153.5 and SSH.

2016-11-21T17:00:04-0600 USR0030
Successfully logged in using root, from 10.193.0.7 and GUI.

2016-11-21T17:00:04-0600 LOG007
The previous log entry was repeated 1 times.

2016-11-21T16:59:16-0600 USR0030
Successfully logged in using root, from 208.80.153.5 and SSH.

2016-11-21T16:58:51-0600 USR0032
The session for root from 10.193.0.7 using GUI is logged off.

2016-11-21T16:58:48-0600 JCP036
The (installation or configuration) job JID_797690674507 is successfully deleted.

2016-11-21T16:57:47-0600 JCP027
Job created successfully.

2016-11-21T16:57:28-0600 USR0173
The Front Panel USB port switched automatically from iDRAC to operating system.

2016-11-21T16:57:28-0600 USR0171
The Front Panel USB port is detached from the iDRAC Disk.USBFront.1. Device Details: Device Class 3, Vendor ID 0B38, Product ID 0010.

2016-11-21T16:57:23-0600 USR0170
The Front Panel USB port is attached to iDRAC Disk.USBFront.1. Device details: Device class 3, Vendor ID 0B38, Manufacturer Name Not Available, Product ID 0010, Product Name Not Available, Serial Number Not Available.

2016-11-21T16:57:15-0600 PR36
Version change detected for BIOS firmware. Previous version:2.1.6, Current version:2.2.5

2016-11-21T16:55:55-0600 USR0030
Successfully logged in using root, from 208.80.153.5 and SSH.

2016-11-21T16:55:52-0600 USR0031
Unable to log in for root from 208.80.153.5 using SSH.

2016-11-21T16:55:41-0600 USR0030
Successfully logged in using root, from 208.80.153.5 and SSH.

2016-11-21T16:55:35-0600 USR0031
Unable to log in for root from 208.80.153.5 using SSH.

2016-11-21T16:55:35-0600 LOG007
The previous log entry was repeated 1 times.

2016-11-21T16:55:26-0600 SYS1003
System CPU Resetting.

2016-11-21T16:55:25-0600 SYS1000
System is turning on.

2016-11-21T16:55:16-0600 SYS1003
System CPU Resetting.

Looks like the memory errors are not new:

https://phabricator.wikimedia.org/T130702#2330334

The memory module on this host apparently was replaced: T149526#2756898 maybe memory slot is what is broken?

For checking later:

P5309 ids at the time of the es2019 crash

1	$ cat *.dblist \| while read db; do mysql -BN -A -h es2019.codfw.wmnet $db -e "SELECT '$db', max(blob_id) FROM blobs_cluster25"; done
2	enwiki 137353196
3	bgwiki 1424466
4	bgwiktionary 33930
5	cswiki 3085923
6	enwikiquote 386310
7	enwiktionary 12334923
8	eowiki 789524
9	fiwiki 2238381
10	idwiki 3505748
11	itwiki 17523821
12	nlwiki 8367741
13	nowiki 3092125
14	plwiki 8287923
15	ptwiki 8959556
16	svwiki 11156711
17	thwiki 1401691
18	trwiki 3784403
19	zhwiki 11290429
20	aawiki 175
21	aawikibooks 25
22	aawiktionary 44
23	abwiki 6357
24	abwiktionary 29
25	acewiki 17137
26	advisorywiki 17
27	adywiki 3315
28	afwiki 263642
29	afwikibooks 310
30	afwikiquote 633
31	afwiktionary 12385
32	akwiki 2539
33	akwikibooks 34
34	akwiktionary 22
35	alswiki 214056
36	alswikibooks 36
37	alswikiquote 23
38	alswiktionary 34
39	amwiki 33514
40	amwikiquote 116
41	amwiktionary 2009
42	angwiki 17329
43	angwikibooks 318
44	angwikiquote 22
45	angwikisource 20
46	angwiktionary 5799
47	anwiki 115477
48	anwiktionary 6094
49	arbcom_dewiki 12362
50	arbcom_enwiki 7400
51	arbcom_fiwiki 282
52	arbcom_nlwiki 11117
53	arcwiki 5976
54	arwikibooks 17326
55	arwikimedia 117
56	arwikinews 17536
57	arwikiquote 13891
58	arwikisource 49101
59	arwikiversity 15499
60	arwiktionary 74220
61	arzwiki 143132
62	astwiki 188653
63	astwikibooks 32
64	astwikiquote 28
65	astwiktionary 37490
66	aswiki 36709
67	aswikibooks 36
68	aswikisource 5055
69	aswiktionary 25
70	auditcomwiki NULL
71	avwiki 10878
72	avwiktionary 20
73	aywiki 8257
74	aywikibooks 20
75	aywiktionary 2860
76	azbwiki 70378
77	azwiki 815826
78	azwikibooks 7558
79	azwikiquote 16708
80	azwikisource 18677
81	azwiktionary 116848
82	barwiki 176315
83	bat_smgwiki 28523
84	bawiki 262373
85	bawikibooks 29
86	bclwiki 19305
87	bdwikimedia 1373
88	betawikiversity 46541
89	bewiki 661474
90	bewikibooks 413
91	bewikimedia 5002
92	bewikiquote 1194
93	bewikisource 8525
94	bewiktionary 9131
95	be_x_oldwiki 247101
96	bgwikibooks 950
97	bgwikinews 1443
98	bgwikiquote 8891
99	bgwikisource 3433
100	bhwiki 174313
101	bhwiktionary 28
102	biwiki 4172
103	biwikibooks 16
104	biwiktionary 25
105	bjnwiki 9662
106	bmwiki 2438
107	bmwikibooks 14
108	bmwikiquote 16
109	bmwiktionary 46
110	bnwiki 699259
111	bnwikibooks 4794
112	bnwikisource 368017
113	bnwiktionary 6760
114	boardgovcomwiki NULL
115	boardwiki 3612
116	bowiki 12480
117	bowikibooks 49
118	bowiktionary 28
119	bpywiki 46508
120	brwiki 210090
121	brwikimedia 5407
122	brwikiquote 373
123	brwikisource 29032
124	brwiktionary 58315
125	bswiki 468078
126	bswikibooks 530
127	bswikinews 4527
128	bswikiquote 5504
129	bswikisource 759
130	bswiktionary 15772
131	bugwiki 28599
132	bxrwiki 14927
133	cawikibooks 14227
134	cawikimedia 5216
135	cawikinews 47542
136	cawikiquote 32226
137	cawikisource 23349
138	cawiktionary 343856
139	cbk_zamwiki 16210
140	cdowiki 9830
141	cebwiki 6354983
142	cewiki 869126
143	chairwiki NULL
144	chapcomwiki 3969
145	checkuserwiki 7267
146	chowiki 98
147	chrwiki 3780
148	chrwiktionary 289854
149	chwiki 2195
150	chwikibooks 37
151	chwiktionary 21
152	chywiki 2631
153	ckbwiki 188087
154	cnwikimedia 94
155	collabwiki 18221
156	cowiki 26179
157	cowikibooks 53
158	cowikimedia 599
159	cowikiquote 28
160	cowiktionary 15302
161	crhwiki 15247
162	crwiki 4581
163	crwikiquote 29
164	crwiktionary 34
165	csbwiki 13947
166	csbwiktionary 4245
167	cswikibooks 5172
168	cswikinews 15795
169	cswikiquote 18037
170	cswikisource 41783
171	cswikiversity 29015
172	cswiktionary 247435
173	cuwiki 5736
174	cvwiki 81445
175	cvwikibooks 298
176	cywiki 592309
177	cywikibooks 312
178	cywikiquote 755
179	cywikisource 661
180	cywiktionary 37725
181	dawiki 1349235
182	dawikibooks 5047
183	dawikiquote 2552
184	dawikisource 7415
185	dawiktionary 81201
186	dewikibooks 88669
187	dewikinews 69460
188	dewikiquote 16715
189	dewikisource 561946
190	dewikiversity 84366
191	dewikivoyage 277541
192	dewiktionary 1402174
193	diqwiki 72718
194	dkwikimedia 1383
195	donatewiki 2215
196	dsbwiki 14589
197	dvwiki 7369
198	dvwiktionary 1385
199	dzwiki 1725
200	dzwiktionary 38
201	ecwikimedia 6
202	eewiki 3147
203	elwiki 1442861
204	elwikibooks 14517
205	elwikinews 8493
206	elwikiquote 4010
207	elwikisource 20226
208	elwikiversity 2148
209	elwikivoyage 10632
210	elwiktionary 487387
211	emlwiki 25590
212	enwikibooks 425730
213	enwikinews 1342139
214	enwikisource 1375680
215	enwikiversity 373179
216	enwikivoyage 612000
217	eowikibooks 4221
218	eowikinews 56082
219	eowikiquote 2973
220	eowikisource 6086
221	eowiktionary 144140
222	eswikibooks 70592
223	eswikinews 132928
224	eswikiquote 48085
225	eswikisource 146939
226	eswikiversity 31304
227	eswikivoyage 69212
228	eswiktionary 1708537
229	etwiki 808631
230	etwikibooks 1203
231	etwikimedia 486
232	etwikiquote 1446
233	etwikisource 15072
234	etwiktionary 171069
235	euwiki 1338300
236	euwikibooks 2014
237	euwikiquote 882
238	euwiktionary 107021
239	execwiki NULL
240	extwiki 9676
241	fawikibooks 27219
242	fawikinews 47312
243	fawikiquote 34912
244	fawikisource 8443
245	fawikivoyage 46335
246	fawiktionary 109214
247	fdcwiki 2788
248	ffwiki 1954
249	fiu_vrowiki 9759
250	fiwikibooks 13103
251	fiwikimedia 1156
252	fiwikinews 2792
253	fiwikiquote 8921
254	fiwikisource 14124
255	fiwikiversity 11067
256	fiwikivoyage 26409
257	fiwiktionary 703411
258	fjwiki 2183
259	fjwiktionary 57484
260	foundationwiki 13492
261	fowiki 51520
262	fowikisource 254
263	fowiktionary 2926
264	frpwiki 9664
265	frrwiki 35286
266	frwikibooks 94786
267	frwikinews 152761
268	frwikiquote 45319
269	frwikisource 1535610
270	frwikiversity 164844
271	frwikivoyage 123641
272	furwiki 9067
273	fywiki 124547
274	fywikibooks 265
275	fywiktionary 14150
276	gagwiki 9099
277	ganwiki 28455
278	gawiki 113844
279	gawikibooks 42
280	gawikiquote 45
281	gawiktionary 7730
282	gdwiki 67178
283	gdwiktionary 9167
284	glkwiki 6747
285	glwiki 855346
286	glwikibooks 2717
287	glwikiquote 3964
288	glwikisource 3899
289	glwiktionary 79721
290	gnwiki 10396
291	gnwikibooks 15
292	gnwiktionary 861
293	gomwiki 85035
294	gotwiki 5616
295	gotwikibooks 45
296	grantswiki NULL
297	guwiki 136188
298	guwikibooks 51
299	guwikiquote 5609
300	guwikisource 29361
301	guwiktionary 2732
302	gvwiki 23876
303	gvwiktionary 2174
304	hakwiki 17926
305	hawiki 4740
306	hawiktionary 761
307	hawwiki 7227
308	hewikibooks 24164
309	hewikinews 5290
310	hewikiquote 15273
311	hewikisource 155373
312	hewikivoyage 84413
313	hewiktionary 55909
314	hifwiki 41629
315	hiwiki 787713
316	hiwikibooks 2017
317	hiwikiquote 5787
318	hiwiktionary 134217
319	howiki 85
320	hrwiki 607483
321	hrwikibooks 1367
322	hrwikiquote 4667
323	hrwikisource 4002
324	hrwiktionary 49510
325	hsbwiki 38785
326	hsbwiktionary 1981
327	htwiki 63721
328	htwikisource 27
329	huwikibooks 63263
330	huwikinews 121
331	huwikiquote 4607
332	huwikisource 5124
333	huwiktionary 439224
334	hywiki 2119469
335	hywikibooks 1312
336	hywikiquote 25734
337	hywikisource 44438
338	hywiktionary 506550
339	hzwiki 88
340	iawiki 70723
341	iawikibooks 340
342	iawiktionary 4121
343	idwikibooks 14332
344	idwikiquote 4273
345	idwikisource 5172
346	idwiktionary 100390
347	iegcomwiki 452
348	iewiki 11181
349	iewikibooks 219
350	iewiktionary 2461
351	igwiki 5856
352	iiwiki 68
353	ikwiki 2861
354	ikwiktionary 91
355	ilowiki 78388
356	ilwikimedia NULL
357	incubatorwiki 1557201
358	internalwiki 91
359	iowiki 78551
360	iowiktionary 324191
361	iswiki 133026
362	iswikibooks 2020
363	iswikiquote 597
364	iswikisource 3443
365	iswiktionary 21305
366	itwikibooks 47458
367	itwikinews 20983
368	itwikiquote 164536
369	itwikisource 358710
370	itwikiversity 37826
371	itwikivoyage 181264
372	itwiktionary 729483
373	iuwiki 2345
374	iuwiktionary 1114
375	jamwiki 9094
376	jawikibooks 19686
377	jawikinews 21255
378	jawikiquote 3656
379	jawikisource 20230
380	jawikiversity 1724
381	jawiktionary 219500
382	jbowiki 5571
383	jbowiktionary 2258
384	jvwiki 235927
385	jvwiktionary 50360
386	kaawiki 7160
387	kabwiki 12592
388	kawiki 573117
389	kawikibooks 1550
390	kawikiquote 6290
391	kawiktionary 16945
392	kbdwiki 8840
393	kgwiki 3989
394	kiwiki 3037
395	kjwiki 55
396	kkwiki 738234
397	kkwikibooks 686
398	kkwikiquote 14
399	kkwiktionary 13573
400	klwiki 5463
401	klwiktionary 2812
402	kmwiki 38946
403	kmwikibooks 1411
404	kmwiktionary 25675
405	knwiki 246607
406	knwikibooks 43
407	knwikiquote 1209
408	knwikisource 34891
409	knwiktionary 101903
410	koiwiki 4914
411	kowikibooks 5349
412	kowikinews 7398
413	kowikiquote 4289
414	kowikisource 52761
415	kowikiversity 7509
416	kowiktionary 1041061
417	krcwiki 11031
418	krwiki 114
419	krwikiquote 13
420	kshwiki 9956
421	kswiki 2000
422	kswikibooks 32
423	kswikiquote 15
424	kswiktionary 470
425	kuwiki 92048
426	kuwikibooks 479
427	kuwikiquote 722
428	kuwiktionary 707518
429	kvwiki 14475
430	kwwiki 10487
431	kwwikiquote 32
432	kwwiktionary 1919
433	kywiki 77547
434	kywikibooks 301
435	kywikiquote 276
436	kywiktionary 51983
437	ladwiki 27329
438	lawiki 549203
439	lawikibooks 370
440	lawikiquote 1574
441	lawikisource 12923
442	lawiktionary 30102
443	lbewiki 5178
444	lbwiki 309650
445	lbwikibooks 20
446	lbwikiquote 45
447	lbwiktionary 18129
448	legalteamwiki 3894
449	lezwiki 20547
450	lgwiki 4123
451	lijwiki 12745
452	liwiki 47015
453	liwikibooks 219
454	liwikiquote 794
455	liwikisource 262
456	liwiktionary 67440
457	lmowiki 98848
458	lnwiki 8482
459	lnwikibooks 50
460	lnwiktionary 1607
461	loginwiki 10
462	lowiki 9568
463	lowiktionary 60478
464	lrcwiki 35810
465	ltgwiki 3503
466	ltwiki 566559
467	ltwikibooks 1367
468	ltwikiquote 8300
469	ltwikisource 1462
470	ltwiktionary 149600
471	lvwiki 453022
472	lvwikibooks 44
473	lvwiktionary 21228
474	maiwiki 69622
475	map_bmswiki 18307
476	mdfwiki 4654
477	mediawikiwiki 1065341
478	mgwiki 149533
479	mgwikibooks 180
480	mgwiktionary 7051401
481	mhrwiki 32061
482	mhwiki 82
483	mhwiktionary 17
484	minwiki 771996
485	miwiki 8718
486	miwikibooks 48
487	miwiktionary 3215
488	mkwiki 436330
489	mkwikibooks 961
490	mkwikimedia 288
491	mkwikisource 1883
492	mkwiktionary 13204
493	mlwiki 556570
494	mlwikibooks 3992
495	mlwikiquote 2335
496	mlwikisource 46082
497	mlwiktionary 79097
498	mnwiki 109247
499	mnwikibooks 64
500	mnwiktionary 111381
501	movementroleswiki NULL
502	mowiki 226
503	mowiktionary 21
504	mrjwiki 15607
505	mrwiki 238074
506	mrwikibooks 774
507	mrwikiquote 575
508	mrwikisource 5885
509	mrwiktionary 1788
510	mswiki 805382
511	mswikibooks 1625
512	mswiktionary 8245
513	mtwiki 30061
514	mtwiktionary 4476
515	muswiki 59
516	mwlwiki 9962
517	mxwikimedia 1795
518	myvwiki 10621
519	mywiki 92827
520	mywikibooks 21
521	mywiktionary 100692
522	mznwiki 23033
523	nahwiki 42179
524	nahwikibooks 27
525	nahwiktionary 11278
526	napwiki 33796
527	nawiki 5393
528	nawikibooks 27
529	nawikiquote 28
530	nawiktionary 2653
531	nds_nlwiki 25132
532	ndswiki 71127
533	ndswikibooks 26
534	ndswikiquote 17
535	ndswiktionary 50949
536	newiki 161356
537	newikibooks 5676
538	newiktionary 13061
539	newwiki 62296
540	ngwiki 74
541	nlwikibooks 34341
542	nlwikimedia 9350
543	nlwikinews 269
544	nlwikiquote 15590
545	nlwikisource 13591
546	nlwikivoyage 28810
547	nlwiktionary 799151
548	nnwiki 400868
549	nnwikiquote 2411
550	nnwiktionary 14888
551	noboard_chapterswikimedia 565
552	nostalgiawiki 3
553	novwiki 8841
554	nowikibooks 3576
555	nowikimedia 3119
556	nowikinews 1870
557	nowikiquote 2029
558	nowikisource 23145
559	nowiktionary 167300
560	nrmwiki 8419
561	nsowiki 12508
562	nvwiki 14048
563	nycwikimedia 314
564	nywiki 1771
565	nzwikimedia 23
566	ocwiki 244460
567	ocwikibooks 317
568	ocwiktionary 54881
569	officewiki 71743
570	olowiki 13436
571	ombudsmenwiki 1155
572	omwiki 4524
573	omwiktionary 1383
574	orwiki 89173
575	orwikisource 11738
576	orwiktionary 74596
577	oswiki 37073
578	otrs_wikiwiki 19635
579	outreachwiki 56635
580	pagwiki 8461
581	pamwiki 23125
582	papwiki 7590
583	pa_uswikimedia 82
584	pawiki 146687
585	pawikibooks 516
586	pawiktionary 4231
587	pcdwiki 10228
588	pdcwiki 7657
589	pflwiki 14486
590	pihwiki 3301
591	piwiki 5335
592	piwiktionary 17
593	plwikibooks 25451
594	plwikimedia 16232
595	plwikinews 11790
596	plwikiquote 62487
597	plwikisource 551412
598	plwikivoyage 35733
599	plwiktionary 1284430
600	pmswiki 110075
601	pnbwiki 105042
602	pnbwiktionary 4040
603	pntwiki 2356
604	projectcomwiki 10
605	pswiki 33247
606	pswikibooks 30
607	pswiktionary 28306
608	ptwikibooks 145708
609	ptwikinews 59188
610	ptwikiquote 15187
611	ptwikisource 54910
612	ptwikiversity 23482
613	ptwikivoyage 59284
614	ptwiktionary 399012
615	qualitywiki 46
616	quwiki 43130
617	quwikibooks 25
618	quwikiquote 7
619	quwiktionary 1106
620	rmwiki 13909
621	rmwikibooks 16
622	rmwiktionary 20
623	rmywiki 3334
624	rnwiki 1897
625	rnwiktionary 17
626	roa_rupwiki 16989
627	roa_rupwiktionary 2955
628	roa_tarawiki 10937
629	rowikibooks 7300
630	rowikinews 3899
631	rowikiquote 2107
632	rowikisource 13858
633	rowikivoyage 9629
634	rowiktionary 229980
635	rswikimedia 2846
636	ruewiki 16405
637	ruwikibooks 32489
638	ruwikimedia 23432
639	ruwikinews 164626
640	ruwikiquote 60689
641	ruwikisource 804450
642	ruwikiversity 26953
643	ruwikivoyage 85170
644	ruwiktionary 2197500
645	rwwiki 6757
646	rwwiktionary 1126
647	sahwiki 41462
648	sahwikisource 532
649	sawiki 110116
650	sawikibooks 649
651	sawikiquote 7919
652	sawikisource 37614
653	sawiktionary 235217
654	scnwiki 54000
655	scnwiktionary 14058
656	scowiki 156488
657	scwiki 18516
658	scwiktionary 35
659	sdwiki 31043
660	sdwikinews 95
661	sdwiktionary 4657
662	searchcomwiki NULL
663	sewiki 34375
664	sewikibooks 41
665	sewikimedia 27887
666	sgwiki 1403
667	sgwiktionary 865
668	shwiki 19722123
669	shwiktionary 728335
670	simplewiki 952441
671	simplewikibooks 582
672	simplewikiquote 206
673	simplewiktionary 76656
674	siwiki 85534
675	siwikibooks 1251
676	siwiktionary 2247
677	skwiki 854903
678	skwikibooks 3775
679	skwikiquote 5668
680	skwikisource 1024
681	skwiktionary 35115
682	slwiki 597674
683	slwikibooks 2480
684	slwikiquote 2944
685	slwikisource 31329
686	slwikiversity 9647
687	slwiktionary 14086
688	smwiki 3289
689	smwiktionary 21518
690	snwiki 11308
691	snwiktionary 42
692	sourceswiki 87137
693	sowiki 35097
694	sowiktionary 5064
695	spcomwiki NULL
696	specieswiki 870749
697	sqwiki 269309
698	sqwikibooks 2441
699	sqwikinews 927
700	sqwikiquote 694
701	sqwiktionary 41019
702	srnwiki 3574
703	srwiki 3275891
704	srwikibooks 2327
705	srwikinews 20260
706	srwikiquote 1920
707	srwikisource 11672
708	srwiktionary 169117
709	sswiki 2668
710	sswiktionary 956
711	stewardwiki 2080
712	stqwiki 10888
713	strategywiki 319
714	stwiki 2707
715	stwiktionary 1888
716	suwiki 55310
717	suwikibooks 51
718	suwikiquote 1272
719	suwiktionary 1124
720	svwikibooks 5801
721	svwikinews 1988
722	svwikiquote 2948
723	svwikisource 62758
724	svwikiversity 5414
725	svwikivoyage 14838
726	svwiktionary 548706
727	swwiki 93196
728	swwikibooks 29
729	swwiktionary 19374
730	szlwiki 22380
731	tawiki 553608
732	tawikibooks 1922
733	tawikinews 10032
734	tawikiquote 4022
735	tawikisource 237949
736	tawiktionary 194212
737	tcywiki 35302
738	tenwiki 1507
739	test2wiki 224134
740	testwiki 123678
741	testwikidatawiki 82093
742	tetwiki 6830
743	tewiki 672086
744	tewikibooks 452
745	tewikiquote 2942
746	tewikisource 76623
747	tewiktionary 295706
748	tgwiki 283822
749	tgwikibooks 390
750	tgwiktionary 40149
751	thwikibooks 11276
752	thwikinews 104
753	thwikiquote 5860
754	thwikisource 11656
755	thwiktionary 323587
756	tiwiki 1863
757	tiwiktionary 783
758	tkwiki 19206
759	tkwikibooks 37
760	tkwikiquote 19
761	tkwiktionary 3460
762	tlwiki 215546
763	tlwikibooks 3792
764	tlwiktionary 36356
765	tnwiki 3049
766	tnwiktionary 785
767	towiki 2186
768	towiktionary 24
769	tpiwiki 6174
770	tpiwiktionary 949
771	transitionteamwiki 292
772	trwikibooks 5405
773	trwikimedia 415
774	trwikinews 64744
775	trwikiquote 30906
776	trwikisource 8286
777	trwiktionary 433083
778	tswiki 2837
779	tswiktionary 743
780	ttwiki 777740
781	ttwikibooks 366
782	ttwikiquote 36
783	ttwiktionary 21975
784	tumwiki 2136
785	twwiki 2227
786	twwiktionary 19
787	tyvwiki 10334
788	tywiki 4548
789	uawikimedia 20129
790	udmwiki 12942
791	ugwiki 12039
792	ugwikibooks 49
793	ugwikiquote 21
794	ugwiktionary 6769
795	ukwikibooks 4686
796	ukwikimedia 8444
797	ukwikinews 12632
798	ukwikiquote 24095
799	ukwikisource 31565
800	ukwikivoyage 8362
801	ukwiktionary 35194
802	urwiki 991030
803	urwikibooks 1028
804	urwikiquote 1790
805	urwiktionary 2351
806	usabilitywiki 449
807	uzwiki 625221
808	uzwikibooks 117
809	uzwikiquote 269
810	uzwiktionary 228037
811	vecwiki 47794
812	vecwikisource 5534
813	vecwiktionary 10092
814	vepwiki 29031
815	vewiki 1599
816	viwikibooks 35743
817	viwikiquote 10319
818	viwikisource 9809
819	viwikivoyage 17013
820	viwiktionary 164294
821	vlswiki 23486
822	votewiki 399
823	vowiki 182916
824	vowikibooks 36
825	vowikiquote 45
826	vowiktionary 13867
827	warwiki 2366587
828	wawiki 21739
829	wawikibooks 14
830	wawiktionary 39109
831	wg_enwiki NULL
832	wikimania2005wiki 34
833	wikimania2006wiki 543
834	wikimania2007wiki 2242
835	wikimania2008wiki 1611
836	wikimania2009wiki 460
837	wikimania2010wiki 188
838	wikimania2011wiki 260
839	wikimania2012wiki 364
840	wikimania2013wiki 11113
841	wikimania2014wiki 25825
842	wikimania2015wiki 26322
843	wikimania2016wiki 20531
844	wikimania2017wiki 12721
845	wikimaniateamwiki 390
846	wowiki 5467
847	wowikiquote 212
848	wowiktionary 2420
849	wuuwiki 31111
850	xalwiki 7047
851	xhwiki 4121
852	xhwikibooks 20
853	xhwiktionary 37
854	xmfwiki 32852
855	yiwiki 43185
856	yiwikisource 457
857	yiwiktionary 986
858	yowiki 60603
859	yowikibooks 29
860	yowiktionary 35
861	zawiki 3033
862	zawikibooks 12
863	zawikiquote 17
864	zawiktionary 64
865	zeawiki 16026
866	zerowiki 5050
867	zh_classicalwiki 32882
868	zh_min_nanwiki 629749
869	zh_min_nanwikibooks 60
870	zh_min_nanwikiquote 68
871	zh_min_nanwikisource 2184
872	zh_min_nanwiktionary 28121
873	zhwikibooks 17037
874	zhwikinews 19776
875	zhwikiquote 13621
876	zhwikisource 249295
877	zhwikivoyage 40085
878	zhwiktionary 514763
879	zh_yuewiki 200281
880	zuwiki 4969
881	zuwikibooks 20
882	zuwiktionary 1812
883	commonswiki 82350084
884	dewiki 28595018
885	wikidatawiki 241176029
886	frwiki 27512786
887	jawiki 10291153
888	ruwiki 19072238
889	arwiki 7231773
890	cawiki 4196777
891	eswiki 21588048
892	fawiki 6468735
893	frwiktionary 4974604
894	hewiki 3930946
895	huwiki 3214305
896	kowiki 4813984
897	metawiki 6343097
898	rowiki 2122294
899	ukwiki 4967529
900	viwiki 9187800

I vote for repooling this host again to see if it fails again when it gets load

Marostegui added a project: DBA.Apr 24 2017, 8:19 AM

Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2017-04-26T15:24:31Z] <marostegui> Shutdown es2019 for maintenance with papaul and Dell - T149526

Hi Papaul,

Thank you for contacting Dell EMC Basic Server Support.

This mail is with reference to the (Memory and CPU Issue) you had reported on your PowerEdge(R730XD).

Please find the service request number for your reference in below. Please reply back with the address of the location where you want the part to be delivered

Service Tag : 8PVW382

Service request number : 947570621

Please feel free to reply to the email if you have any other concerns since I would be able to reply back and help you further.

Regards

Suraj Kumar

Enterprise Tech Support Analyst

Dell EMC | NA Basic Server Support

Enterprise Remote Services and Solutions

Hi Papaul,

I will get the motherboard and the memory module replaced at the same time but at the same time would like to request you to help me with the address of the location where you want the parts and the engineer to be sent .

Thanks and regards

Suraj Kumar

Thanks Papaul! As per our chat, I have brought MySQL, ping me when you need it down again,

Mentioned in SAL (#wikimedia-operations) [2017-04-27T14:29:54Z] <marostegui> Stop MySQL and shutdown es2019 for HW replacement - T149526

Main board replacement
DIMM B4 Replaced
DIMM A1 Replaced
BIOS update from 2.2.5 to 2.4.3

Papaul reassigned this task from Papaul to Marostegui.Apr 27 2017, 4:54 PM

Thanks @Papaul!
Let's see how it goes.

Marostegui moved this task from In progress to Blocked external/Not db team on the DBA board.Apr 27 2017, 6:49 PM

Let's close this for now.

Pending to run compare.py around the ids obtained before.

jcrespo claimed this task.May 5 2017, 10:44 AM

jcrespo moved this task from Blocked external/Not db team to Pending comment on the DBA board.

jcrespo removed a project: ops-codfw.

Left running on neodymium (I did some optimizations to ignore values below and beyond max id respectively):

while read db id; do echo "./compare.py es1019.eqiad.wmnet es2019.codfw.wmnet $db blobs_cluster25 blob_id --step=100 --from-value $(($id - 300000)) --to-value $(($id + 300000))"; ./compare.py es1019.eqiad.wmnet es2019.codfw.wmnet $db blobs_cluster25 blob_id --step=100 --from-value $(($id - 300000)) --to-value $(($id + 300000)) ; done < P5309

Change 352752 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool es2019

https://gerrit.wikimedia.org/r/352752

gerritbot added a project: Patch-For-Review.May 9 2017, 7:08 AM

Change 352752 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool es2019

https://gerrit.wikimedia.org/r/352752

Mentioned in SAL (#wikimedia-operations) [2017-05-09T07:12:23Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool es2019 - T149526 (duration: 00m 39s)

As there were no differences found by compare.py I have repooled this host. Finger crossed so it doesn't crash anymore!

Marostegui added a parent task: T130702: Several es20XX servers keep crashing (es2017, es2019, es2015, es2014) since 23 March.Jan 4 2019, 3:24 PM

es2019 crashed again
Closed, ResolvedPublic
Actions

Description

Details

Related Objects
Search...

Event Timeline

es2019 crashed againClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

es2019 crashed again
Closed, ResolvedPublic
Actions

Related Objects
Search...