Page MenuHomePhabricator

es2019 crashed again
Closed, ResolvedPublic

Description

Despite T130702 and T147769, es2 systems keep going down, at least one per month.

Details

Related Gerrit Patches:
operations/mediawiki-config : masterdb-codfw.php: Repool es2019

Event Timeline

jcrespo created this task.Oct 30 2016, 4:32 PM
Restricted Application added a project: Operations. · View Herald TranscriptOct 30 2016, 4:32 PM
Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-operations) [2016-10-30T16:35:25Z] <jynus> powercycle es2019 after crash T149526

No trace of HW logs I assume? :(

Fri May 27 2016 17:31:34	Correctable memory error rate exceeded for DIMM_A1.
Fri May 27 2016 18:42:52	Correctable memory error rate exceeded for DIMM_A1.
Tue Jun 07 2016 17:21:45	Correctable memory error rate exceeded for DIMM_A1.	
Tue Jun 07 2016 17:27:14	Correctable memory error rate exceeded for DIMM_A1.	
Tue Jun 07 2016 17:27:31	Correctable memory error rate exceeded for DIMM_A1.	
Sun Oct 30 2016 15:14:10	Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
Sun Oct 30 2016 15:14:13	CPU 2 has an internal error (IERR).
MEM0001: Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
 2016-10-30T15:14:10-0500
Log Sequence Number: 362
Detailed Description:
The memory has encountered a uncorrectable error. System performance may be degraded. The operating system and/or applications may fail as a result.
Recommended Action:
Re-install the memory component. If the problem persists, contact technical support. Refer to the product documentation to choose a convenient contact method.
Comment: root
CPU0000: CPU 2 has an internal error (IERR).
 2016-10-30T15:14:14-0500
Log Sequence Number: 366
Detailed Description:
System event log and OS logs may indicate that the exception is external to the processor.
Recommended Action:
Review System Event Log and Operating System Logs. If the issue persists, contact technical support. Refer to the product documentation to choose a convenient contact method.
Comment: root

This is from the hardware logs

The last time we replaced the memory on DIMM A1 on this system. What i will do is to swap DIMM B2 with DIMM B1 and clean the logs. If the message show on DIMM B2 then I will request new memory.

jcrespo added a comment.EditedNov 2 2016, 6:39 PM

Open to schedule it when you have the time. Should we update the BIOS (again?). CC'ing @RobH here.

I would upgrade the BIOS if we need to reboot the server anyways.

Papaul triaged this task as Medium priority.Nov 16 2016, 3:46 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-21T15:58:12Z] <marostegui> Shutting down MySQL es2019 for HW maintenance - T149526

Mentioned in SAL (#wikimedia-operations) [2016-11-21T15:59:31Z] <marostegui> Powering off es2019 for HW maintenance - T149526

1- Swapped CPU 2 to CPU1
2 - Update BIOS from 2.1.6 to 2.2.5
3- Clear syslog

Leaving this task open for now .

Thanks. I have started MySQL and replication again - we will see how it behaves!

Papaul closed this task as Resolved.Jan 13 2017, 12:59 AM

This ask has been open for almost 2 months. closing it. it can be reopen anytime we have the issue again.

jcrespo reopened this task as Open.Apr 22 2017, 7:38 AM

Server crashed again as per: T130702#3203266

Normal","Mon Nov 21 2016 16:22:07","Log cleared.
Critical","Sat Apr 22 2017 07:52:10","CPU 2 has an internal error (IERR).
Normal","Sat Apr 22 2017 06:56:24","A problem was detected related to the previous server boot.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_B4.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
Critical","Sat Apr 22 2017 06:56:24","Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
elukey added a subscriber: elukey.Apr 22 2017, 7:39 AM
This comment was removed by elukey.
Severity	Date and Time	Message ID	Summary	Comment
 	  	2017-04-22T07:37:48-0500	USR0032	
The session for root from 10.64.32.20 using SSH is logged off.
	
 
 	  	2017-04-22T07:37:41-0500	USR0030	
Successfully logged in using root, from 10.192.16.172 and GUI.
	
 
 	  	2017-04-22T07:36:16-0500	USR0030	
Successfully logged in using root, from 10.64.32.20 and SSH.
	
 
 	  	2017-04-22T07:36:08-0500	USR0032	
The session for root from 208.80.154.149 using SSH is logged off.
	
 
 	  	2017-04-22T07:30:25-0500	USR0032	
The session for root from 10.64.32.20 using SSH is logged off.
	
 
 	  	2017-04-22T07:27:55-0500	USR0030	
Successfully logged in using root, from 208.80.154.149 and SSH.
	
 
 	  	2017-04-22T07:22:50-0500	USR0030	
Successfully logged in using root, from 10.64.32.20 and SSH.
	
 
 	  	2017-04-22T06:56:31-0500	MEM0001	
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
	
 
 	  	2017-04-22T06:56:30-0500	MEM0001	
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
	
 
 	  	2017-04-22T06:56:28-0500	MEM0001	
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
	
 
 	  	2017-04-22T06:56:27-0500	MEM0001	
Multi-bit memory errors detected on a memory device at location(s) DIMM_A1.
	
 
 	  	2017-04-22T06:56:26-0500	MEM0001	
Multi-bit memory errors detected on a memory device at location(s) DIMM_B4.
	
 
 	  	2017-04-22T06:56:24-0500	UEFI0079	
One or more Uncorrectable Memory errors occurred in the previous boot.
	
 
 	  	2017-04-22T06:56:24-0500	PST0090	
A problem was detected related to the previous server boot.
	
 
 	  	2017-04-22T07:52:10-0500	CPU0000	
CPU 2 has an internal error (IERR).
	
 
 	  	2017-04-22T07:52:10-0500	SYS1003	
System CPU Resetting.
	
 
 	  	2017-04-22T07:52:09-0500	RAC0703	
Requested system hardreset.
	
 
 	  	2017-04-22T07:52:09-0500	SYS1003	
System CPU Resetting.
	
 
 	  	2017-04-22T03:58:35-0500	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-04-15T08:15:00-0500	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-04-15T03:58:37-0500	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-04-08T08:10:41-0500	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-04-08T03:58:44-0500	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-04-01T08:10:57-0500	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-04-01T03:58:49-0500	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-03-25T08:12:35-0500	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-03-25T03:58:52-0500	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-03-18T08:10:54-0500	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-03-18T03:58:55-0500	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-03-11T07:10:17-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-03-11T02:59:02-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-03-04T07:20:34-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-03-04T02:59:05-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-02-25T07:14:41-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-02-25T02:59:06-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-02-18T07:18:58-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-02-18T02:59:12-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-02-11T07:18:01-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-02-11T02:59:14-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-02-04T07:11:28-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-02-04T02:59:20-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-01-28T07:19:12-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-01-28T02:59:22-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-01-21T07:11:32-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-01-21T02:59:31-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-01-14T07:15:38-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-01-14T02:59:34-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2017-01-07T07:14:37-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2017-01-07T02:59:35-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2016-12-31T07:10:41-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2016-12-31T02:59:43-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2016-12-24T07:14:53-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2016-12-24T02:59:47-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2016-12-17T07:14:41-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2016-12-17T02:59:50-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2016-12-10T07:13:03-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2016-12-10T02:59:51-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2016-12-03T07:09:55-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2016-12-03T02:59:55-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2016-11-26T07:14:52-0600	CTL38	
The Patrol Read operation completed for Integrated RAID Controller 1.
	
 
 	  	2016-11-26T03:00:00-0600	CTL37	
A Patrol Read operation started for Integrated RAID Controller 1.
	
 
 	  	2016-11-22T19:30:50-0600	USR0032	
The session for root from 208.80.154.149 using SSH is logged off.
	
 
 	  	2016-11-22T19:30:45-0600	JCP036	
The (installation or configuration) job JID_798645851255 is successfully deleted.
	
 
 	  	2016-11-22T19:29:45-0600	JCP027	
Job created successfully.
	
 
 	  	2016-11-22T19:29:37-0600	USR0030	
Successfully logged in using root, from 208.80.154.149 and SSH.
	
 
 	  	2016-11-22T19:29:37-0600	LOG007	
The previous log entry was repeated 1 times.
	
 
 	  	2016-11-22T19:28:31-0600	SYS1003	
System CPU Resetting.
	
 
 	  	2016-11-22T19:24:13-0600	JCP036	
The (installation or configuration) job JID_798641868867 is successfully deleted.
	
 
 	  	2016-11-22T19:23:51-0600	USR0032	
The session for root from 208.80.154.149 using SSH is logged off.
	
 
 	  	2016-11-22T19:23:06-0600	JCP027	
Job created successfully.
	
 
 	  	2016-11-22T19:23:02-0600	USR0030	
Successfully logged in using root, from 208.80.154.149 and SSH.
	
 
 	  	2016-11-22T19:23:02-0600	LOG007	
The previous log entry was repeated 1 times.
	
 
 	  	2016-11-22T19:21:52-0600	SYS1003	
System CPU Resetting.
	
 
 	  	2016-11-21T17:11:23-0600	USR0032	
The session for root from 208.80.153.5 using SSH is logged off.
	
 
 	  	2016-11-21T17:11:03-0600	USR0032	
The session for root from 10.193.0.7 using GUI is logged off.
	
 
 	  	2016-11-21T17:09:41-0600	USR0174	
The Front Panel USB device is removed from the operating system.
	
 
 	  	2016-11-21T17:09:07-0600	NIC101	
The NIC Integrated 1 Port 1 network link is started.
	
 
 	  	2016-11-21T17:08:26-0600	USR0030	
Successfully logged in using root, from 208.80.153.5 and SSH.
	
 
 	  	2016-11-21T17:08:15-0600	USR0032	
The session for root from 91.198.174.112 using SSH is logged off.
	
 
 	  	2016-11-21T17:08:13-0600	USR0032	
The session for root from 208.80.153.5 using SSH is logged off.
	
 
 	  	2016-11-21T17:08:13-0600	LOG007	
The previous log entry was repeated 1 times.
	
 
 	  	2016-11-21T17:02:30-0600	USR0030	
Successfully logged in using root, from 208.80.153.5 and SSH.
	
 
 	  	2016-11-21T17:00:04-0600	USR0030	
Successfully logged in using root, from 10.193.0.7 and GUI.
	
 
 	  	2016-11-21T17:00:04-0600	LOG007	
The previous log entry was repeated 1 times.
	
 
 	  	2016-11-21T16:59:16-0600	USR0030	
Successfully logged in using root, from 208.80.153.5 and SSH.
	
 
 	  	2016-11-21T16:58:51-0600	USR0032	
The session for root from 10.193.0.7 using GUI is logged off.
	
 
 	  	2016-11-21T16:58:48-0600	JCP036	
The (installation or configuration) job JID_797690674507 is successfully deleted.
	
 
 	  	2016-11-21T16:57:47-0600	JCP027	
Job created successfully.
	
 
 	  	2016-11-21T16:57:28-0600	USR0173	
The Front Panel USB port switched automatically from iDRAC to operating system.
	
 
 	  	2016-11-21T16:57:28-0600	USR0171	
The Front Panel USB port is detached from the iDRAC Disk.USBFront.1. Device Details: Device Class 3, Vendor ID 0B38, Product ID 0010.
	
 
 	  	2016-11-21T16:57:23-0600	USR0170	
The Front Panel USB port is attached to iDRAC Disk.USBFront.1. Device details: Device class 3, Vendor ID 0B38, Manufacturer Name Not Available, Product ID 0010, Product Name Not Available, Serial Number Not Available.
	
 
 	  	2016-11-21T16:57:15-0600	PR36	
Version change detected for BIOS firmware. Previous version:2.1.6, Current version:2.2.5
	
 
 	  	2016-11-21T16:55:55-0600	USR0030	
Successfully logged in using root, from 208.80.153.5 and SSH.
	
 
 	  	2016-11-21T16:55:52-0600	USR0031	
Unable to log in for root from 208.80.153.5 using SSH.
	
 
 	  	2016-11-21T16:55:41-0600	USR0030	
Successfully logged in using root, from 208.80.153.5 and SSH.
	
 
 	  	2016-11-21T16:55:35-0600	USR0031	
Unable to log in for root from 208.80.153.5 using SSH.
	
 
 	  	2016-11-21T16:55:35-0600	LOG007	
The previous log entry was repeated 1 times.
	
 
 	  	2016-11-21T16:55:26-0600	SYS1003	
System CPU Resetting.
	
 
 	  	2016-11-21T16:55:25-0600	SYS1000	
System is turning on.
	
 
 	  	2016-11-21T16:55:16-0600	SYS1003	
System CPU Resetting.

The memory module on this host apparently was replaced: T149526#2756898 maybe memory slot is what is broken?

For checking later:

1$ cat *.dblist | while read db; do mysql -BN -A -h es2019.codfw.wmnet $db -e "SELECT '$db', max(blob_id) FROM blobs_cluster25"; done
2enwiki 137353196
3bgwiki 1424466
4bgwiktionary 33930
5cswiki 3085923
6enwikiquote 386310
7enwiktionary 12334923
8eowiki 789524
9fiwiki 2238381
10idwiki 3505748
11itwiki 17523821
12nlwiki 8367741
13nowiki 3092125
14plwiki 8287923
15ptwiki 8959556
16svwiki 11156711
17thwiki 1401691
18trwiki 3784403
19zhwiki 11290429
20aawiki 175
21aawikibooks 25
22aawiktionary 44
23abwiki 6357
24abwiktionary 29
25acewiki 17137
26advisorywiki 17
27adywiki 3315
28afwiki 263642
29afwikibooks 310
30afwikiquote 633
31afwiktionary 12385
32akwiki 2539
33akwikibooks 34
34akwiktionary 22
35alswiki 214056
36alswikibooks 36
37alswikiquote 23
38alswiktionary 34
39amwiki 33514
40amwikiquote 116
41amwiktionary 2009
42angwiki 17329
43angwikibooks 318
44angwikiquote 22
45angwikisource 20
46angwiktionary 5799
47anwiki 115477
48anwiktionary 6094
49arbcom_dewiki 12362
50arbcom_enwiki 7400
51arbcom_fiwiki 282
52arbcom_nlwiki 11117
53arcwiki 5976
54arwikibooks 17326
55arwikimedia 117
56arwikinews 17536
57arwikiquote 13891
58arwikisource 49101
59arwikiversity 15499
60arwiktionary 74220
61arzwiki 143132
62astwiki 188653
63astwikibooks 32
64astwikiquote 28
65astwiktionary 37490
66aswiki 36709
67aswikibooks 36
68aswikisource 5055
69aswiktionary 25
70auditcomwiki NULL
71avwiki 10878
72avwiktionary 20
73aywiki 8257
74aywikibooks 20
75aywiktionary 2860
76azbwiki 70378
77azwiki 815826
78azwikibooks 7558
79azwikiquote 16708
80azwikisource 18677
81azwiktionary 116848
82barwiki 176315
83bat_smgwiki 28523
84bawiki 262373
85bawikibooks 29
86bclwiki 19305
87bdwikimedia 1373
88betawikiversity 46541
89bewiki 661474
90bewikibooks 413
91bewikimedia 5002
92bewikiquote 1194
93bewikisource 8525
94bewiktionary 9131
95be_x_oldwiki 247101
96bgwikibooks 950
97bgwikinews 1443
98bgwikiquote 8891
99bgwikisource 3433
100bhwiki 174313
101bhwiktionary 28
102biwiki 4172
103biwikibooks 16
104biwiktionary 25
105bjnwiki 9662
106bmwiki 2438
107bmwikibooks 14
108bmwikiquote 16
109bmwiktionary 46
110bnwiki 699259
111bnwikibooks 4794
112bnwikisource 368017
113bnwiktionary 6760
114boardgovcomwiki NULL
115boardwiki 3612
116bowiki 12480
117bowikibooks 49
118bowiktionary 28
119bpywiki 46508
120brwiki 210090
121brwikimedia 5407
122brwikiquote 373
123brwikisource 29032
124brwiktionary 58315
125bswiki 468078
126bswikibooks 530
127bswikinews 4527
128bswikiquote 5504
129bswikisource 759
130bswiktionary 15772
131bugwiki 28599
132bxrwiki 14927
133cawikibooks 14227
134cawikimedia 5216
135cawikinews 47542
136cawikiquote 32226
137cawikisource 23349
138cawiktionary 343856
139cbk_zamwiki 16210
140cdowiki 9830
141cebwiki 6354983
142cewiki 869126
143chairwiki NULL
144chapcomwiki 3969
145checkuserwiki 7267
146chowiki 98
147chrwiki 3780
148chrwiktionary 289854
149chwiki 2195
150chwikibooks 37
151chwiktionary 21
152chywiki 2631
153ckbwiki 188087
154cnwikimedia 94
155collabwiki 18221
156cowiki 26179
157cowikibooks 53
158cowikimedia 599
159cowikiquote 28
160cowiktionary 15302
161crhwiki 15247
162crwiki 4581
163crwikiquote 29
164crwiktionary 34
165csbwiki 13947
166csbwiktionary 4245
167cswikibooks 5172
168cswikinews 15795
169cswikiquote 18037
170cswikisource 41783
171cswikiversity 29015
172cswiktionary 247435
173cuwiki 5736
174cvwiki 81445
175cvwikibooks 298
176cywiki 592309
177cywikibooks 312
178cywikiquote 755
179cywikisource 661
180cywiktionary 37725
181dawiki 1349235
182dawikibooks 5047
183dawikiquote 2552
184dawikisource 7415
185dawiktionary 81201
186dewikibooks 88669
187dewikinews 69460
188dewikiquote 16715
189dewikisource 561946
190dewikiversity 84366
191dewikivoyage 277541
192dewiktionary 1402174
193diqwiki 72718
194dkwikimedia 1383
195donatewiki 2215
196dsbwiki 14589
197dvwiki 7369
198dvwiktionary 1385
199dzwiki 1725
200dzwiktionary 38
201ecwikimedia 6
202eewiki 3147
203elwiki 1442861
204elwikibooks 14517
205elwikinews 8493
206elwikiquote 4010
207elwikisource 20226
208elwikiversity 2148
209elwikivoyage 10632
210elwiktionary 487387
211emlwiki 25590
212enwikibooks 425730
213enwikinews 1342139
214enwikisource 1375680
215enwikiversity 373179
216enwikivoyage 612000
217eowikibooks 4221
218eowikinews 56082
219eowikiquote 2973
220eowikisource 6086
221eowiktionary 144140
222eswikibooks 70592
223eswikinews 132928
224eswikiquote 48085
225eswikisource 146939
226eswikiversity 31304
227eswikivoyage 69212
228eswiktionary 1708537
229etwiki 808631
230etwikibooks 1203
231etwikimedia 486
232etwikiquote 1446
233etwikisource 15072
234etwiktionary 171069
235euwiki 1338300
236euwikibooks 2014
237euwikiquote 882
238euwiktionary 107021
239execwiki NULL
240extwiki 9676
241fawikibooks 27219
242fawikinews 47312
243fawikiquote 34912
244fawikisource 8443
245fawikivoyage 46335
246fawiktionary 109214
247fdcwiki 2788
248ffwiki 1954
249fiu_vrowiki 9759
250fiwikibooks 13103
251fiwikimedia 1156
252fiwikinews 2792
253fiwikiquote 8921
254fiwikisource 14124
255fiwikiversity 11067
256fiwikivoyage 26409
257fiwiktionary 703411
258fjwiki 2183
259fjwiktionary 57484
260foundationwiki 13492
261fowiki 51520
262fowikisource 254
263fowiktionary 2926
264frpwiki 9664
265frrwiki 35286
266frwikibooks 94786
267frwikinews 152761
268frwikiquote 45319
269frwikisource 1535610
270frwikiversity 164844
271frwikivoyage 123641
272furwiki 9067
273fywiki 124547
274fywikibooks 265
275fywiktionary 14150
276gagwiki 9099
277ganwiki 28455
278gawiki 113844
279gawikibooks 42
280gawikiquote 45
281gawiktionary 7730
282gdwiki 67178
283gdwiktionary 9167
284glkwiki 6747
285glwiki 855346
286glwikibooks 2717
287glwikiquote 3964
288glwikisource 3899
289glwiktionary 79721
290gnwiki 10396
291gnwikibooks 15
292gnwiktionary 861
293gomwiki 85035
294gotwiki 5616
295gotwikibooks 45
296grantswiki NULL
297guwiki 136188
298guwikibooks 51
299guwikiquote 5609
300guwikisource 29361
301guwiktionary 2732
302gvwiki 23876
303gvwiktionary 2174
304hakwiki 17926
305hawiki 4740
306hawiktionary 761
307hawwiki 7227
308hewikibooks 24164
309hewikinews 5290
310hewikiquote 15273
311hewikisource 155373
312hewikivoyage 84413
313hewiktionary 55909
314hifwiki 41629
315hiwiki 787713
316hiwikibooks 2017
317hiwikiquote 5787
318hiwiktionary 134217
319howiki 85
320hrwiki 607483
321hrwikibooks 1367
322hrwikiquote 4667
323hrwikisource 4002
324hrwiktionary 49510
325hsbwiki 38785
326hsbwiktionary 1981
327htwiki 63721
328htwikisource 27
329huwikibooks 63263
330huwikinews 121
331huwikiquote 4607
332huwikisource 5124
333huwiktionary 439224
334hywiki 2119469
335hywikibooks 1312
336hywikiquote 25734
337hywikisource 44438
338hywiktionary 506550
339hzwiki 88
340iawiki 70723
341iawikibooks 340
342iawiktionary 4121
343idwikibooks 14332
344idwikiquote 4273
345idwikisource 5172
346idwiktionary 100390
347iegcomwiki 452
348iewiki 11181
349iewikibooks 219
350iewiktionary 2461
351igwiki 5856
352iiwiki 68
353ikwiki 2861
354ikwiktionary 91
355ilowiki 78388
356ilwikimedia NULL
357incubatorwiki 1557201
358internalwiki 91
359iowiki 78551
360iowiktionary 324191
361iswiki 133026
362iswikibooks 2020
363iswikiquote 597
364iswikisource 3443
365iswiktionary 21305
366itwikibooks 47458
367itwikinews 20983
368itwikiquote 164536
369itwikisource 358710
370itwikiversity 37826
371itwikivoyage 181264
372itwiktionary 729483
373iuwiki 2345
374iuwiktionary 1114
375jamwiki 9094
376jawikibooks 19686
377jawikinews 21255
378jawikiquote 3656
379jawikisource 20230
380jawikiversity 1724
381jawiktionary 219500
382jbowiki 5571
383jbowiktionary 2258
384jvwiki 235927
385jvwiktionary 50360
386kaawiki 7160
387kabwiki 12592
388kawiki 573117
389kawikibooks 1550
390kawikiquote 6290
391kawiktionary 16945
392kbdwiki 8840
393kgwiki 3989
394kiwiki 3037
395kjwiki 55
396kkwiki 738234
397kkwikibooks 686
398kkwikiquote 14
399kkwiktionary 13573
400klwiki 5463
401klwiktionary 2812
402kmwiki 38946
403kmwikibooks 1411
404kmwiktionary 25675
405knwiki 246607
406knwikibooks 43
407knwikiquote 1209
408knwikisource 34891
409knwiktionary 101903
410koiwiki 4914
411kowikibooks 5349
412kowikinews 7398
413kowikiquote 4289
414kowikisource 52761
415kowikiversity 7509
416kowiktionary 1041061
417krcwiki 11031
418krwiki 114
419krwikiquote 13
420kshwiki 9956
421kswiki 2000
422kswikibooks 32
423kswikiquote 15
424kswiktionary 470
425kuwiki 92048
426kuwikibooks 479
427kuwikiquote 722
428kuwiktionary 707518
429kvwiki 14475
430kwwiki 10487
431kwwikiquote 32
432kwwiktionary 1919
433kywiki 77547
434kywikibooks 301
435kywikiquote 276
436kywiktionary 51983
437ladwiki 27329
438lawiki 549203
439lawikibooks 370
440lawikiquote 1574
441lawikisource 12923
442lawiktionary 30102
443lbewiki 5178
444lbwiki 309650
445lbwikibooks 20
446lbwikiquote 45
447lbwiktionary 18129
448legalteamwiki 3894
449lezwiki 20547
450lgwiki 4123
451lijwiki 12745
452liwiki 47015
453liwikibooks 219
454liwikiquote 794
455liwikisource 262
456liwiktionary 67440
457lmowiki 98848
458lnwiki 8482
459lnwikibooks 50
460lnwiktionary 1607
461loginwiki 10
462lowiki 9568
463lowiktionary 60478
464lrcwiki 35810
465ltgwiki 3503
466ltwiki 566559
467ltwikibooks 1367
468ltwikiquote 8300
469ltwikisource 1462
470ltwiktionary 149600
471lvwiki 453022
472lvwikibooks 44
473lvwiktionary 21228
474maiwiki 69622
475map_bmswiki 18307
476mdfwiki 4654
477mediawikiwiki 1065341
478mgwiki 149533
479mgwikibooks 180
480mgwiktionary 7051401
481mhrwiki 32061
482mhwiki 82
483mhwiktionary 17
484minwiki 771996
485miwiki 8718
486miwikibooks 48
487miwiktionary 3215
488mkwiki 436330
489mkwikibooks 961
490mkwikimedia 288
491mkwikisource 1883
492mkwiktionary 13204
493mlwiki 556570
494mlwikibooks 3992
495mlwikiquote 2335
496mlwikisource 46082
497mlwiktionary 79097
498mnwiki 109247
499mnwikibooks 64
500mnwiktionary 111381
501movementroleswiki NULL
502mowiki 226
503mowiktionary 21
504mrjwiki 15607
505mrwiki 238074
506mrwikibooks 774
507mrwikiquote 575
508mrwikisource 5885
509mrwiktionary 1788
510mswiki 805382
511mswikibooks 1625
512mswiktionary 8245
513mtwiki 30061
514mtwiktionary 4476
515muswiki 59
516mwlwiki 9962
517mxwikimedia 1795
518myvwiki 10621
519mywiki 92827
520mywikibooks 21
521mywiktionary 100692
522mznwiki 23033
523nahwiki 42179
524nahwikibooks 27
525nahwiktionary 11278
526napwiki 33796
527nawiki 5393
528nawikibooks 27
529nawikiquote 28
530nawiktionary 2653
531nds_nlwiki 25132
532ndswiki 71127
533ndswikibooks 26
534ndswikiquote 17
535ndswiktionary 50949
536newiki 161356
537newikibooks 5676
538newiktionary 13061
539newwiki 62296
540ngwiki 74
541nlwikibooks 34341
542nlwikimedia 9350
543nlwikinews 269
544nlwikiquote 15590
545nlwikisource 13591
546nlwikivoyage 28810
547nlwiktionary 799151
548nnwiki 400868
549nnwikiquote 2411
550nnwiktionary 14888
551noboard_chapterswikimedia 565
552nostalgiawiki 3
553novwiki 8841
554nowikibooks 3576
555nowikimedia 3119
556nowikinews 1870
557nowikiquote 2029
558nowikisource 23145
559nowiktionary 167300
560nrmwiki 8419
561nsowiki 12508
562nvwiki 14048
563nycwikimedia 314
564nywiki 1771
565nzwikimedia 23
566ocwiki 244460
567ocwikibooks 317
568ocwiktionary 54881
569officewiki 71743
570olowiki 13436
571ombudsmenwiki 1155
572omwiki 4524
573omwiktionary 1383
574orwiki 89173
575orwikisource 11738
576orwiktionary 74596
577oswiki 37073
578otrs_wikiwiki 19635
579outreachwiki 56635
580pagwiki 8461
581pamwiki 23125
582papwiki 7590
583pa_uswikimedia 82
584pawiki 146687
585pawikibooks 516
586pawiktionary 4231
587pcdwiki 10228
588pdcwiki 7657
589pflwiki 14486
590pihwiki 3301
591piwiki 5335
592piwiktionary 17
593plwikibooks 25451
594plwikimedia 16232
595plwikinews 11790
596plwikiquote 62487
597plwikisource 551412
598plwikivoyage 35733
599plwiktionary 1284430
600pmswiki 110075
601pnbwiki 105042
602pnbwiktionary 4040
603pntwiki 2356
604projectcomwiki 10
605pswiki 33247
606pswikibooks 30
607pswiktionary 28306
608ptwikibooks 145708
609ptwikinews 59188
610ptwikiquote 15187
611ptwikisource 54910
612ptwikiversity 23482
613ptwikivoyage 59284
614ptwiktionary 399012
615qualitywiki 46
616quwiki 43130
617quwikibooks 25
618quwikiquote 7
619quwiktionary 1106
620rmwiki 13909
621rmwikibooks 16
622rmwiktionary 20
623rmywiki 3334
624rnwiki 1897
625rnwiktionary 17
626roa_rupwiki 16989
627roa_rupwiktionary 2955
628roa_tarawiki 10937
629rowikibooks 7300
630rowikinews 3899
631rowikiquote 2107
632rowikisource 13858
633rowikivoyage 9629
634rowiktionary 229980
635rswikimedia 2846
636ruewiki 16405
637ruwikibooks 32489
638ruwikimedia 23432
639ruwikinews 164626
640ruwikiquote 60689
641ruwikisource 804450
642ruwikiversity 26953
643ruwikivoyage 85170
644ruwiktionary 2197500
645rwwiki 6757
646rwwiktionary 1126
647sahwiki 41462
648sahwikisource 532
649sawiki 110116
650sawikibooks 649
651sawikiquote 7919
652sawikisource 37614
653sawiktionary 235217
654scnwiki 54000
655scnwiktionary 14058
656scowiki 156488
657scwiki 18516
658scwiktionary 35
659sdwiki 31043
660sdwikinews 95
661sdwiktionary 4657
662searchcomwiki NULL
663sewiki 34375
664sewikibooks 41
665sewikimedia 27887
666sgwiki 1403
667sgwiktionary 865
668shwiki 19722123
669shwiktionary 728335
670simplewiki 952441
671simplewikibooks 582
672simplewikiquote 206
673simplewiktionary 76656
674siwiki 85534
675siwikibooks 1251
676siwiktionary 2247
677skwiki 854903
678skwikibooks 3775
679skwikiquote 5668
680skwikisource 1024
681skwiktionary 35115
682slwiki 597674
683slwikibooks 2480
684slwikiquote 2944
685slwikisource 31329
686slwikiversity 9647
687slwiktionary 14086
688smwiki 3289
689smwiktionary 21518
690snwiki 11308
691snwiktionary 42
692sourceswiki 87137
693sowiki 35097
694sowiktionary 5064
695spcomwiki NULL
696specieswiki 870749
697sqwiki 269309
698sqwikibooks 2441
699sqwikinews 927
700sqwikiquote 694
701sqwiktionary 41019
702srnwiki 3574
703srwiki 3275891
704srwikibooks 2327
705srwikinews 20260
706srwikiquote 1920
707srwikisource 11672
708srwiktionary 169117
709sswiki 2668
710sswiktionary 956
711stewardwiki 2080
712stqwiki 10888
713strategywiki 319
714stwiki 2707
715stwiktionary 1888
716suwiki 55310
717suwikibooks 51
718suwikiquote 1272
719suwiktionary 1124
720svwikibooks 5801
721svwikinews 1988
722svwikiquote 2948
723svwikisource 62758
724svwikiversity 5414
725svwikivoyage 14838
726svwiktionary 548706
727swwiki 93196
728swwikibooks 29
729swwiktionary 19374
730szlwiki 22380
731tawiki 553608
732tawikibooks 1922
733tawikinews 10032
734tawikiquote 4022
735tawikisource 237949
736tawiktionary 194212
737tcywiki 35302
738tenwiki 1507
739test2wiki 224134
740testwiki 123678
741testwikidatawiki 82093
742tetwiki 6830
743tewiki 672086
744tewikibooks 452
745tewikiquote 2942
746tewikisource 76623
747tewiktionary 295706
748tgwiki 283822
749tgwikibooks 390
750tgwiktionary 40149
751thwikibooks 11276
752thwikinews 104
753thwikiquote 5860
754thwikisource 11656
755thwiktionary 323587
756tiwiki 1863
757tiwiktionary 783
758tkwiki 19206
759tkwikibooks 37
760tkwikiquote 19
761tkwiktionary 3460
762tlwiki 215546
763tlwikibooks 3792
764tlwiktionary 36356
765tnwiki 3049
766tnwiktionary 785
767towiki 2186
768towiktionary 24
769tpiwiki 6174
770tpiwiktionary 949
771transitionteamwiki 292
772trwikibooks 5405
773trwikimedia 415
774trwikinews 64744
775trwikiquote 30906
776trwikisource 8286
777trwiktionary 433083
778tswiki 2837
779tswiktionary 743
780ttwiki 777740
781ttwikibooks 366
782ttwikiquote 36
783ttwiktionary 21975
784tumwiki 2136
785twwiki 2227
786twwiktionary 19
787tyvwiki 10334
788tywiki 4548
789uawikimedia 20129
790udmwiki 12942
791ugwiki 12039
792ugwikibooks 49
793ugwikiquote 21
794ugwiktionary 6769
795ukwikibooks 4686
796ukwikimedia 8444
797ukwikinews 12632
798ukwikiquote 24095
799ukwikisource 31565
800ukwikivoyage 8362
801ukwiktionary 35194
802urwiki 991030
803urwikibooks 1028
804urwikiquote 1790
805urwiktionary 2351
806usabilitywiki 449
807uzwiki 625221
808uzwikibooks 117
809uzwikiquote 269
810uzwiktionary 228037
811vecwiki 47794
812vecwikisource 5534
813vecwiktionary 10092
814vepwiki 29031
815vewiki 1599
816viwikibooks 35743
817viwikiquote 10319
818viwikisource 9809
819viwikivoyage 17013
820viwiktionary 164294
821vlswiki 23486
822votewiki 399
823vowiki 182916
824vowikibooks 36
825vowikiquote 45
826vowiktionary 13867
827warwiki 2366587
828wawiki 21739
829wawikibooks 14
830wawiktionary 39109
831wg_enwiki NULL
832wikimania2005wiki 34
833wikimania2006wiki 543
834wikimania2007wiki 2242
835wikimania2008wiki 1611
836wikimania2009wiki 460
837wikimania2010wiki 188
838wikimania2011wiki 260
839wikimania2012wiki 364
840wikimania2013wiki 11113
841wikimania2014wiki 25825
842wikimania2015wiki 26322
843wikimania2016wiki 20531
844wikimania2017wiki 12721
845wikimaniateamwiki 390
846wowiki 5467
847wowikiquote 212
848wowiktionary 2420
849wuuwiki 31111
850xalwiki 7047
851xhwiki 4121
852xhwikibooks 20
853xhwiktionary 37
854xmfwiki 32852
855yiwiki 43185
856yiwikisource 457
857yiwiktionary 986
858yowiki 60603
859yowikibooks 29
860yowiktionary 35
861zawiki 3033
862zawikibooks 12
863zawikiquote 17
864zawiktionary 64
865zeawiki 16026
866zerowiki 5050
867zh_classicalwiki 32882
868zh_min_nanwiki 629749
869zh_min_nanwikibooks 60
870zh_min_nanwikiquote 68
871zh_min_nanwikisource 2184
872zh_min_nanwiktionary 28121
873zhwikibooks 17037
874zhwikinews 19776
875zhwikiquote 13621
876zhwikisource 249295
877zhwikivoyage 40085
878zhwiktionary 514763
879zh_yuewiki 200281
880zuwiki 4969
881zuwikibooks 20
882zuwiktionary 1812
883commonswiki 82350084
884dewiki 28595018
885wikidatawiki 241176029
886frwiki 27512786
887jawiki 10291153
888ruwiki 19072238
889arwiki 7231773
890cawiki 4196777
891eswiki 21588048
892fawiki 6468735
893frwiktionary 4974604
894hewiki 3930946
895huwiki 3214305
896kowiki 4813984
897metawiki 6343097
898rowiki 2122294
899ukwiki 4967529
900viwiki 9187800

I vote for repooling this host again to see if it fails again when it gets load

Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2017-04-26T15:24:31Z] <marostegui> Shutdown es2019 for maintenance with papaul and Dell - T149526

Hi Papaul,

Thank you for contacting Dell EMC Basic Server Support.

This mail is with reference to the (Memory and CPU Issue) you had reported on your PowerEdge(R730XD).

Please find the service request number for your reference in below. Please reply back with the address of the location where you want the part to be delivered

Service Tag : 8PVW382

Service request number : 947570621

Please feel free to reply to the email if you have any other concerns since I would be able to reply back and help you further.

Regards

Suraj Kumar

Enterprise Tech Support Analyst

Dell EMC | NA Basic Server Support

Enterprise Remote Services and Solutions

Hi Papaul,

I will get the motherboard and the memory module replaced at the same time but at the same time would like to request you to help me with the address of the location where you want the parts and the engineer to be sent .

Thanks and regards

Suraj Kumar

Thanks Papaul! As per our chat, I have brought MySQL, ping me when you need it down again,

Mentioned in SAL (#wikimedia-operations) [2017-04-27T14:29:54Z] <marostegui> Stop MySQL and shutdown es2019 for HW replacement - T149526

Main board replacement
DIMM B4 Replaced
DIMM A1 Replaced
BIOS update from 2.2.5 to 2.4.3

Papaul reassigned this task from Papaul to Marostegui.Apr 27 2017, 4:54 PM

Thanks @Papaul!
Let's see how it goes.

Marostegui closed this task as Resolved.May 3 2017, 7:31 AM

Let's close this for now.

jcrespo reopened this task as Open.May 5 2017, 10:26 AM

Pending to run compare.py around the ids obtained before.

jcrespo claimed this task.May 5 2017, 10:44 AM
jcrespo moved this task from Blocked external/Not db team to Next on the DBA board.
jcrespo removed a project: ops-codfw.

Left running on neodymium (I did some optimizations to ignore values below and beyond max id respectively):

while read db id; do echo "./compare.py es1019.eqiad.wmnet es2019.codfw.wmnet $db blobs_cluster25 blob_id --step=100 --from-value $(($id - 300000)) --to-value $(($id + 300000))"; ./compare.py es1019.eqiad.wmnet es2019.codfw.wmnet $db blobs_cluster25 blob_id --step=100 --from-value $(($id - 300000)) --to-value $(($id + 300000)) ; done < P5309

Change 352752 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool es2019

https://gerrit.wikimedia.org/r/352752

Change 352752 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool es2019

https://gerrit.wikimedia.org/r/352752

Mentioned in SAL (#wikimedia-operations) [2017-05-09T07:12:23Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool es2019 - T149526 (duration: 00m 39s)

Marostegui closed this task as Resolved.May 9 2017, 7:13 AM

As there were no differences found by compare.py I have repooled this host. Finger crossed so it doesn't crash anymore!