Page MenuHomePhabricator

Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (305 w, 6 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Today

Papaul updated the task description for T265419: (Need By: TBD) rack/setup/install ms-be20[58-61].
Wed, Oct 28, 3:23 PM · Operations, ops-codfw, DC-Ops
Papaul moved T266577: codfw: relocate sessionstore2002 and mc2029 from C4 to C3 from Backlog to Racking Tasks on the ops-codfw board.
Wed, Oct 28, 3:17 PM · User-jijiki, serviceops, ops-codfw, Operations
Papaul updated the task description for T243450: Audit & update spares part tracking for all sites.
Wed, Oct 28, 3:11 PM · ops-eqiad, ops-codfw, DC-Ops, Operations
Papaul closed T265558: decommission wtp2001 through wtp2020 as Resolved.

complete

Wed, Oct 28, 3:08 PM · Operations, ops-codfw, DC-Ops, decommission-hardware
Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Wed, Oct 28, 3:07 PM · Operations, ops-codfw, DC-Ops, decommission-hardware
Papaul triaged T266577: codfw: relocate sessionstore2002 and mc2029 from C4 to C3 as Medium priority.
Wed, Oct 28, 3:04 PM · User-jijiki, serviceops, ops-codfw, Operations

Yesterday

Papaul created T266577: codfw: relocate sessionstore2002 and mc2029 from C4 to C3 .
Tue, Oct 27, 4:25 PM · User-jijiki, serviceops, ops-codfw, Operations
Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Tue, Oct 27, 4:05 PM · Operations, ops-codfw, DC-Ops, decommission-hardware
Papaul added a comment to T265558: decommission wtp2001 through wtp2020.
[edit interfaces interface-range vlan-private1-c-codfw]
-    member-range ge-4/0/1 to ge-4/0/10;
[edit interfaces interface-range disabled]
     member xe-2/0/5 { ... }
+    member ge-4/0/1;
+    member ge-4/0/2;
+    member ge-4/0/3;
+    member ge-4/0/4;
+    member ge-4/0/5;
+    member ge-4/0/6;
+    member ge-4/0/7;
+    member ge-4/0/8;
+    member ge-4/0/9;
+    member ge-4/0/10;
[edit interfaces ge-4/0/1]
-   description wtp2011;
[edit interfaces ge-4/0/2]
-   description wtp2012;
[edit interfaces ge-4/0/3]
-   description wtp2013;
[edit interfaces ge-4/0/4]
-   description wtp2014;
[edit interfaces ge-4/0/5]
-   description wtp2015;
[edit interfaces ge-4/0/6]
-   description wtp2016;
[edit interfaces ge-4/0/7]
-   description wtp2017;
[edit interfaces ge-4/0/8]
-   description wtp2018;
[edit interfaces ge-4/0/9]
-   description wtp2019;
[edit interfaces ge-4/0/10]
-   description wtp2020;
Tue, Oct 27, 3:56 PM · Operations, ops-codfw, DC-Ops, decommission-hardware
Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Tue, Oct 27, 1:23 PM · Operations, ops-codfw, DC-Ops, decommission-hardware

Fri, Oct 23

Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Fri, Oct 23, 9:41 PM · Operations, ops-codfw, DC-Ops, decommission-hardware
Papaul added a comment to T266261: rearrange networking for cloudceph200[1-3]-dev.

@Andrew I can help with it.

Fri, Oct 23, 4:39 AM · cloud-services-team (Kanban)

Thu, Oct 22

Papaul closed T264386: decommission es2017.codfw.wmnet as Resolved.

complete

Thu, Oct 22, 7:02 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul closed T264386: decommission es2017.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Thu, Oct 22, 7:02 PM · DBA
Papaul updated the task description for T264386: decommission es2017.codfw.wmnet.
Thu, Oct 22, 7:02 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul reassigned T266214: Degraded RAID on ms-be2017 from Papaul to fgiunchedi.

@fgiunchedi disk replaced

Thu, Oct 22, 7:01 PM · SRE-swift-storage, Operations, ops-codfw
Papaul added a comment to T265558: decommission wtp2001 through wtp2020.
[edit interfaces interface-range disabled]
     member xe-2/0/6 { ... }
+    member ge-4/0/17;
+    member ge-4/0/18;
+    member ge-4/0/19;
+    member ge-4/0/20;
+    member ge-4/0/22;
+    member ge-4/0/23;
+    member ge-4/0/24;
+    member ge-4/0/25;
[edit interfaces]
-   ge-4/0/17 {
-       description wtp2001;
-       enable;
-   }
-   ge-4/0/18 {
-       description wtp2002;
-       enable;
-   }
-   ge-4/0/19 {
-       description wtp2003;
-       enable;
-   }
-   ge-4/0/20 {
-       description wtp2004;
-       enable;
-   }
-   ge-4/0/22 {
-       description wtp2006;
-   }
-   ge-4/0/23 {
-       description wtp2007;
-       enable;
-   }
-   ge-4/0/24 {
-       description wtp2008;
-       enable;
-   }
-   ge-4/0/25 {
-       description wtp2009;
-       enable;
-   }
Thu, Oct 22, 5:03 PM · Operations, ops-codfw, DC-Ops, decommission-hardware
Papaul claimed T264386: decommission es2017.codfw.wmnet.
Thu, Oct 22, 4:49 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul updated the task description for T264386: decommission es2017.codfw.wmnet.
Thu, Oct 22, 4:48 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul added a comment to T264386: decommission es2017.codfw.wmnet.
papaul@asw-a-codfw# show | compare
[edit interfaces interface-range vlan-private1-a-codfw]
-    member ge-6/0/19;
[edit interfaces interface-range disabled]
     member xe-2/0/6 { ... }
+    member ge-6/0/19;
[edit interfaces]
-   ge-6/0/19 {
-       description es2017;
-       enable;
-   }
Thu, Oct 22, 4:37 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul moved T265435: codfw: Testing Out Sample PDUs from Backlog to Racking Tasks on the ops-codfw board.
Thu, Oct 22, 3:36 PM · ops-codfw, Operations, DC-Ops
Papaul moved T265419: (Need By: TBD) rack/setup/install ms-be20[58-61] from Backlog to Racking Tasks on the ops-codfw board.
Thu, Oct 22, 3:35 PM · Operations, ops-codfw, DC-Ops
Papaul moved T265558: decommission wtp2001 through wtp2020 from Backlog to Decommission on the ops-codfw board.
Thu, Oct 22, 3:35 PM · Operations, ops-codfw, DC-Ops, decommission-hardware
Papaul moved T266214: Degraded RAID on ms-be2017 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Thu, Oct 22, 3:35 PM · SRE-swift-storage, Operations, ops-codfw

Wed, Oct 21

Papaul closed T265412: patch in FB peering into cr2-eqdfw as Resolved.

complete

Wed, Oct 21, 3:00 PM · netops, ops-codfw, Operations
Papaul updated the task description for T265412: patch in FB peering into cr2-eqdfw.
Wed, Oct 21, 2:56 PM · netops, ops-codfw, Operations
Papaul added a comment to T265412: patch in FB peering into cr2-eqdfw.
Physical interface: xe-0/1/2
    Laser bias current                        :  37.980 mA
    Laser output power                        :  0.3080 mW / -5.11 dBm
    Module temperature                        :  36 degrees C / 97 degrees F
    Module voltage                            :  3.2780 V
    Laser receiver power                      :  0.6121 mW / -2.13 dBm
Wed, Oct 21, 2:54 PM · netops, ops-codfw, Operations
Papaul added a comment to T265412: patch in FB peering into cr2-eqdfw.
papaul@cr2-eqdfw> show interfaces xe-0/1/2 descriptions
Interface       Admin Link Description
xe-0/1/2        up    up   Reserved for Facebook PNI - no-mon
Wed, Oct 21, 2:52 PM · netops, ops-codfw, Operations

Mon, Oct 19

Papaul closed T265729: decommission cp2003, cp2009, cp2015, cp2021 as Resolved.

complete

Mon, Oct 19, 3:43 PM · Operations, ops-codfw, decommission-hardware
Papaul updated the task description for T265729: decommission cp2003, cp2009, cp2015, cp2021.
Mon, Oct 19, 3:43 PM · Operations, ops-codfw, decommission-hardware
Papaul closed T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers as Resolved.

@fgiunchedi we just decom cp2021 which was in rack D2 on U3 so no need to move thanos-fe2003 i can use U2 and U3 to rack the new ms-be server

Mon, Oct 19, 3:42 PM · ops-codfw, Operations
Papaul updated the task description for T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers .
Mon, Oct 19, 3:41 PM · ops-codfw, Operations
Papaul added a comment to T265412: patch in FB peering into cr2-eqdfw.

On site visit set for the 21St

Mon, Oct 19, 3:27 PM · netops, ops-codfw, Operations

Fri, Oct 16

Papaul updated the task description for T265729: decommission cp2003, cp2009, cp2015, cp2021.
Fri, Oct 16, 9:47 PM · Operations, ops-codfw, decommission-hardware
Papaul updated the task description for T265729: decommission cp2003, cp2009, cp2015, cp2021.
Fri, Oct 16, 9:38 PM · Operations, ops-codfw, decommission-hardware
Papaul added a comment to T265729: decommission cp2003, cp2009, cp2015, cp2021.
[edit interfaces interface-range disabled]
     member ge-1/0/5 { ... }
+    member xe-2/0/6;
[edit interfaces]
-   xe-2/0/6 {
-       description cp2003;
-       enable;
-   }
[edit interfaces interface-range disabled]
     member ge-1/0/9 { ... }
+    member xe-2/0/6;
[edit interfaces]
-   xe-2/0/6 {
-       description cp2009;
-       enable;
-   }
[edit interfaces interface-range disabled]
     member ge-1/0/1 { ... }
+    member xe-2/0/5;
[edit interfaces]
-   xe-2/0/5 {
-       description cp2015;
-       enable;
-   }
edit interfaces interface-range disabled]
     member ge-1/0/5 { ... }
+    member xe-2/0/2;
[edit interfaces]
-   xe-2/0/2 {
-       description cp2021;
-       enable;
-   }
Fri, Oct 16, 9:35 PM · Operations, ops-codfw, decommission-hardware

Thu, Oct 15

Papaul triaged T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers as Medium priority.
Thu, Oct 15, 6:20 PM · ops-codfw, Operations
Papaul created T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers .
Thu, Oct 15, 6:20 PM · ops-codfw, Operations
Papaul closed T264156: decommission es2016.codfw.wmnet as Resolved.

complete

Thu, Oct 15, 6:04 PM · ops-codfw, Operations, decommission-hardware
Papaul closed T264156: decommission es2016.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Thu, Oct 15, 6:04 PM · DBA
Papaul updated the task description for T264156: decommission es2016.codfw.wmnet.
Thu, Oct 15, 6:04 PM · ops-codfw, Operations, decommission-hardware
Papaul closed T264700: decommission es2015.codfw.wmnet as Resolved.

complete

Thu, Oct 15, 6:03 PM · ops-codfw, Operations, DC-Ops, decommission-hardware
Papaul closed T264700: decommission es2015.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Thu, Oct 15, 6:03 PM · DBA
Papaul updated the task description for T264700: decommission es2015.codfw.wmnet.
Thu, Oct 15, 6:03 PM · ops-codfw, Operations, DC-Ops, decommission-hardware
Papaul closed T264261: decommission es2011.codfw.wmnet as Resolved.

complete

Thu, Oct 15, 5:52 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul closed T264261: decommission es2011.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Thu, Oct 15, 5:52 PM · DBA
Papaul updated the task description for T264261: decommission es2011.codfw.wmnet.
Thu, Oct 15, 5:51 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul reassigned T264698: Degraded RAID on mw2279 from Papaul to jijiki.

@jijiki disk replaced

Thu, Oct 15, 5:00 PM · serviceops, Operations, ops-codfw
Papaul updated the task description for T265412: patch in FB peering into cr2-eqdfw.
Thu, Oct 15, 3:06 PM · netops, ops-codfw, Operations
Papaul reassigned T265208: Degraded RAID on ms-be2036 from Papaul to fgiunchedi.

@fgiunchedi disk replaced

Thu, Oct 15, 2:48 PM · Operations, ops-codfw

Wed, Oct 14

Papaul closed T262809: ps1-b3-codfw AB feed current > 12A as Resolved.

I balance the power from

	 A:X 	 Master_X 	 9.95 A 	
0
31
	 Normal 
	 A:Y 	 Master_Y 	 11.39 A 	
0
31
	 Normal 
	 A:Z 	 Master_Z 	 10.65 A 	
0
31
	 Normal 
	 B:X 	 Link_X 	 9.88 A 	
0
31
	 Normal 
	 B:Y 	 Link_Y 	 9.27 A 	
0
31
	 Normal 
	 B:Z 	 Link_Z 	 8.81 A

to

	 A:X 	 Master_X 	 9.83 A 	
0
31
	 Normal 
	 A:Y 	 Master_Y 	 10.33 A 	
0
31
	 Normal 
	 A:Z 	 Master_Z 	 9.90 A 	
0
31
	 Normal 
	 B:X 	 Link_X 	 9.88 A 	
0
31
	 Normal 
	 B:Y 	 Link_Y 	 10.35 A 	
0
31
	 Normal 
	 B:Z 	 Link_Z 	 9.65 A
Wed, Oct 14, 5:13 PM · Operations, ops-codfw
Papaul updated the task description for T264156: decommission es2016.codfw.wmnet.
Wed, Oct 14, 3:52 PM · ops-codfw, Operations, decommission-hardware
Papaul added a comment to T264156: decommission es2016.codfw.wmnet.
papaul@asw-d-codfw# show | compare
[edit interfaces interface-range vlan-private1-d-codfw]
-    member ge-1/0/5;
[edit interfaces interface-range disabled]
     member ge-6/0/19 { ... }
+    member ge-1/0/5;
[edit interfaces]
-   ge-1/0/5 {
-       description es2016;
-       enable;
-   }
Wed, Oct 14, 3:52 PM · ops-codfw, Operations, decommission-hardware
Papaul updated the task description for T264700: decommission es2015.codfw.wmnet.
Wed, Oct 14, 3:48 PM · ops-codfw, Operations, DC-Ops, decommission-hardware
Papaul added a comment to T264700: decommission es2015.codfw.wmnet.
papaul@asw-c-codfw# show | compare
[edit interfaces interface-range vlan-private1-c-codfw]
-    member ge-1/0/1;
[edit interfaces interface-range disabled]
     member ge-1/0/0 { ... }
+    member ge-1/0/1;
[edit interfaces]
-   ge-1/0/1 {
-       description es2015;
-   }
Wed, Oct 14, 3:48 PM · ops-codfw, Operations, DC-Ops, decommission-hardware
Papaul updated the task description for T264261: decommission es2011.codfw.wmnet.
Wed, Oct 14, 3:45 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul added a comment to T264261: decommission es2011.codfw.wmnet.
papaul@asw-b-codfw# show | compare
[edit interfaces interface-range vlan-private1-b-codfw]
-    member ge-1/0/9;
[edit interfaces interface-range disabled]
     member ge-6/0/19 { ... }
+    member ge-1/0/9;
[edit interfaces]
-   ge-1/0/9 {
-       description es2011;
-       enable;
-   }
Wed, Oct 14, 3:44 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul added a comment to T265208: Degraded RAID on ms-be2036.

@fgiunchedi yes we do have 3 spares. Will replace when back on site tomorrow.

Wed, Oct 14, 2:17 PM · Operations, ops-codfw

Tue, Oct 13

Papaul added a comment to T264698: Degraded RAID on mw2279.

Create Dispatch: Success
You have successfully submitted request SR1039679642.

Tue, Oct 13, 9:41 PM · serviceops, Operations, ops-codfw
Papaul added a comment to T264698: Degraded RAID on mw2279.

@jijiki I will request a disk replacement

Tue, Oct 13, 5:57 PM · serviceops, Operations, ops-codfw
Papaul added a comment to T265208: Degraded RAID on ms-be2036.

@fgiunchedi looks like icinga is happy now

Tue, Oct 13, 5:56 PM · Operations, ops-codfw
Papaul added a comment to T265208: Degraded RAID on ms-be2036.

The Flash/SD-CARD problem was fixed by formatting the NAND and draining the power

Tue, Oct 13, 5:56 PM · Operations, ops-codfw
Papaul added a comment to T263837: Degraded RAID on es2026.

return tracking information

Tue, Oct 13, 5:15 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T265208: Degraded RAID on ms-be2036.

Upgrade ILO from 2.50 to 2.74

Tue, Oct 13, 4:39 PM · Operations, ops-codfw
Papaul added a comment to T265208: Degraded RAID on ms-be2036.

@fgiunchedi

Embedded Flash/SD-CARD 		Controller firmware revision 2.10.00 Embedded media manager failed media attach
Tue, Oct 13, 4:38 PM · Operations, ops-codfw
Papaul added a comment to T260670: db2125 crashed - mgmt iface also not available.

Both CPU replaced, servers is back up

Tue, Oct 13, 3:40 PM · User-Kormat, ops-codfw, DBA, Operations
Papaul added a comment to T263837: Degraded RAID on es2026.
pt1979@es2026:~$ sudo megacli -PDRbld -ShowProg -physdrv[32:2] -aALL
Tue, Oct 13, 3:08 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T263837: Degraded RAID on es2026.

new disk in place

 		Status 	Name 	State 	Slot Number 	Size 	Security Status 	Bus Protocol 	Media Type 	Hot Spare 	Remaining Rated Write Endurance
			Physical Disk 0:1:0 	Online 	0 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:1 	Online 	1 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:2 	Online 	2 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:3 	Online 	3 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:4 	Online 	4 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:5 	Online 	5 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:6 	Online 	6 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:7 	Online 	7 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:8 	Online 	8 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:9 	Online 	9 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
Tue, Oct 13, 3:02 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T263837: Degraded RAID on es2026.

Will do that once on site

Tue, Oct 13, 12:48 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T263837: Degraded RAID on es2026.

@Marostegui since the server is under warranty, it is best to use a disk that is under warranty as well.

Tue, Oct 13, 12:45 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul triaged T265208: Degraded RAID on ms-be2036 as Medium priority.
Tue, Oct 13, 12:43 PM · Operations, ops-codfw
Papaul triaged T264698: Degraded RAID on mw2279 as Medium priority.
Tue, Oct 13, 12:42 PM · serviceops, Operations, ops-codfw

Mon, Oct 12

Papaul moved T265208: Degraded RAID on ms-be2036 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Mon, Oct 12, 3:59 PM · Operations, ops-codfw
Papaul moved T264386: decommission es2017.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Mon, Oct 12, 3:59 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul moved T264156: decommission es2016.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Mon, Oct 12, 3:59 PM · ops-codfw, Operations, decommission-hardware
Papaul moved T264261: decommission es2011.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Mon, Oct 12, 3:58 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul moved T264700: decommission es2015.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Mon, Oct 12, 3:58 PM · ops-codfw, Operations, DC-Ops, decommission-hardware
Papaul moved T264698: Degraded RAID on mw2279 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Mon, Oct 12, 3:58 PM · serviceops, Operations, ops-codfw

Wed, Sep 30

Papaul moved T263837: Degraded RAID on es2026 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Sep 30, 5:47 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul closed T263740: decommission es2013.codfw.wmnet as Resolved.

complete

Wed, Sep 30, 5:45 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul closed T263740: decommission es2013.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Wed, Sep 30, 5:45 PM · DBA
Papaul updated the task description for T263740: decommission es2013.codfw.wmnet.
Wed, Sep 30, 5:45 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul closed T264063: decommission es2019.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Wed, Sep 30, 5:44 PM · DBA
Papaul closed T264063: decommission es2019.codfw.wmnet as Resolved.
Wed, Sep 30, 5:44 PM · DC-Ops, ops-codfw, Operations, decommission-hardware
Papaul added a comment to T264063: decommission es2019.codfw.wmnet.

complete

Wed, Sep 30, 5:43 PM · DC-Ops, ops-codfw, Operations, decommission-hardware
Papaul updated the task description for T264063: decommission es2019.codfw.wmnet.
Wed, Sep 30, 5:43 PM · DC-Ops, ops-codfw, Operations, decommission-hardware
Papaul updated the task description for T264063: decommission es2019.codfw.wmnet.
Wed, Sep 30, 4:41 PM · DC-Ops, ops-codfw, Operations, decommission-hardware
Papaul added a comment to T264063: decommission es2019.codfw.wmnet.
papaul@asw-d-codfw# show | compare
[edit interfaces interface-range vlan-private1-d-codfw]
-    member ge-6/0/19;
[edit interfaces interface-range disabled]
     member ge-1/0/4 { ... }
+    member ge-6/0/19;
[edit interfaces]
-   ge-6/0/19 {
-       description es2019;
-       enable;
-   }
Wed, Sep 30, 4:30 PM · DC-Ops, ops-codfw, Operations, decommission-hardware
Papaul added a comment to T263837: Degraded RAID on es2026.

Return tracking information

Wed, Sep 30, 4:10 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T263837: Degraded RAID on es2026.

I create another dispatch to request a new disk and shipped the one received on 9/25/2020 back.

Wed, Sep 30, 3:11 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T263837: Degraded RAID on es2026.
 		Status 	Name 	State 	Slot Number 	Size 	Security Status 	Bus Protocol 	Media Type 	Hot Spare 	Remaining Rated Write Endurance
			Physical Disk 0:1:0 	Online 	0 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:1 	Online 	1 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:2 	Online 	2 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:3 	Online 	3 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:4 	Online 	4 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:5 	Online 	5 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:6 	Online 	6 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:7 	Online 	7 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:8 	Online 	8 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
			Physical Disk 0:1:9 	Online 	9 	1862.5 GB	Not Capable 	SATA 	HDD 	No 	Not Applicable
Wed, Sep 30, 2:46 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T263837: Degraded RAID on es2026.

The disk from one of the decom es server works

Wed, Sep 30, 2:46 PM · Patch-For-Review, DBA, Operations, ops-codfw

Tue, Sep 29

Papaul added a comment to T260670: db2125 crashed - mgmt iface also not available.
2020-09-29 20:47:59 	287 	SYS1003 	System CPU Resetting.
2020-09-29 20:47:51 	286 	PWR2270 	The Intel Management Engine has encountered a Health Event.
2020-09-29 20:47:47 	285 	SYS1000 	System is turning on.
2020-09-29 20:47:38 	284 	SYS1001 	System is turning off.
2020-09-29 20:47:38 	283 	SYS1003 	System CPU Resetting.
2020-09-29 20:47:21 	282 	RAC0703 	Requested system hardreset.
2020-09-29 20:47:12 	281 	**``//CPU0000//``** 	Internal error has occurred check for additional logs.
Tue, Sep 29, 10:55 PM · User-Kormat, ops-codfw, DBA, Operations
Papaul updated the task description for T263740: decommission es2013.codfw.wmnet.
Tue, Sep 29, 3:24 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul updated the task description for T263740: decommission es2013.codfw.wmnet.
Tue, Sep 29, 3:13 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul added a comment to T263740: decommission es2013.codfw.wmnet.
papaul@asw-d-codfw# show | compare
[edit interfaces interface-range vlan-private1-d-codfw]
-    member ge-1/0/4;
[edit interfaces interface-range disabled]
     member xe-7/0/3 { ... }
+    member ge-1/0/4;
[edit interfaces]
-   ge-1/0/4 {
-       description es2013;
-       enable;
-   }
Tue, Sep 29, 3:13 PM · Operations, DC-Ops, ops-codfw, decommission-hardware
Papaul placed T245192: Investigate Juniper storm control up for grabs.
Tue, Sep 29, 1:28 AM · Operations, netops
Papaul added a comment to T263714: elastic2037 DIMM errors logged in racadm getsel.

return information

Tue, Sep 29, 1:27 AM · Operations, ops-codfw

Sep 28 2020

Papaul closed T263714: elastic2037 DIMM errors logged in racadm getsel as Resolved.

DiMM B1 replaced. All good now.

Sep 28 2020, 4:11 PM · Operations, ops-codfw
Papaul closed T263615: decommission es2018.codfw.wmnet as Resolved.

complete

Sep 28 2020, 3:45 PM · ops-codfw, Operations, decommission-hardware