User Details
User Details
- User Since
- Dec 18 2014, 3:39 PM (305 w, 6 d)
- Availability
- Available
- LDAP User
- Papaul
- MediaWiki User
- Unknown
Today
Today
Papaul updated the task description for T265419: (Need By: TBD) rack/setup/install ms-be20[58-61].
Papaul moved T266577: codfw: relocate sessionstore2002 and mc2029 from C4 to C3 from Backlog to Racking Tasks on the ops-codfw board.
Papaul updated the task description for T243450: Audit & update spares part tracking for all sites.
complete
Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Papaul triaged T266577: codfw: relocate sessionstore2002 and mc2029 from C4 to C3 as Medium priority.
Yesterday
Yesterday
Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Papaul added a comment to T265558: decommission wtp2001 through wtp2020.
[edit interfaces interface-range vlan-private1-c-codfw]
- member-range ge-4/0/1 to ge-4/0/10;
[edit interfaces interface-range disabled]
member xe-2/0/5 { ... }
+ member ge-4/0/1;
+ member ge-4/0/2;
+ member ge-4/0/3;
+ member ge-4/0/4;
+ member ge-4/0/5;
+ member ge-4/0/6;
+ member ge-4/0/7;
+ member ge-4/0/8;
+ member ge-4/0/9;
+ member ge-4/0/10;
[edit interfaces ge-4/0/1]
- description wtp2011;
[edit interfaces ge-4/0/2]
- description wtp2012;
[edit interfaces ge-4/0/3]
- description wtp2013;
[edit interfaces ge-4/0/4]
- description wtp2014;
[edit interfaces ge-4/0/5]
- description wtp2015;
[edit interfaces ge-4/0/6]
- description wtp2016;
[edit interfaces ge-4/0/7]
- description wtp2017;
[edit interfaces ge-4/0/8]
- description wtp2018;
[edit interfaces ge-4/0/9]
- description wtp2019;
[edit interfaces ge-4/0/10]
- description wtp2020;Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Fri, Oct 23
Fri, Oct 23
Papaul updated the task description for T265558: decommission wtp2001 through wtp2020.
Papaul added a comment to T266261: rearrange networking for cloudceph200[1-3]-dev.
@Andrew I can help with it.
Thu, Oct 22
Thu, Oct 22
complete
Papaul closed T264386: decommission es2017.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Papaul updated the task description for T264386: decommission es2017.codfw.wmnet.
@fgiunchedi disk replaced
Papaul added a comment to T265558: decommission wtp2001 through wtp2020.
[edit interfaces interface-range disabled]
member xe-2/0/6 { ... }
+ member ge-4/0/17;
+ member ge-4/0/18;
+ member ge-4/0/19;
+ member ge-4/0/20;
+ member ge-4/0/22;
+ member ge-4/0/23;
+ member ge-4/0/24;
+ member ge-4/0/25;
[edit interfaces]
- ge-4/0/17 {
- description wtp2001;
- enable;
- }
- ge-4/0/18 {
- description wtp2002;
- enable;
- }
- ge-4/0/19 {
- description wtp2003;
- enable;
- }
- ge-4/0/20 {
- description wtp2004;
- enable;
- }
- ge-4/0/22 {
- description wtp2006;
- }
- ge-4/0/23 {
- description wtp2007;
- enable;
- }
- ge-4/0/24 {
- description wtp2008;
- enable;
- }
- ge-4/0/25 {
- description wtp2009;
- enable;
- }Papaul updated the task description for T264386: decommission es2017.codfw.wmnet.
Papaul added a comment to T264386: decommission es2017.codfw.wmnet.
papaul@asw-a-codfw# show | compare
[edit interfaces interface-range vlan-private1-a-codfw]
- member ge-6/0/19;
[edit interfaces interface-range disabled]
member xe-2/0/6 { ... }
+ member ge-6/0/19;
[edit interfaces]
- ge-6/0/19 {
- description es2017;
- enable;
- }Papaul moved T265435: codfw: Testing Out Sample PDUs from Backlog to Racking Tasks on the ops-codfw board.
Papaul moved T265419: (Need By: TBD) rack/setup/install ms-be20[58-61] from Backlog to Racking Tasks on the ops-codfw board.
Papaul moved T265558: decommission wtp2001 through wtp2020 from Backlog to Decommission on the ops-codfw board.
Papaul moved T266214: Degraded RAID on ms-be2017 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Oct 21
Wed, Oct 21
complete
Papaul updated the task description for T265412: patch in FB peering into cr2-eqdfw.
Papaul added a comment to T265412: patch in FB peering into cr2-eqdfw.
Physical interface: xe-0/1/2
Laser bias current : 37.980 mA
Laser output power : 0.3080 mW / -5.11 dBm
Module temperature : 36 degrees C / 97 degrees F
Module voltage : 3.2780 V
Laser receiver power : 0.6121 mW / -2.13 dBmPapaul added a comment to T265412: patch in FB peering into cr2-eqdfw.
papaul@cr2-eqdfw> show interfaces xe-0/1/2 descriptions Interface Admin Link Description xe-0/1/2 up up Reserved for Facebook PNI - no-mon
Mon, Oct 19
Mon, Oct 19
complete
Papaul updated the task description for T265729: decommission cp2003, cp2009, cp2015, cp2021.
Papaul closed T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers as Resolved.
@fgiunchedi we just decom cp2021 which was in rack D2 on U3 so no need to move thanos-fe2003 i can use U2 and U3 to rack the new ms-be server
Papaul updated the task description for T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers .
Papaul added a comment to T265412: patch in FB peering into cr2-eqdfw.
On site visit set for the 21St
Fri, Oct 16
Fri, Oct 16
Papaul updated the task description for T265729: decommission cp2003, cp2009, cp2015, cp2021.
Papaul updated the task description for T265729: decommission cp2003, cp2009, cp2015, cp2021.
Papaul added a comment to T265729: decommission cp2003, cp2009, cp2015, cp2021.
[edit interfaces interface-range disabled]
member ge-1/0/5 { ... }
+ member xe-2/0/6;
[edit interfaces]
- xe-2/0/6 {
- description cp2003;
- enable;
- }[edit interfaces interface-range disabled]
member ge-1/0/9 { ... }
+ member xe-2/0/6;
[edit interfaces]
- xe-2/0/6 {
- description cp2009;
- enable;
- }[edit interfaces interface-range disabled]
member ge-1/0/1 { ... }
+ member xe-2/0/5;
[edit interfaces]
- xe-2/0/5 {
- description cp2015;
- enable;
- }edit interfaces interface-range disabled]
member ge-1/0/5 { ... }
+ member xe-2/0/2;
[edit interfaces]
- xe-2/0/2 {
- description cp2021;
- enable;
- }Thu, Oct 15
Thu, Oct 15
Papaul triaged T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers as Medium priority.
complete
Papaul closed T264156: decommission es2016.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Papaul updated the task description for T264156: decommission es2016.codfw.wmnet.
complete
Papaul closed T264700: decommission es2015.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Papaul updated the task description for T264700: decommission es2015.codfw.wmnet.
complete
Papaul closed T264261: decommission es2011.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Papaul updated the task description for T264261: decommission es2011.codfw.wmnet.
@jijiki disk replaced
Papaul updated the task description for T265412: patch in FB peering into cr2-eqdfw.
@fgiunchedi disk replaced
Wed, Oct 14
Wed, Oct 14
I balance the power from
A:X Master_X 9.95 A 0 31 Normal A:Y Master_Y 11.39 A 0 31 Normal A:Z Master_Z 10.65 A 0 31 Normal B:X Link_X 9.88 A 0 31 Normal B:Y Link_Y 9.27 A 0 31 Normal B:Z Link_Z 8.81 A
to
A:X Master_X 9.83 A 0 31 Normal A:Y Master_Y 10.33 A 0 31 Normal A:Z Master_Z 9.90 A 0 31 Normal B:X Link_X 9.88 A 0 31 Normal B:Y Link_Y 10.35 A 0 31 Normal B:Z Link_Z 9.65 A
Papaul updated the task description for T264156: decommission es2016.codfw.wmnet.
Papaul added a comment to T264156: decommission es2016.codfw.wmnet.
papaul@asw-d-codfw# show | compare
[edit interfaces interface-range vlan-private1-d-codfw]
- member ge-1/0/5;
[edit interfaces interface-range disabled]
member ge-6/0/19 { ... }
+ member ge-1/0/5;
[edit interfaces]
- ge-1/0/5 {
- description es2016;
- enable;
- }Papaul updated the task description for T264700: decommission es2015.codfw.wmnet.
Papaul added a comment to T264700: decommission es2015.codfw.wmnet.
papaul@asw-c-codfw# show | compare
[edit interfaces interface-range vlan-private1-c-codfw]
- member ge-1/0/1;
[edit interfaces interface-range disabled]
member ge-1/0/0 { ... }
+ member ge-1/0/1;
[edit interfaces]
- ge-1/0/1 {
- description es2015;
- }Papaul updated the task description for T264261: decommission es2011.codfw.wmnet.
Papaul added a comment to T264261: decommission es2011.codfw.wmnet.
papaul@asw-b-codfw# show | compare
[edit interfaces interface-range vlan-private1-b-codfw]
- member ge-1/0/9;
[edit interfaces interface-range disabled]
member ge-6/0/19 { ... }
+ member ge-1/0/9;
[edit interfaces]
- ge-1/0/9 {
- description es2011;
- enable;
- }Papaul added a comment to T265208: Degraded RAID on ms-be2036.
@fgiunchedi yes we do have 3 spares. Will replace when back on site tomorrow.
Tue, Oct 13
Tue, Oct 13
Papaul added a comment to T264698: Degraded RAID on mw2279.
Create Dispatch: Success
You have successfully submitted request SR1039679642.
Papaul added a comment to T264698: Degraded RAID on mw2279.
@jijiki I will request a disk replacement
Papaul added a comment to T265208: Degraded RAID on ms-be2036.
@fgiunchedi looks like icinga is happy now
Papaul added a comment to T265208: Degraded RAID on ms-be2036.
The Flash/SD-CARD problem was fixed by formatting the NAND and draining the power
Papaul added a comment to T263837: Degraded RAID on es2026.
return tracking information
Papaul added a comment to T265208: Degraded RAID on ms-be2036.
Upgrade ILO from 2.50 to 2.74
Papaul added a comment to T265208: Degraded RAID on ms-be2036.
Embedded Flash/SD-CARD Controller firmware revision 2.10.00 Embedded media manager failed media attach
Papaul added a comment to T260670: db2125 crashed - mgmt iface also not available.
Both CPU replaced, servers is back up
Papaul added a comment to T263837: Degraded RAID on es2026.
pt1979@es2026:~$ sudo megacli -PDRbld -ShowProg -physdrv[32:2] -aALL
Papaul added a comment to T263837: Degraded RAID on es2026.
new disk in place
Status Name State Slot Number Size Security Status Bus Protocol Media Type Hot Spare Remaining Rated Write Endurance Physical Disk 0:1:0 Online 0 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:1 Online 1 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:2 Online 2 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:3 Online 3 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:4 Online 4 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:5 Online 5 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:6 Online 6 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:7 Online 7 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:8 Online 8 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:9 Online 9 1862.5 GB Not Capable SATA HDD No Not Applicable
Papaul added a comment to T263837: Degraded RAID on es2026.
Will do that once on site
Papaul added a comment to T263837: Degraded RAID on es2026.
@Marostegui since the server is under warranty, it is best to use a disk that is under warranty as well.
Mon, Oct 12
Mon, Oct 12
Papaul moved T265208: Degraded RAID on ms-be2036 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Papaul moved T264386: decommission es2017.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Papaul moved T264156: decommission es2016.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Papaul moved T264261: decommission es2011.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Papaul moved T264700: decommission es2015.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Papaul moved T264698: Degraded RAID on mw2279 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Sep 30
Wed, Sep 30
Papaul moved T263837: Degraded RAID on es2026 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
complete
Papaul closed T263740: decommission es2013.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Papaul updated the task description for T263740: decommission es2013.codfw.wmnet.
Papaul closed T264063: decommission es2019.codfw.wmnet, a subtask of T261717: Productionize es20[26-34] and es10[26-34], as Resolved.
Papaul added a comment to T264063: decommission es2019.codfw.wmnet.
complete
Papaul updated the task description for T264063: decommission es2019.codfw.wmnet.
Papaul updated the task description for T264063: decommission es2019.codfw.wmnet.
Papaul added a comment to T264063: decommission es2019.codfw.wmnet.
papaul@asw-d-codfw# show | compare
[edit interfaces interface-range vlan-private1-d-codfw]
- member ge-6/0/19;
[edit interfaces interface-range disabled]
member ge-1/0/4 { ... }
+ member ge-6/0/19;
[edit interfaces]
- ge-6/0/19 {
- description es2019;
- enable;
- }Papaul added a comment to T263837: Degraded RAID on es2026.
Return tracking information
Papaul added a comment to T263837: Degraded RAID on es2026.
I create another dispatch to request a new disk and shipped the one received on 9/25/2020 back.
Papaul added a comment to T263837: Degraded RAID on es2026.
Status Name State Slot Number Size Security Status Bus Protocol Media Type Hot Spare Remaining Rated Write Endurance Physical Disk 0:1:0 Online 0 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:1 Online 1 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:2 Online 2 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:3 Online 3 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:4 Online 4 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:5 Online 5 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:6 Online 6 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:7 Online 7 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:8 Online 8 1862.5 GB Not Capable SATA HDD No Not Applicable Physical Disk 0:1:9 Online 9 1862.5 GB Not Capable SATA HDD No Not Applicable
Papaul added a comment to T263837: Degraded RAID on es2026.
The disk from one of the decom es server works
Tue, Sep 29
Tue, Sep 29
Papaul added a comment to T260670: db2125 crashed - mgmt iface also not available.
2020-09-29 20:47:59 287 SYS1003 System CPU Resetting. 2020-09-29 20:47:51 286 PWR2270 The Intel Management Engine has encountered a Health Event. 2020-09-29 20:47:47 285 SYS1000 System is turning on. 2020-09-29 20:47:38 284 SYS1001 System is turning off. 2020-09-29 20:47:38 283 SYS1003 System CPU Resetting. 2020-09-29 20:47:21 282 RAC0703 Requested system hardreset. 2020-09-29 20:47:12 281 **``//CPU0000//``** Internal error has occurred check for additional logs.
Papaul updated the task description for T263740: decommission es2013.codfw.wmnet.
Papaul updated the task description for T263740: decommission es2013.codfw.wmnet.
Papaul added a comment to T263740: decommission es2013.codfw.wmnet.
papaul@asw-d-codfw# show | compare
[edit interfaces interface-range vlan-private1-d-codfw]
- member ge-1/0/4;
[edit interfaces interface-range disabled]
member xe-7/0/3 { ... }
+ member ge-1/0/4;
[edit interfaces]
- ge-1/0/4 {
- description es2013;
- enable;
- }Papaul placed T245192: Investigate Juniper storm control up for grabs.
Papaul added a comment to T263714: elastic2037 DIMM errors logged in racadm getsel.
return information
Sep 28 2020
Sep 28 2020
DiMM B1 replaced. All good now.
complete