Page MenuHomePhabricator

Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (243 w, 4 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Yesterday

Papaul closed T230597: can't SSH to elastic2050.mgmt as Resolved.

Upgrade firmware as well
Before
BIOS Version 1.5.6
iDRAC Firmware Version 3.21.21.21

Mon, Aug 19, 3:05 PM · ops-codfw, DC-Ops, Discovery-Search (Current work), Operations

Fri, Aug 16

Papaul added a comment to T230597: can't SSH to elastic2050.mgmt .

@Mathew.onipe any reason why this is set to high priority ?

Fri, Aug 16, 3:46 PM · ops-codfw, DC-Ops, Discovery-Search (Current work), Operations
Papaul moved T221068: decom ms-be201[345] from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:44 PM · decommission, ops-codfw, media-storage, User-fgiunchedi, Operations
Papaul moved T228281: decommission db2045.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:43 PM · Operations, ops-codfw, DC-Ops, decommission
Papaul moved T227565: decommission db2038 from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:43 PM · Operations, ops-codfw, DC-Ops, decommission
Papaul moved T229543: Decommission db2058.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:43 PM · DC-Ops, ops-codfw, decommission, Operations
Papaul moved T229784: Decommission db2035 from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:42 PM · ops-codfw, DC-Ops, Operations, decommission
Papaul moved T230107: Decommission db2069.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:42 PM · DC-Ops, ops-codfw, decommission, Operations
Papaul moved T230311: Decommission db2043.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:42 PM · Patch-For-Review, ops-codfw, decommission, Operations
Papaul moved T230391: Decommission db2050.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:42 PM · DC-Ops, ops-codfw, decommission, Operations
Papaul moved T230394: Decommission db2057.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Fri, Aug 16, 3:42 PM · DC-Ops, ops-codfw, decommission, Operations
Papaul closed T112774: solve mtp panel issue for row uplinks as Resolved.

Resolving this task since we will not be using and we are not using the patch panels. This will not be setup anymore

Fri, Aug 16, 3:41 PM · DC-Ops, Operations, ops-codfw

Thu, Aug 15

Papaul added a comment to T230459: Replace db2044 with db2063.

ssh issue

papaul@papaulpc:~$ ssh root@db2063.mgmt.codfw.wmnet
Unable to negotiate with UNKNOWN port 65535: no matching cipher found. Their offer: aes256-cbc,aes128-cbc,3des-cbc

I have to have run ssh command with -c aes256-cbc to access mgmt

papaul@papaulpc:~$ ssh -c aes256-cbc root@db2063.mgmt.codfw.wmnet
root@db2063.mgmt.codfw.wmnet's password:

After the upgrade and resting the ILO I was able to access the mgmt without the -c aes256-cbc

Thu, Aug 15, 3:30 PM · DBA

Tue, Aug 13

Papaul reassigned T230275: Degraded RAID on ms-be2021 from Papaul to fgiunchedi.

@fgiunchedi Disk replaced.

Tue, Aug 13, 3:35 PM · media-storage, Operations, ops-codfw
Papaul triaged T230275: Degraded RAID on ms-be2021 as Normal priority.
Tue, Aug 13, 3:29 PM · media-storage, Operations, ops-codfw
Papaul closed T221572: (OoW) wtp2019 shows error messages in the racadm getsel's output as Resolved.

I checked the server this morning no errors showing in log. closing the task

Tue, Aug 13, 3:28 PM · ops-codfw, Operations
Papaul added a comment to T227298: elastic2054 unresponsive.

Return information

Tue, Aug 13, 3:25 PM · Discovery, Operations
Papaul closed T227298: elastic2054 unresponsive as Resolved.

DIMM A2 replaced and log cleared . Closing this task for now .

Tue, Aug 13, 3:20 PM · Discovery, Operations

Fri, Aug 9

Papaul added a comment to T227298: elastic2054 unresponsive.

You have successfully submitted request SR995910217.

Fri, Aug 9, 9:39 PM · Discovery, Operations

Thu, Aug 8

Papaul added a comment to T226422: update RE-S-X6-64G-S in cr[12]-codfw.

Return information for bad SCB

Thu, Aug 8, 4:13 PM · netops, ops-codfw, Operations

Wed, Aug 7

Papaul reopened T227298: elastic2054 unresponsive as "Open".
Wed, Aug 7, 11:51 PM · Discovery, Operations
Papaul added a comment to T227298: elastic2054 unresponsive.

My last comment on July 8 was "I swapped B2 with A2, no more error. leaving this task open for a week. If we do have the same problem on A2, I will request a replacement." It looks like we do have the error on A2 now . I will request a DIMM replacement

Wed, Aug 7, 11:50 PM · Discovery, Operations

Tue, Aug 6

Papaul added a comment to T221572: (OoW) wtp2019 shows error messages in the racadm getsel's output.

@ssastry upgrade complete, I have no more errors showing in the IDRAC log, I am leaving the task open until next week then will resolve it if no errors.

Tue, Aug 6, 8:14 PM · ops-codfw, Operations
Papaul added a comment to T221572: (OoW) wtp2019 shows error messages in the racadm getsel's output.

Instructions: The System Event Log contains information about the managed system. To sort the log by column, click a column header.

Clear Log
Save As

	 	Tue Apr 23 2019 05:44:33	The system board PS1 PG Fail voltage is within range.	
	 	Tue Apr 23 2019 05:44:28	The system board PS1 PG Fail voltage is outside of range.	
	 	Mon Apr 22 2019 00:48:36	Power supply 1 is operating normally.	
	 	Mon Apr 22 2019 00:48:36	Power supply 2 is operating normally.	
	 	Mon Apr 22 2019 00:48:34	An over current fault detected on power supply 2.	
	 	Mon Apr 22 2019 00:48:34	An over current fault detected on power supply 1.	
	 	Mon Apr 22 2019 00:48:31	CPU 2 has an internal error (IERR).	
	 	Wed Oct 26 2016 15:58:55	Log cleared.
Tue, Aug 6, 5:59 PM · ops-codfw, Operations
Papaul reassigned T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet from Papaul to Marostegui.

Fixed

Tue, Aug 6, 3:00 PM · ops-codfw, Operations, DBA
Papaul added a comment to T225121: (Need By: Sept 30) upgrade msw1-eqiad from EX4200 to EX4300.

@Cmjohnson I put together a "How to" at the link below on how to upgrade the switch. Please let me know if you have any questions.

Tue, Aug 6, 5:37 AM · netops, Operations, ops-eqiad
Papaul closed T194171: (OoW) rdb2002 correctable memory errors as Resolved.

This server is decommissioned so we can close this task. T209425

Tue, Aug 6, 12:09 AM · Operations, ops-codfw

Mon, Aug 5

Papaul reassigned T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet from Papaul to Marostegui.

@Marostegui All yours

Mon, Aug 5, 11:45 PM · ops-codfw, Operations, DBA
Papaul updated the task description for T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet.
Mon, Aug 5, 11:43 PM · ops-codfw, Operations, DBA
Papaul updated the task description for T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet.
Mon, Aug 5, 11:19 PM · ops-codfw, Operations, DBA
Papaul updated the task description for T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet.
Mon, Aug 5, 11:03 PM · ops-codfw, Operations, DBA
Papaul added a comment to T227548: SSH to mw2269.mgmt not working.

@jijiki can we close this?

Mon, Aug 5, 10:36 PM · Operations, ops-codfw
Papaul closed T200678: (OoW) wtp2011 memory correctable errors as Resolved.

BIOS and IDRAC are now up to date on the server no memory errors in the log. closing this task for now. We can reopen if we do see again any errors.
thanks @Arlolra thanks

Mon, Aug 5, 10:19 PM · Operations, ops-codfw
Papaul updated the task description for T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet.
Mon, Aug 5, 8:33 PM · ops-codfw, Operations, DBA
Papaul closed T229638: Decommission old mws1-codfw as Resolved.

This is complete

Mon, Aug 5, 5:10 PM · netops, ops-codfw, Operations
Papaul updated the task description for T229638: Decommission old mws1-codfw.
Mon, Aug 5, 5:10 PM · netops, ops-codfw, Operations
Papaul updated the task description for T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet.
Mon, Aug 5, 4:26 PM · ops-codfw, Operations, DBA
Papaul closed Unknown Object (Task), a subtask of T220170: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment, as Resolved.
Mon, Aug 5, 4:26 PM · Goal, DBA
Papaul added a comment to T227408: (OoW) restbase2009 lockup.

@jijiki I made a procurement task the the storage battery at T229847

Mon, Aug 5, 4:10 PM · serviceops, ops-codfw, Operations
Papaul added a comment to T227408: (OoW) restbase2009 lockup.

@jijiki please repool the server when you have a minute. We will have to order a new Storage battery for the server since all the decom HP servers are GEN8 and this one is a GEN9 so different storage battery.

Mon, Aug 5, 3:55 PM · serviceops, ops-codfw, Operations
Papaul added a comment to T227408: (OoW) restbase2009 lockup.

@jijiki i need this serveur power down
thanks

Mon, Aug 5, 2:10 PM · serviceops, ops-codfw, Operations

Fri, Aug 2

Papaul updated the task description for T229638: Decommission old mws1-codfw.
Fri, Aug 2, 1:26 AM · netops, ops-codfw, Operations
Papaul triaged T229638: Decommission old mws1-codfw as Normal priority.
Fri, Aug 2, 1:26 AM · netops, ops-codfw, Operations
Papaul closed T224250: Setup new msw1-codfw, a subtask of T225121: (Need By: Sept 30) upgrade msw1-eqiad from EX4200 to EX4300, as Resolved.
Fri, Aug 2, 1:25 AM · netops, Operations, ops-eqiad
Papaul closed T224250: Setup new msw1-codfw as Resolved.

Resolving this. The decom task for this T229638

Fri, Aug 2, 1:25 AM · ops-codfw, Operations, netops
Papaul updated the task description for T224250: Setup new msw1-codfw.
Fri, Aug 2, 1:24 AM · ops-codfw, Operations, netops
Papaul created T229638: Decommission old mws1-codfw.
Fri, Aug 2, 1:24 AM · netops, ops-codfw, Operations
Papaul updated the task description for T224250: Setup new msw1-codfw.
Fri, Aug 2, 12:58 AM · ops-codfw, Operations, netops
Papaul added a comment to T221984: scs-a1-codfw: update serial in netbox.

@RobH when I took the old ms1 down this morning i did have room to look at scs-a1 Serial # . I update Network with the needed information, Please feel free to resolve this task.

Fri, Aug 2, 12:27 AM · netbox, Operations, ops-codfw

Thu, Aug 1

Papaul added a comment to T227552: pc2010 possibly broken memory.

I checked IDRAC logs this morning, all looks good so far

Thu, Aug 1, 1:35 PM · Operations, ops-codfw, DBA
Papaul reassigned T227408: (OoW) restbase2009 lockup from Papaul to wiki_willy.

@jijiki I will talking to @wiki_willy to see what are our options on this.

Thu, Aug 1, 1:32 PM · serviceops, ops-codfw, Operations
Papaul closed T225090: Decommission db2042, a subtask of T221533: Decommission old coredb machines (<=db2042), as Resolved.
Thu, Aug 1, 1:21 PM · DBA
Papaul closed T225090: Decommission db2042 as Resolved.

Complete

Thu, Aug 1, 1:21 PM · Patch-For-Review, Operations, ops-codfw, decommission
Papaul updated the task description for T225090: Decommission db2042.
Thu, Aug 1, 1:20 PM · Patch-For-Review, Operations, ops-codfw, decommission

Wed, Jul 31

Papaul moved T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet from Backlog to Racking Tasks on the ops-codfw board.
Wed, Jul 31, 7:57 PM · ops-codfw, Operations, DBA
Papaul moved T229283: Degraded RAID on ms-be2021 from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Jul 31, 7:56 PM · Operations, ops-codfw
Papaul updated the task description for T225090: Decommission db2042.
Wed, Jul 31, 7:52 PM · Patch-For-Review, Operations, ops-codfw, decommission
Papaul updated the task description for T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet.
Wed, Jul 31, 4:37 PM · ops-codfw, Operations, DBA
Papaul reassigned T227552: pc2010 possibly broken memory from Papaul to Marostegui.

@Marostegui DIMM A1 replaced
log cleared

Wed, Jul 31, 4:34 PM · Operations, ops-codfw, DBA
Papaul reassigned T229302: Degraded RAID on db2063 from Papaul to Marostegui.

Disk replaced

Wed, Jul 31, 3:12 PM · DBA, Operations, ops-codfw
Papaul updated the task description for T226422: update RE-S-X6-64G-S in cr[12]-codfw.
Wed, Jul 31, 2:49 PM · netops, ops-codfw, Operations
Papaul added a comment to T147161: Installer suggests $wgMainCacheType = CACHE_ACCEL, potentially breaking sessions.

I am having the same problem. Everything was working fine on my Wiki until this morning when I tried to login i was getting the message below

There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking. Go back to the previous page, reload that page and then try again.
MediaWiki 1.30.0
Wed, Jul 31, 2:33 AM · Patch-For-Review, MediaWiki-Authentication-and-authorization, MediaWiki-Cache, MediaWiki-Installer

Tue, Jul 30

Papaul triaged T229283: Degraded RAID on ms-be2021 as Normal priority.
Tue, Jul 30, 7:55 PM · Operations, ops-codfw
Papaul added a comment to T227552: pc2010 possibly broken memory.

@Marostegui I was wrong on my comment about the main board. We replace DIMM B1 on July 22nd and the sever crashed again on the 24th. On the 29th i swapped B1 with A1 and on the 30th the server crashed again but the error now was on A1 no longer on B1 so the DIMM that DELL sent to us was bad not the main board so i requested another DIMM .

Tue, Jul 30, 5:53 PM · Operations, ops-codfw, DBA
Papaul added a comment to T227552: pc2010 possibly broken memory.

@Marostegui This system crashed again . This time the error is on DIMM A1 see below.

	"Correctable memory error logging disabled for a memory device at location DIMM_A1. 	Mon 29 Jul 2019 21:18:34"

I will open a case with Dell to request a main board replacement.

Tue, Jul 30, 1:45 PM · Operations, ops-codfw, DBA

Mon, Jul 29

Papaul reassigned T227113: rack/setup/install db21[21-30].codfw.wmnet from Papaul to Marostegui.

@Marostegui all yours

Mon, Jul 29, 3:35 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Mon, Jul 29, 3:35 PM · Goal, Operations, DBA, ops-codfw
Papaul added a comment to T227552: pc2010 possibly broken memory.

Swapped DIMM B1 with DIMM A1 to see if we have the same problem on DIMM A1 if we do, we will have to replace he main-board.
@Marostegui Please let me know it the system crash again .

Mon, Jul 29, 2:54 PM · Operations, ops-codfw, DBA

Thu, Jul 25

Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Thu, Jul 25, 12:30 AM · Goal, Operations, DBA, ops-codfw

Wed, Jul 24

Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 9:34 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 9:30 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 5:20 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 5:07 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 4:56 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 4:47 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 4:40 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Wed, Jul 24, 4:39 PM · Goal, Operations, DBA, ops-codfw

Tue, Jul 23

Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Tue, Jul 23, 11:49 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T225090: Decommission db2042.
Tue, Jul 23, 5:24 PM · Patch-For-Review, Operations, ops-codfw, decommission
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Tue, Jul 23, 5:16 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Tue, Jul 23, 4:53 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Tue, Jul 23, 4:29 PM · Goal, Operations, DBA, ops-codfw
Papaul closed T194174: (OoW) wtp2013 memory correctable errors as Resolved.
  • Replace DIMM B2
  • Clear log
  • Upgrae BIOS from 2.3 to 2.6

-Upgrade IDRAC from 1.57 to 2.61

Tue, Jul 23, 4:03 PM · Operations, ops-codfw

Mon, Jul 22

Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Mon, Jul 22, 6:24 PM · Goal, Operations, DBA, ops-codfw
Papaul updated the task description for T228112: Cable mr1-codfw<->cr1/2-codfw through asw-a-codfw.
Mon, Jul 22, 5:30 PM · Operations, netops, ops-codfw
Papaul updated the task description for T227113: rack/setup/install db21[21-30].codfw.wmnet.
Mon, Jul 22, 5:08 PM · Goal, Operations, DBA, ops-codfw
Papaul reassigned T227552: pc2010 possibly broken memory from Papaul to Marostegui.

DIMM replaced and Firmware upgrade

Mon, Jul 22, 4:12 PM · Operations, ops-codfw, DBA
Papaul closed Unknown Object (Task), a subtask of T220170: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment, as Resolved.
Mon, Jul 22, 4:06 PM · Goal, DBA

Jul 19 2019

Papaul added a comment to T227552: pc2010 possibly broken memory.

Create Dispatch: Success
You have successfully submitted request SR994795023.

Jul 19 2019, 3:46 PM · Operations, ops-codfw, DBA

Jul 18 2019

Papaul closed T225988: decommission db2039, a subtask of T221533: Decommission old coredb machines (<=db2042), as Resolved.
Jul 18 2019, 11:13 PM · DBA
Papaul closed T225988: decommission db2039 as Resolved.

This is complete

Jul 18 2019, 11:13 PM · Patch-For-Review, DC-Ops, ops-codfw, Operations, decommission
Papaul added a comment to T226422: update RE-S-X6-64G-S in cr[12]-codfw.

@ayounsi please pick a day and time sometimes next week (Mon - Wed) and let me know.

Jul 18 2019, 10:03 PM · netops, ops-codfw, Operations
Papaul added a comment to T228112: Cable mr1-codfw<->cr1/2-codfw through asw-a-codfw.

@ayounsi please pick a day and time sometimes next week (Mon - Wed) and let me know.

Jul 18 2019, 10:00 PM · Operations, netops, ops-codfw
Papaul added a comment to T194174: (OoW) wtp2013 memory correctable errors.

Log is showing a lot of "Correctable memory error rate exceeded for DIMM_B2". We will have to take the system down and swap the memory in DIMM B2 with a DIMM from one of the decom server onsite for now.

Jul 18 2019, 2:51 PM · Operations, ops-codfw
Papaul closed T205712: (OoW) wtp2020: correctable memory errors as Resolved.

No errors showing in log and all Hardware showing green, firmware is at version 2.6. we can resolve this task for now and reopen in case we have the issue again.

Jul 18 2019, 2:42 PM · Operations, ops-codfw
Papaul added a comment to T221572: (OoW) wtp2019 shows error messages in the racadm getsel's output.

This server needs to to offline for me to perform all the troubleshooting recommended by Dell.
https://www.dell.com/support/article/us/en/04/qna42558/dell-poweredge-vlt0204-system-board-psx-pg-fail-voltage-is-outside-of-range?lang=en

Jul 18 2019, 2:27 PM · ops-codfw, Operations
Papaul updated the task description for T225988: decommission db2039.
Jul 18 2019, 2:14 PM · Patch-For-Review, DC-Ops, ops-codfw, Operations, decommission

Jul 17 2019

Papaul added a comment to T200678: (OoW) wtp2011 memory correctable errors.

No memory errors showing on this system in the log . Upgrade IDRAC from 1.5 to 2.6 . We have a new BIOS version available we need to depool the server for the upgrade

Jul 17 2019, 8:46 PM · Operations, ops-codfw
Papaul moved T225630: ms-be2018 sdc unreadable sector from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Jul 17 2019, 8:28 PM · Operations, ops-codfw
Papaul moved T227408: (OoW) restbase2009 lockup from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Jul 17 2019, 8:27 PM · serviceops, ops-codfw, Operations