Papaul (Papaul)
User

Projects (6)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (160 w, 6 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Today

Papaul reassigned T184787: Degraded RAID on ms-be2023 from Papaul to fgiunchedi.

Disk replacement complete.

Wed, Jan 17, 7:39 PM · Operations, ops-codfw

Yesterday

Papaul added a comment to T184788: mw2140 unresponsive, mgmt not accessible.

Hi Papaul

Tue, Jan 16, 4:55 PM · ops-codfw, Operations
Papaul added a comment to T184788: mw2140 unresponsive, mgmt not accessible.

Did the same process like in T166328 . The IDRAC is back up for now. I also spoke with Dell, there will be sending me a new main board.

Tue, Jan 16, 4:45 PM · ops-codfw, Operations
Papaul reassigned T169501: Move some masters away from B6 from Papaul to Marostegui.
Tue, Jan 16, 3:59 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T169501: Move some masters away from B6.

@Marostegui Assigning this back to you . Just assign me the decommissioned task when ready.

Tue, Jan 16, 3:59 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T184787: Degraded RAID on ms-be2023.

Dear Mr Papaul Tshibamba,

Tue, Jan 16, 3:53 PM · Operations, ops-codfw
Papaul reassigned T184836: Degraded RAID on db2036 from Papaul to Marostegui.

@Marostegui Disk replacement complete.

Tue, Jan 16, 3:31 PM · DBA, Operations, ops-codfw

Mon, Jan 15

Papaul triaged T184788: mw2140 unresponsive, mgmt not accessible as Normal priority.
Mon, Jan 15, 6:36 AM · ops-codfw, Operations
Papaul triaged T184787: Degraded RAID on ms-be2023 as Normal priority.
Mon, Jan 15, 6:34 AM · Operations, ops-codfw
Papaul added a comment to T184888: Replace codfw x1 master (db2033) (WAS: Failed BBU on db2033 (x1 master)).

@Marostegui sorry but we don't have any used BBU from a decommissioned host that we can use . (we have no decommissioned HP servers)

Mon, Jan 15, 6:29 AM · DBA

Wed, Jan 10

Papaul reassigned T184464: Degraded RAID on db2060 from Papaul to Marostegui.

Disk replacement complete

Wed, Jan 10, 4:58 PM · DBA, Operations, ops-codfw

Mon, Jan 8

Papaul added a comment to T184464: Degraded RAID on db2060.

Dear Mr Papaul Tshibamba,

Mon, Jan 8, 7:59 PM · DBA, Operations, ops-codfw
Papaul reassigned T184285: Degraded RAID on db2055 from Papaul to Marostegui.

Disk replacement complete.

Mon, Jan 8, 6:58 PM · DBA, Operations, ops-codfw

Fri, Jan 5

Papaul lowered the priority of T184285: Degraded RAID on db2055 from High to Normal.
Fri, Jan 5, 5:59 PM · DBA, Operations, ops-codfw
Papaul added a comment to T184285: Degraded RAID on db2055.

Dear Mr Papaul Tshibamba,

Fri, Jan 5, 5:59 PM · DBA, Operations, ops-codfw

Thu, Jan 4

Papaul reassigned T183887: db2054: Disk with predictive failure from Papaul to Marostegui.

Disk replacement complete

Thu, Jan 4, 7:26 PM · Operations, ops-codfw, DBA
Papaul reassigned T183167: Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabric from Papaul to RobH.
Thu, Jan 4, 4:46 PM · ops-codfw, Cloud-VPS, Operations
Papaul added a comment to T183167: Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabric.

@RobH
Row B rack B1
labtestvirt2002 ge-1/0/12

Thu, Jan 4, 4:46 PM · ops-codfw, Cloud-VPS, Operations
Papaul added a comment to T183167: Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabric.

This was not on my dashboard so I missed it. I will get on it when back at the DC tomorrow.

Thu, Jan 4, 1:00 AM · ops-codfw, Cloud-VPS, Operations

Wed, Jan 3

Papaul added a comment to T183887: db2054: Disk with predictive failure.

Dear Mr Papaul Tshibamba,

Wed, Jan 3, 10:04 PM · Operations, ops-codfw, DBA
Papaul added a comment to T183750: pc2005 crashed: CPU2 internal error.

@Marostegui that works for me

Wed, Jan 3, 6:16 PM · Patch-For-Review, Operations, ops-codfw, DBA
Papaul added a comment to T183750: pc2005 crashed: CPU2 internal error.

Moved CPU1 to CPU2
Upgrade IDRAC from version 2.21 to 2.50
Upgrade BIOS from version 2.1.7 to 2.6.0

Wed, Jan 3, 6:00 PM · Patch-For-Review, Operations, ops-codfw, DBA
Papaul reassigned T181263: mw2251 failed memory dimm from Papaul to elukey.

Memory replacement complete
Upgrade IDRAC from version 2.41 to 2.50
Upgrade BIOS from version 2.3.4 to 2.6.0

Wed, Jan 3, 4:57 PM · Operations, ops-codfw

Tue, Jan 2

Papaul added a comment to T181263: mw2251 failed memory dimm.

Dear Tshibamba, Papaul,

Tue, Jan 2, 11:13 PM · Operations, ops-codfw
Papaul added a comment to T183887: db2054: Disk with predictive failure.

Dear Mr Papaul Tshibamba,

Tue, Jan 2, 3:43 PM · Operations, ops-codfw, DBA

Dec 15 2017

Papaul closed T181725: Disconnect furud's disk shelves as Resolved.

complete

Dec 15 2017, 3:24 PM · Operations, ops-codfw

Dec 13 2017

Papaul updated the task description for T182397: Decomission eventlog2001.
Dec 13 2017, 5:30 PM · DC-Ops, ops-codfw, Analytics, Operations
Papaul updated the task description for T182397: Decomission eventlog2001.
Dec 13 2017, 4:03 PM · DC-Ops, ops-codfw, Analytics, Operations

Dec 5 2017

Papaul reassigned T181779: Degraded RAID on db2044 from Papaul to Marostegui.

Disk replacement complete

Dec 5 2017, 7:46 PM · DBA, Operations, ops-codfw

Dec 4 2017

Papaul added a comment to T181779: Degraded RAID on db2044.

Dear Mr Papaul Tshibamba,

Dec 4 2017, 4:18 PM · DBA, Operations, ops-codfw

Nov 28 2017

Papaul reassigned T181412: HP RAID Battery issue on elastic2004 from Papaul to Gehel.

Battery replacement complete
Firmware update complete

Nov 28 2017, 8:38 PM · Discovery-Search (Current work), Elasticsearch, Operations, ops-codfw, Discovery
Papaul reassigned T181266: Degraded RAID on db2055 from Papaul to Marostegui.

@Marostegui it looks like UPS give me wrong timing i got the park. Disk replacement complete

Nov 28 2017, 7:50 PM · DBA, Operations, ops-codfw
Papaul added a comment to T181412: HP RAID Battery issue on elastic2004.

@Gehel UPS just give me wrong timing> I just got email confirmation that i got the package on site.

Nov 28 2017, 7:36 PM · Discovery-Search (Current work), Elasticsearch, Operations, ops-codfw, Discovery
Papaul added a comment to T181266: Degraded RAID on db2055.

@Marostegui Tracking information shows 10:30am CT as delivering time and it is almost 2pm. I contact UPS they let me know that due to the pass holidays the package will not be delivered until 7pm. We will have to postpone the replacement for tomorrow.

Nov 28 2017, 7:29 PM · DBA, Operations, ops-codfw
Papaul added a comment to T181412: HP RAID Battery issue on elastic2004.

@Gehel Tracking information shows 10:30am CT as delivering time and it is almost 2pm. I contact UPS they let me know that due to the pass holidays the package will not be delivered until 7pm. We will have to postpone the replacement for tomorrow.

Nov 28 2017, 7:29 PM · Discovery-Search (Current work), Elasticsearch, Operations, ops-codfw, Discovery

Nov 27 2017

Papaul added a comment to T181412: HP RAID Battery issue on elastic2004.

I will because i have to update all the firmware too on the server. So will let you know tomorrow once I received the part

Nov 27 2017, 5:27 PM · Discovery-Search (Current work), Elasticsearch, Operations, ops-codfw, Discovery
Papaul added a comment to T181412: HP RAID Battery issue on elastic2004.

@Gehel I will be receiving the part tomorrow

Nov 27 2017, 5:24 PM · Discovery-Search (Current work), Elasticsearch, Operations, ops-codfw, Discovery
Papaul added a comment to T181266: Degraded RAID on db2055.

I am will be receiving the replacement drive tomorrow.

Nov 27 2017, 3:24 PM · DBA, Operations, ops-codfw
Papaul added a comment to T181266: Degraded RAID on db2055.

Dear Mr Papaul Tshibamba,

Nov 27 2017, 3:20 PM · DBA, Operations, ops-codfw

Nov 22 2017

Papaul added a comment to T180211: Degraded RAID on wtp2017.

@akosiaris you welcome

Nov 22 2017, 5:09 PM · Operations, ops-codfw

Nov 21 2017

Papaul reassigned T180927: db2068 storage crash from Papaul to Marostegui.

Firmware update complete

Nov 21 2017, 4:53 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul reassigned T180211: Degraded RAID on wtp2017 from Papaul to akosiaris.

Call Dell, they said that the server part warranty has expired so i have some spare 500G SATA disks on site that i used for the system.

Nov 21 2017, 4:47 PM · Operations, ops-codfw

Nov 20 2017

Papaul added a comment to T180211: Degraded RAID on wtp2017.

@akosiaris thank you.

Nov 20 2017, 5:54 PM · Operations, ops-codfw
Papaul added a comment to T180927: db2068 storage crash.

The ILO is up to date. I need to update the Storage and BIOS on the system but the Service pack disk that i have is old, there is a new Service pack ISO on the HP site that I need to download and the file is about 6.57GB and i am on MIFI. I will download the file once home and update the system tomorrow.
http://h17007.www1.hpe.com/us/en/enterprise/servers/products/service_pack/spp/index.aspx?version=2017.04.0

Nov 20 2017, 4:56 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul updated subscribers of T180211: Degraded RAID on wtp2017.

@Joe or @akosiaris since this systems is using software raid there is nothing showing in hardware log which disk is bad and the system diagnostic came up with no error. Can you please pull up at the OS level a log that can help me determinate which disk is bad so i can submit it to Dell? Or is this another bug (false alert)

Nov 20 2017, 4:02 PM · Operations, ops-codfw

Nov 17 2017

Papaul added a comment to T180724: mw2251 hardware error.

Full memory test came out with no errors. I went ahead and update the IDRAC firmware as well. The system is back up online.

Nov 17 2017, 12:16 AM · ops-codfw, Operations

Nov 16 2017

Papaul added a comment to T180724: mw2251 hardware error.

Step 1;
login to the IDRAC to check log files, log file is showing some memory error on DIMM_A1 " Correctable memory error rate exceeded for DIMM_A1"

Nov 16 2017, 9:44 PM · ops-codfw, Operations

Nov 13 2017

Papaul closed T180200: Broken memory on mw2108 as Resolved.

Memory test came up with no errors. IF this happen again we will have to purchase a 8GB DDR3 because I have no 8GB memory on site and the system is out of warranty. System is back up running for now.

Nov 13 2017, 4:55 PM · Operations, ops-codfw
Papaul closed T177638: check mw2160 power supply redundancy as Resolved.

Resolving this since we didn't have any more issue for a week.

Nov 13 2017, 3:23 PM · ops-codfw, Operations
Papaul closed T177639: check mw2176 power supply redundancy as Resolved.

Resolving this since we didn't have any more issue for a week.

Nov 13 2017, 3:22 PM · ops-codfw, Operations

Nov 10 2017

Papaul triaged T180200: Broken memory on mw2108 as Normal priority.
Nov 10 2017, 4:23 PM · Operations, ops-codfw
Papaul added a comment to T180200: Broken memory on mw2108.

Thank you will work on it on Monday.

Nov 10 2017, 4:22 PM · Operations, ops-codfw

Nov 6 2017

Papaul added a comment to T177638: check mw2160 power supply redundancy .

same fix as in T177639

Nov 6 2017, 4:53 PM · ops-codfw, Operations
Papaul added a comment to T177639: check mw2176 power supply redundancy.
  • Swapped PS1 and PS2
  • Upgrade firmware
Nov 6 2017, 4:49 PM · ops-codfw, Operations
Papaul updated subscribers of T177638: check mw2160 power supply redundancy .

@MoritzAccountTest same as mw2176
The system is showing that PS1 Present/failure. I need the system put in maintenance mode for me to troubleshoot.

Nov 6 2017, 3:58 PM · ops-codfw, Operations
Papaul updated subscribers of T177639: check mw2176 power supply redundancy.

@MoritzAccountTest
The system is showing that PS1 Present/failure. I need the system put in maintenance mode for me to troubleshoot.

Nov 6 2017, 3:57 PM · ops-codfw, Operations

Oct 30 2017

Papaul closed T178613: add missing asset tag and correct location in rack for cr1-eqdfw as Resolved.

complete

Oct 30 2017, 3:44 PM · Operations, DC-Ops, ops-eqdfw

Oct 16 2017

Papaul added a comment to T178140: db2081 unreachable.

@Marostegui when you call Dell the first thing they will tell you is to update the firmware that is the reason i didn't call them and i went ahead and update all the firmwares and left the task open to monitor the system. In case this happen again then I will get in touch with Dell.

Oct 16 2017, 3:35 PM · DBA, ops-codfw, Operations
Papaul added a comment to T178140: db2081 unreachable.

Please see attachment for the error that was on the screen

Oct 16 2017, 3:22 PM · DBA, ops-codfw, Operations
Papaul triaged T178140: db2081 unreachable as Normal priority.
Oct 16 2017, 2:26 PM · DBA, ops-codfw, Operations

Oct 10 2017

Papaul reassigned T175685: Decommission db2010 and move m1 codfw to db2078 from Papaul to RobH.
Oct 10 2017, 3:55 PM · hardware-requests, ops-codfw, Operations, DBA
Papaul updated the task description for T175685: Decommission db2010 and move m1 codfw to db2078.
Oct 10 2017, 3:50 PM · hardware-requests, ops-codfw, Operations, DBA
Papaul added a comment to T177720: db2038 two disks with predictive failure.

done

Oct 10 2017, 3:21 PM · Operations, ops-codfw, DBA
Papaul added a comment to T177720: db2038 two disks with predictive failure.

ok

Oct 10 2017, 3:18 PM · Operations, ops-codfw, DBA
Papaul added a comment to T177720: db2038 two disks with predictive failure.

Complete. Let me know when ready for slot 7

Oct 10 2017, 3:04 PM · Operations, ops-codfw, DBA
Papaul added a comment to T177720: db2038 two disks with predictive failure.

ok I i will replaced first slot 1

Oct 10 2017, 3:00 PM · Operations, ops-codfw, DBA
Papaul added a comment to T177720: db2038 two disks with predictive failure.

@Marostegui I have some 600GB 15k that I can pull out off db2025. Just keep in mind that those are Dell disks

Oct 10 2017, 2:55 PM · Operations, ops-codfw, DBA

Oct 9 2017

Papaul added a comment to T177720: db2038 two disks with predictive failure.

@Marostegui this server is out of warranty 2017-07-10. We need to find out if any of the decommissioned servers have the same disks that we can use.

Oct 9 2017, 7:45 AM · Operations, ops-codfw, DBA

Oct 5 2017

Papaul added a comment to T175685: Decommission db2010 and move m1 codfw to db2078.

switch port information

Oct 5 2017, 3:40 PM · hardware-requests, ops-codfw, Operations, DBA
Papaul added a comment to T175685: Decommission db2010 and move m1 codfw to db2078.

Disk wipe in progress .

Oct 5 2017, 3:02 PM · hardware-requests, ops-codfw, Operations, DBA

Oct 2 2017

Papaul reassigned T174764: db2044 HW RAID failure from Papaul to Marostegui.

Main board replacement complete.

Oct 2 2017, 7:04 PM · Patch-For-Review, ops-codfw, Operations, DBA

Sep 28 2017

Papaul added a comment to T174764: db2044 HW RAID failure.

@Marostegui yes I will make sure PXE is disable.

Sep 28 2017, 3:04 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul added a comment to T174764: db2044 HW RAID failure.

After reviewing all the logs , HP decide to send a new mainboard because the controller is on the mainboard. The tech will be onsite to perform the replacement on Monday 2nd between 9am and 1 pm.

Sep 28 2017, 2:54 PM · Patch-For-Review, ops-codfw, Operations, DBA

Sep 27 2017

Papaul added a comment to T174764: db2044 HW RAID failure.

Dear Mr Papaul Tshibamba,

Sep 27 2017, 2:48 PM · Patch-For-Review, ops-codfw, Operations, DBA

Sep 26 2017

Papaul added a comment to T176506: rack/setup/install furud.codfw.wmnet.

The Power storage MD1400 doesn't have an entry in Racktables (HW type). Please add that in racktables.

Sep 26 2017, 7:39 PM · Analytics, Operations
Papaul reassigned T176506: rack/setup/install furud.codfw.wmnet from Papaul to RobH.

switch port information

Sep 26 2017, 4:36 PM · Analytics, Operations
Papaul updated the task description for T176506: rack/setup/install furud.codfw.wmnet.
Sep 26 2017, 4:27 PM · Analytics, Operations

Sep 25 2017

Papaul reassigned T176427: unrack/decom pfw1-codfw and pfw2-codfw from Papaul to RobH.

complete

Sep 25 2017, 3:39 PM · hardware-requests, ops-codfw, fundraising-tech-ops, netops, Operations
Papaul updated the task description for T176427: unrack/decom pfw1-codfw and pfw2-codfw.
Sep 25 2017, 3:38 PM · hardware-requests, ops-codfw, fundraising-tech-ops, netops, Operations
Papaul added a comment to T176427: unrack/decom pfw1-codfw and pfw2-codfw.

pfw1-codfw port 33
pfw2-codfw port 34

Sep 25 2017, 3:18 PM · hardware-requests, ops-codfw, fundraising-tech-ops, netops, Operations
Papaul updated the task description for T176427: unrack/decom pfw1-codfw and pfw2-codfw.
Sep 25 2017, 3:18 PM · hardware-requests, ops-codfw, fundraising-tech-ops, netops, Operations
Papaul added a comment to T176573: db2047 got rebooted.

The temperature in C6 is about 106 F and the ILO log shows Critical Temperature Threshold Exceeded on 9-24 at 11:30 pm (see attachment)

Sep 25 2017, 3:09 PM · ops-codfw, Patch-For-Review, DBA, Operations

Sep 19 2017

Papaul reassigned T176175: connect second ethernet interface for fundraising codfw hosts from Papaul to Jgreen.

@Jgreen Complete

Sep 19 2017, 3:26 PM · ops-codfw, netops, fundraising-tech-ops, Operations

Sep 13 2017

Papaul added a comment to T175685: Decommission db2010 and move m1 codfw to db2078.

@jcrespo this is assigned to me is there anything i have to do on my side?

Sep 13 2017, 2:16 PM · hardware-requests, ops-codfw, Operations, DBA
Papaul reassigned T163346: mw2256 - hardware issue from Papaul to elukey.

@elukey I think we can put the system back in production to test it out after the Dell Engineer replaced all the parts above.

Sep 13 2017, 2:14 PM · Operations, ops-codfw

Sep 12 2017

Papaul added a comment to T163346: mw2256 - hardware issue.

Tech arrived on site at 10:13am and started working on the server at 10:30.

Sep 12 2017, 7:05 PM · Operations, ops-codfw
Papaul added a comment to T175228: Degraded RAID on db2010.

Disk in slot 2 replacement complete.

Sep 12 2017, 5:08 PM · DBA, Operations, ops-codfw
Papaul added a comment to T175228: Degraded RAID on db2010.

Disk replacement in slot 0 complete

Sep 12 2017, 2:49 PM · DBA, Operations, ops-codfw

Sep 11 2017

Papaul added a comment to T175228: Degraded RAID on db2010.

Ok I have disk in slot 5 replaced

Sep 11 2017, 2:58 PM · DBA, Operations, ops-codfw
Papaul triaged T175228: Degraded RAID on db2010 as High priority.
Sep 11 2017, 2:56 PM · DBA, Operations, ops-codfw
Papaul closed T174534: Degraded RAID on ms-be2024 as Resolved.

This servers looks good so far so resolving the task for now.

Sep 11 2017, 2:55 PM · media-storage, Operations, ops-codfw
Papaul closed T174777: Degraded RAID on ms-be2023 as Resolved.

This package shows delivered so I am resolving this task

Sep 11 2017, 2:54 PM · Operations, ops-codfw
Papaul added a comment to T175228: Degraded RAID on db2010.

@jcrespo on db2010 I have 5 bad disks is there any particular order you will want them replaced?

Sep 11 2017, 2:51 PM · DBA, Operations, ops-codfw
Papaul claimed T163346: mw2256 - hardware issue.
Sep 11 2017, 2:47 PM · Operations, ops-codfw

Sep 8 2017

Papaul added a comment to T163346: mw2256 - hardware issue.

Hi Papaul,

Sep 8 2017, 5:02 PM · Operations, ops-codfw

Sep 7 2017

Papaul added a comment to T163346: mw2256 - hardware issue.

@elukey got a call from the Dell manager support team here is what going to happen for the next step.
They will send out :

Sep 7 2017, 6:19 PM · Operations, ops-codfw
Papaul added a comment to T175228: Degraded RAID on db2010.

@jcrespo db200[1-9} all have 12x300Gb disks we can pull one out and use it for db2010 for now.

Sep 7 2017, 5:15 PM · DBA, Operations, ops-codfw
Papaul added a comment to T175228: Degraded RAID on db2010.

@jcrespo technically we do not have any 300GB spare disks. I am trying to load my Google spreadsheet for server decommission to see if we we do have a server with 300GB but can't for the moment once i am able i will update task.

Sep 7 2017, 3:05 PM · DBA, Operations, ops-codfw

Sep 6 2017

Papaul reassigned T174777: Degraded RAID on ms-be2023 from Papaul to fgiunchedi.

@fgiunchedi Disk replacement complete. Below please see information for return package.

Sep 6 2017, 5:16 PM · Operations, ops-codfw
Papaul added a comment to T163346: mw2256 - hardware issue.

In the process of updating the firmware on the server, the server got again in a frozen state. nothing on the monitor and no keyboard response as well.

Sep 6 2017, 4:12 PM · Operations, ops-codfw