Page MenuHomePhabricator

Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (218 w, 1 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Thu, Feb 21

Papaul added a comment to T215193: Fix codfw x-connect 65373.

CyrusOne Checked the reading from the fiber patch panel in A8 same readings. So they are still going to run some test out of the cage.

Thu, Feb 21, 5:55 PM · Operations, netops
Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

@fgiunchedi is it possible to depool this server for me to do a firmware upgrade before I resolve the task?

Thu, Feb 21, 4:54 PM · ops-codfw, Operations
Papaul reassigned T216670: Degraded RAID on db2050 from Papaul to Marostegui.

disk replaced

Thu, Feb 21, 3:27 PM · DBA, Operations, ops-codfw

Tue, Feb 19

Papaul added a comment to T214813: Degraded RAID on thumbor2002.

The network cable was was plugged back in after the disk replacement. Should be good now.

Tue, Feb 19, 3:38 PM · serviceops, User-jijiki, Operations, ops-codfw
Papaul added a comment to T216240: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092.

db2089 upgrade complete
Upgrade
BIOS from 2.4.3 to 2.9.1
IDRAC from 2.40. to 2.61

Tue, Feb 19, 3:37 PM · Operations, ops-codfw, DBA
Papaul added a comment to T216240: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092.

Can db2089 be depool please if it is not yet? Thanks

Tue, Feb 19, 2:47 PM · Operations, ops-codfw, DBA

Thu, Feb 14

Papaul reassigned T214840: db2085/db1106 don't boot with 4.9.0-8-amd64 from Papaul to Marostegui.

Upgrade
BIOS from 2.4.3 to 2.9.1
IDRAC from 2.40. to 2.60

Thu, Feb 14, 3:10 PM · ops-codfw, Patch-For-Review, Operations, DBA
Papaul added a comment to T214840: db2085/db1106 don't boot with 4.9.0-8-amd64.

@Marostegui this can be done anytime today. Just let me know when the server is down. Thanks

Thu, Feb 14, 2:20 PM · ops-codfw, Patch-For-Review, Operations, DBA
Papaul added a comment to T214840: db2085/db1106 don't boot with 4.9.0-8-amd64.

@Marostegui in most cases the CPU1/CPU2 Machine check error detected is caused from outdated BIOS. I will recommend that we first update the BIOS. The system BIOS right now is at 2.4.3 and there is a new version out (2.9.1) from 11/02/2019.After this we can check some settings in the BIOS under BIOS profile .

Thu, Feb 14, 12:40 AM · ops-codfw, Patch-For-Review, Operations, DBA
Papaul closed T199247: Decommission baham as Resolved.

Complete

Thu, Feb 14, 12:17 AM · Patch-For-Review, decommission, Operations, ops-codfw

Tue, Feb 12

Papaul updated the task description for T199247: Decommission baham.
Tue, Feb 12, 5:18 PM · Patch-For-Review, decommission, Operations, ops-codfw

Mon, Feb 11

Papaul closed T209921: ms-be2047 spontaneous reboots as Resolved.

Previous hardware has been already returned since last Thursday. (See comment on Feb7) We can resolve this task.

Mon, Feb 11, 3:49 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul closed T209921: ms-be2047 spontaneous reboots, a subtask of T209395: rack/setup/install new ms-be servers ms-be204[4-9] ,ms-be2050, as Resolved.
Mon, Feb 11, 3:49 PM · User-fgiunchedi, Patch-For-Review, Operations, ops-codfw

Fri, Feb 8

Papaul updated the task description for T199247: Decommission baham.
Fri, Feb 8, 6:00 PM · Patch-For-Review, decommission, Operations, ops-codfw
Papaul added a comment to T204567: ms-be2030 spontaneous reboot.

Checked temperature in the rack all looks good. add blanks to the rack since we have only 8 servers in that rack. Leaving the task open for another week.

Fri, Feb 8, 5:49 PM · ops-codfw, Operations
Papaul closed T203434: Decom mw2213 as Resolved.

complete

Fri, Feb 8, 5:45 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul updated the task description for T203434: Decom mw2213.
Fri, Feb 8, 5:44 PM · Patch-For-Review, decommission, ops-codfw, Operations

Thu, Feb 7

Papaul updated the task description for T203434: Decom mw2213.
Thu, Feb 7, 8:39 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Old server has been shipped out. Shipping information below.

Thu, Feb 7, 4:27 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul updated the task description for T199247: Decommission baham.
Thu, Feb 7, 3:58 PM · Patch-For-Review, decommission, Operations, ops-codfw
Papaul updated the task description for T203434: Decom mw2213.
Thu, Feb 7, 3:34 PM · Patch-For-Review, decommission, ops-codfw, Operations

Wed, Feb 6

Papaul reassigned T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev from Papaul to aborrero.

@aborrero @Andrew all yours . Let me know if you have any questions.

Wed, Feb 6, 10:50 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Wed, Feb 6, 10:47 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul added a comment to T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.

second NIC configuration

Wed, Feb 6, 10:47 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Wed, Feb 6, 10:06 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul updated the task description for T203434: Decom mw2213.
Wed, Feb 6, 7:16 PM · Patch-For-Review, decommission, ops-codfw, Operations
Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Wed, Feb 6, 5:42 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Wed, Feb 6, 5:26 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul added a comment to T215301: codfw spare pool system for partman testing.

Put back sda in the server.

Wed, Feb 6, 5:04 PM · Patch-For-Review, Operations, hardware-requests
Papaul reassigned T215301: codfw spare pool system for partman testing from Papaul to CDanis.
  • Remove sda from the server
  • boot the server
  • server boot without a problem
Wed, Feb 6, 5:03 PM · Patch-For-Review, Operations, hardware-requests
Papaul reassigned T214813: Degraded RAID on thumbor2002 from Papaul to jijiki.

Disk replaced, server didn't boot up.

Wed, Feb 6, 3:53 PM · serviceops, User-jijiki, Operations, ops-codfw

Tue, Feb 5

Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Tue, Feb 5, 6:26 PM · cloud-services-team (Kanban), Patch-For-Review, Operations

Mon, Feb 4

Papaul reassigned T209921: ms-be2047 spontaneous reboots from Papaul to fgiunchedi.

@fgiunchedi I replaced the problematic server with the new one Dell shipped to me. The OS is installed and puppet first run done. I will proceed to the disk wipe on the old server on Wednesday before shipping it back to Dell. Let me know if you have any questions.

Mon, Feb 4, 8:06 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul updated subscribers of T214813: Degraded RAID on thumbor2002.

I put back the bad disk and boot the system and the system boot into OS with no problem. it looks like what @jcrespo and other mentioned on IRC the grub is installed on /dev/sda/ only which is the disk that needs to be replaced. so we need to fix this issue first so I can be able to replace the disk.

Mon, Feb 4, 6:19 PM · serviceops, User-jijiki, Operations, ops-codfw
Papaul added a comment to T214813: Degraded RAID on thumbor2002.

Disk with serial number WMAYP0E607DT has been replaced. Server can not find boot device. Server can not boot to OS after disk replacement.

Mon, Feb 4, 5:48 PM · serviceops, User-jijiki, Operations, ops-codfw
Papaul reassigned T214813: Degraded RAID on thumbor2002 from Papaul to RobH.

Can you please update this disk with which disk failed? Thanks

Mon, Feb 4, 4:55 PM · serviceops, User-jijiki, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Removed old puppet cert for ms-be2047.codfw.wmnet

Mon, Feb 4, 4:30 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

update Netbox with new serial number

Mon, Feb 4, 4:20 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Received replacement server

Mon, Feb 4, 4:11 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw

Fri, Jan 25

Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Fri, Jan 25, 11:35 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Fri, Jan 25, 8:41 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul added a comment to T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.

@Andrew there is no raid controller on the new servers. They all have 2x200GB SSD's

Fri, Jan 25, 6:06 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul added a comment to T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.

@Andrew can you also specify on this task in which VLAN eth1 needs to be for cloudvirt200[1-3]. Thanks

Fri, Jan 25, 4:52 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul added a comment to T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.

@Andrew for all those new servers I am using for partman labvirt-ssd.cfg?

Fri, Jan 25, 4:50 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul reassigned T214663: Degraded RAID on db2068 from Papaul to Marostegui.

@Marostegui disk replacement complete

Fri, Jan 25, 4:20 PM · Operations, ops-codfw
Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Fri, Jan 25, 4:06 AM · cloud-services-team (Kanban), Patch-For-Review, Operations

Thu, Jan 24

Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Thu, Jan 24, 4:03 PM · cloud-services-team (Kanban), Patch-For-Review, Operations

Jan 23 2019

Papaul updated the task description for T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Jan 23 2019, 5:49 PM · cloud-services-team (Kanban), Patch-For-Review, Operations
Papaul claimed T214448: rack/setup/install cloudcontrol2001-dev & cloudvirt200[123]-dev.
Jan 23 2019, 3:47 PM · cloud-services-team (Kanban), Patch-For-Review, Operations

Jan 15 2019

Papaul closed T213790: mgmt host interfaces down for rack D7 in codfw due to ge-0/0/30 on msw1-codfw down as Resolved.

looks like the mgmt switch froze have to unplug and plug the power back. Switch is back up

Jan 15 2019, 5:06 PM · netops, Operations

Jan 14 2019

Papaul reassigned T211070: decommission of restbase200[1-6] (lease return in December 2018) from Papaul to RobH.

This is complete. All servers ready to be ship out.

Jan 14 2019, 5:13 PM · Patch-For-Review, Operations, ops-codfw, DC-Ops, decommission
Papaul reassigned T211023: Decommission elastic2001-2024 from Papaul to RobH.

This is complete. All servers ready to be ship out.

Jan 14 2019, 5:13 PM · Operations, decommission, ops-codfw

Jan 9 2019

Papaul closed T213233: asw-c-codfw - FPC 1 PEM 1 is not powered as Resolved.

papaul@asw-c-codfw> show chassis environment | match Power
Power FPC 1 Power Supply 0 OK

FPC 1 Power Supply 1           OK        
FPC 2 Power Supply 0           OK
Jan 9 2019, 3:48 PM · Operations, ops-codfw

Jan 8 2019

Papaul updated the task description for T211070: decommission of restbase200[1-6] (lease return in December 2018).
Jan 8 2019, 4:23 PM · Patch-For-Review, Operations, ops-codfw, DC-Ops, decommission
Papaul reassigned T212833: es2019 is not responsive from Papaul to Marostegui.

BIOS from 2.4.3 to 2.8.0
IDRAC from 2.40 to 2.61

Jan 8 2019, 4:20 PM · ops-codfw, Operations, Patch-For-Review, DBA

Jan 7 2019

Papaul reassigned T212966: Degraded RAID on db2047 from Papaul to Marostegui.

Disk replacement complete

Jan 7 2019, 4:13 PM · DBA, Operations, ops-codfw

Jan 4 2019

Papaul updated the task description for T211070: decommission of restbase200[1-6] (lease return in December 2018).
Jan 4 2019, 5:46 PM · Patch-For-Review, Operations, ops-codfw, DC-Ops, decommission

Jan 3 2019

Papaul updated the task description for T211070: decommission of restbase200[1-6] (lease return in December 2018).
Jan 3 2019, 6:53 PM · Patch-For-Review, Operations, ops-codfw, DC-Ops, decommission
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Jan 3 2019, 6:40 PM · Operations, decommission, ops-codfw

Dec 21 2018

fgiunchedi awarded T209921: ms-be2047 spontaneous reboots a The World Burns token.
Dec 21 2018, 8:49 AM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw

Dec 20 2018

Papaul updated the task description for T211023: Decommission elastic2001-2024.
Dec 20 2018, 8:06 PM · Operations, decommission, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Dell just called me. They will be shipping a new system and will arrive by the first week on January.

Dec 20 2018, 7:52 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul updated the task description for T211070: decommission of restbase200[1-6] (lease return in December 2018).
Dec 20 2018, 5:22 PM · Patch-For-Review, Operations, ops-codfw, DC-Ops, decommission
Papaul updated the task description for T211070: decommission of restbase200[1-6] (lease return in December 2018).
Dec 20 2018, 3:25 PM · Patch-For-Review, Operations, ops-codfw, DC-Ops, decommission
Papaul closed T212402: Broken power supply on elastic2026 as Resolved.

Power cable got loose as well may be when working on asw-b8-codfw on Tuesday. System is back up.

Dec 20 2018, 3:23 PM · Operations, ops-codfw
Papaul closed T212403: Non-redundant power supply on ms-be2048 as Resolved.

Loose power cable. System is back up.

Dec 20 2018, 3:12 PM · Operations, ops-codfw
Papaul triaged T212403: Non-redundant power supply on ms-be2048 as Normal priority.
Dec 20 2018, 1:52 PM · Operations, ops-codfw

Dec 19 2018

Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Redundancy Policy on this system was set to Not redundant or on the other working system it was set to redundant so we change the settings for this system to redundant as well. Monitoring the system again

Dec 19 2018, 9:45 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul reassigned T210467: codfw row D recable and add QFX from Papaul to ayounsi.
Dec 19 2018, 6:05 PM · User-jijiki, Patch-For-Review, ops-codfw, netops, Operations
Papaul updated the task description for T210467: codfw row D recable and add QFX.
Dec 19 2018, 6:05 PM · User-jijiki, Patch-For-Review, ops-codfw, netops, Operations
Papaul reassigned T212277: Upgrade db2057 firmware from Papaul to Marostegui.

Firmware upgrade complete

Dec 19 2018, 4:06 PM · Operations, ops-codfw, DBA
Papaul updated the task description for T210467: codfw row D recable and add QFX.
Dec 19 2018, 3:49 PM · User-jijiki, Patch-For-Review, ops-codfw, netops, Operations
Papaul added a comment to T210467: codfw row D recable and add QFX.

fpc2-fpc8 xe-2/0/41 and xe-2/0/42
fpc7-fpc8 xe-7/0/43 and xe-7/0/44

Dec 19 2018, 3:48 PM · User-jijiki, Patch-For-Review, ops-codfw, netops, Operations

Dec 18 2018

Papaul updated the task description for T211023: Decommission elastic2001-2024.
Dec 18 2018, 10:28 PM · Operations, decommission, ops-codfw
Papaul reassigned T210447: codfw row A recable and add QFX from Papaul to ayounsi.
Dec 18 2018, 6:50 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul updated the task description for T210447: codfw row A recable and add QFX.
Dec 18 2018, 6:49 PM · Patch-For-Review, ops-codfw, netops, Operations

Dec 17 2018

Papaul updated the task description for T211023: Decommission elastic2001-2024.
Dec 17 2018, 9:14 PM · Operations, decommission, ops-codfw
Papaul added a comment to T211023: Decommission elastic2001-2024.

Before

papaul@asw-c-codfw> show interfaces descriptions | match "ge-1/0/1[0-2]"     
ge-1/0/10       up    down elastic2013
ge-1/0/11       up    down elastic2014
ge-1/0/12       up    down elastic2015
Dec 17 2018, 9:12 PM · Operations, decommission, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

The problem happen again twice after replacing CPU1

Dec 17 2018, 8:33 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul updated the task description for T210467: codfw row D recable and add QFX.
Dec 17 2018, 7:23 PM · User-jijiki, Patch-For-Review, ops-codfw, netops, Operations
Papaul updated the task description for T210467: codfw row D recable and add QFX.
Dec 17 2018, 6:28 PM · User-jijiki, Patch-For-Review, ops-codfw, netops, Operations
Papaul added a comment to T210467: codfw row D recable and add QFX.

connected to scs-c1-codfw on port 48

Dec 17 2018, 6:28 PM · User-jijiki, Patch-For-Review, ops-codfw, netops, Operations
Papaul closed T210456: codfw row B recable and add QFX as Resolved.

Complete

Dec 17 2018, 5:39 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul closed T210456: codfw row B recable and add QFX, a subtask of T196489: upgrade all codfw switch stacks to include additional 10G switch per row, as Resolved.
Dec 17 2018, 5:39 PM · ops-codfw, netops, Operations
Papaul updated the task description for T210456: codfw row B recable and add QFX.
Dec 17 2018, 5:38 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul updated the task description for T210447: codfw row A recable and add QFX.
Dec 17 2018, 5:33 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul added a comment to T210447: codfw row A recable and add QFX.

fpc2-fpc8 connection xe-2/0/41 and xe-2/0/42
fpc7-fpc8 connection xe-7/0/43 and xe-7/0/44

Dec 17 2018, 5:33 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

CPU 1 has been replaced. I clear also the log. The system is back up and I will be monitoring it once again.

Dec 17 2018, 5:28 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw

Dec 14 2018

Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Dell will be shipping 1 New CPU by Monday.

Dec 14 2018, 4:23 PM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw

Dec 13 2018

Papaul closed T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) as Resolved.

This can be resolved then since i am done with it .

Dec 13 2018, 6:43 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Dec 13 2018, 4:06 PM · Operations, decommission, ops-codfw
Papaul added a comment to T211023: Decommission elastic2001-2024.
papaul@asw-a-codfw# run show interfaces ge-5/0/8 descriptions 
Interface       Admin Link Description
ge-5/0/8        down  down DISABLED
Dec 13 2018, 4:04 PM · Operations, decommission, ops-codfw
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Dec 13 2018, 3:23 PM · Operations, decommission, ops-codfw
Papaul closed T211715: Interface errors on cr1-codfw:xe-5/3/1 as Resolved.

This is complete

Dec 13 2018, 3:20 PM · Operations, ops-codfw

Dec 12 2018

Papaul updated the task description for T211023: Decommission elastic2001-2024.
Dec 12 2018, 8:42 PM · Operations, decommission, ops-codfw
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Dec 12 2018, 8:35 PM · Operations, decommission, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

same error again at 22:47

Dec 12 2018, 6:22 AM · Patch-For-Review, User-fgiunchedi, Operations, ops-codfw
Papaul added a comment to T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).

@Marostegui no need to close the task. It can be assign to @RobH so he can keep track

Dec 12 2018, 6:13 AM · Patch-For-Review, decommission, Operations, ops-codfw, DBA

Dec 11 2018

Papaul updated the task description for T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).
Dec 11 2018, 8:56 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).

@Marostegui any reason why production DNS is still showing for pc2004?

Dec 11 2018, 8:46 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA