Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (208 w, 2 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Yesterday

Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Dell will be shipping 1 New CPU by Monday.

Fri, Dec 14, 4:23 PM · User-fgiunchedi, Operations, ops-codfw

Thu, Dec 13

Papaul closed T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) as Resolved.

This can be resolved then since i am done with it .

Thu, Dec 13, 6:43 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Thu, Dec 13, 4:06 PM · decommission, ops-codfw, Operations, Discovery-Search (Current work)
Papaul added a comment to T211023: Decommission elastic2001-2024.
papaul@asw-a-codfw# run show interfaces ge-5/0/8 descriptions 
Interface       Admin Link Description
ge-5/0/8        down  down DISABLED
Thu, Dec 13, 4:04 PM · decommission, ops-codfw, Operations, Discovery-Search (Current work)
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Thu, Dec 13, 3:23 PM · decommission, ops-codfw, Operations, Discovery-Search (Current work)
Papaul closed T211715: Interface errors on cr1-codfw:xe-5/3/1 as Resolved.

This is complete

Thu, Dec 13, 3:20 PM · ops-codfw, Operations

Wed, Dec 12

Papaul updated the task description for T211023: Decommission elastic2001-2024.
Wed, Dec 12, 8:42 PM · decommission, ops-codfw, Operations, Discovery-Search (Current work)
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Wed, Dec 12, 8:35 PM · decommission, ops-codfw, Operations, Discovery-Search (Current work)
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

same error again at 22:47

Wed, Dec 12, 6:22 AM · User-fgiunchedi, Operations, ops-codfw
Papaul added a comment to T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).

@Marostegui no need to close the task. It can be assign to @RobH so he can keep track

Wed, Dec 12, 6:13 AM · Patch-For-Review, decommission, Operations, ops-codfw, DBA

Tue, Dec 11

Papaul updated the task description for T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).
Tue, Dec 11, 8:56 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).

@Marostegui any reason why production DNS is still showing for pc2004?

Tue, Dec 11, 8:46 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).
Tue, Dec 11, 8:41 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).
papaul@asw-b-codfw> show interfaces ge-5/0/35 descriptions 
Interface       Admin Link Description
ge-5/0/35       down  down DISABLED
Tue, Dec 11, 8:40 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Tue, Dec 11, 5:28 PM · decommission, ops-codfw, Operations, Discovery-Search (Current work)
Papaul updated the task description for T210456: codfw row B recable and add QFX.
Tue, Dec 11, 4:55 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul updated the task description for T211023: Decommission elastic2001-2024.
Tue, Dec 11, 4:35 PM · decommission, ops-codfw, Operations, Discovery-Search (Current work)
Papaul added a comment to T210456: codfw row B recable and add QFX.

For the connection fpc2-fpc8 on fpc2 I will be using xe-2/0/41 and xe-2/0/42

Tue, Dec 11, 4:28 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Fresh OS installed on the system. Leaving the system up again to see.

Tue, Dec 11, 4:00 PM · User-fgiunchedi, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Stress test on the system cam out with no errors.

Tue, Dec 11, 3:37 PM · User-fgiunchedi, Operations, ops-codfw

Mon, Dec 10

Papaul updated the task description for T210456: codfw row B recable and add QFX.
Mon, Dec 10, 4:32 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul updated the task description for T210456: codfw row B recable and add QFX.
Mon, Dec 10, 4:28 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul updated the task description for T210456: codfw row B recable and add QFX.
Mon, Dec 10, 4:23 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul added a comment to T210456: codfw row B recable and add QFX.

The switch is connected to port 48 on scs-a1-codfw

Mon, Dec 10, 4:22 PM · Patch-For-Review, ops-codfw, netops, Operations
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Running Stress test on the system

Mon, Dec 10, 4:15 PM · User-fgiunchedi, Operations, ops-codfw
Papaul closed T211219: IPMI Sensor error on elastic2051 as Resolved.

This looks good now

Mon, Dec 10, 3:46 PM · Discovery-Search (Current work), DC-Ops

Sat, Dec 8

Papaul updated the task description for T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).
Sat, Dec 8, 6:51 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA

Fri, Dec 7

Papaul reassigned T207965: eqiad: Re-connect cage cameras from Papaul to Cmjohnson.
Fri, Dec 7, 7:17 PM · ops-eqiad, Operations
Papaul added a comment to T207965: eqiad: Re-connect cage cameras .
papaul@asw2-a-eqiad# run show interfaces ge-1/0/2 descriptions                     
Interface       Admin Link Description
ge-1/0/2        up    down cam1-a-b-eqiad.eqiad.wmnet
Fri, Dec 7, 7:16 PM · ops-eqiad, Operations
Papaul reassigned T211065: rack/setup/install codfw logstash elasticsearch storage servers from Papaul to fgiunchedi.

@fgiunchedi all yours

Fri, Dec 7, 3:41 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Fri, Dec 7, 3:41 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Fri, Dec 7, 3:09 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T211170: Time on new servers different from time on puppetmaster1001.

@MoritzMuehlenhoff no there is no server for now which is still in pristine condition right after the OS install. But I am about to install logstatsh200[1-3] once done, I will ping you.

Fri, Dec 7, 2:05 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Fri, Dec 7, 2:21 AM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T211065: rack/setup/install codfw logstash elasticsearch storage servers.

@fgiunchedi the installation is complaining about not finding any swap partition.

Fri, Dec 7, 2:20 AM · Patch-For-Review, Operations, ops-codfw

Thu, Dec 6

Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Andy.Johnson@dell.com

9:40 AM (21 minutes ago)

to me, faidon

Thu, Dec 6, 4:02 PM · User-fgiunchedi, Operations, ops-codfw

Wed, Dec 5

Papaul updated the task description for T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Wed, Dec 5, 10:30 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Wed, Dec 5, 9:30 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

@fgiunchedi please see below.

Wed, Dec 5, 6:20 PM · User-fgiunchedi, Operations, ops-codfw
Papaul triaged T211219: IPMI Sensor error on elastic2051 as Normal priority.
Wed, Dec 5, 5:08 PM · Discovery-Search (Current work), DC-Ops
Papaul added a comment to T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).

@RobH any reason why we have to add the servers that we are returning to the decommission tracking Google sheet since that sheet is to keep track of servers and other devices that are decommissioned and have on site.

Wed, Dec 5, 3:35 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul updated the task description for T209858: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).
Wed, Dec 5, 3:31 PM · Patch-For-Review, decommission, Operations, ops-codfw, DBA
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

After 16 hours of hardware diagnostics, the server came up with no error. I have a Call schedule with Dell in 2 hours to discuss about the next step to take.

Wed, Dec 5, 3:29 PM · User-fgiunchedi, Operations, ops-codfw
Papaul reassigned T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4] from Papaul to Gehel.

@Gehel all yours

Wed, Dec 5, 2:26 AM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Wed, Dec 5, 2:25 AM · Patch-For-Review, Operations, ops-codfw
Papaul renamed T211170: Time on new servers different from time on puppetmaster1001 from Time on new servers different form time on puppetmaster1001 to Time on new servers different from time on puppetmaster1001.
Wed, Dec 5, 1:46 AM · Patch-For-Review, ops-codfw, Operations
Papaul created T211170: Time on new servers different from time on puppetmaster1001.
Wed, Dec 5, 1:46 AM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Wed, Dec 5, 1:11 AM · Patch-For-Review, Operations, ops-codfw

Tue, Dec 4

Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Tue, Dec 4, 10:22 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Tue, Dec 4, 10:14 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Tue, Dec 4, 10:05 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Tue, Dec 4, 9:34 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Tue, Dec 4, 1:43 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T211065: rack/setup/install codfw logstash elasticsearch storage servers.

@fgiunchedi Please provide partman recipe to use. I have 4x4TB disks

Tue, Dec 4, 1:20 PM · Patch-For-Review, Operations, ops-codfw

Mon, Dec 3

Papaul updated the task description for T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Mon, Dec 3, 10:52 PM · Patch-For-Review, Operations, ops-codfw
Papaul triaged T211065: rack/setup/install codfw logstash elasticsearch storage servers as High priority.
Mon, Dec 3, 10:46 PM · Patch-For-Review, Operations, ops-codfw
Papaul created T211065: rack/setup/install codfw logstash elasticsearch storage servers.
Mon, Dec 3, 10:45 PM · Patch-For-Review, Operations, ops-codfw
Papaul closed Unknown Object (Task), a subtask of T205850: Procure and provision Logging pipeline hardware in multiple datacenters, as Resolved.
Mon, Dec 3, 10:38 PM · Patch-For-Review, Wikimedia-Logstash, Operations
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Mon, Dec 3, 10:34 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Mon, Dec 3, 10:09 PM · Patch-For-Review, Operations, ops-codfw
Papaul reassigned T208096: Degraded RAID on ms-be2021 from Papaul to fgiunchedi.

@fgiunchedi All looks good on the system. You can resolve the task when finished double checking.

Mon, Dec 3, 4:50 PM · Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

update from Dell

Mon, Dec 3, 3:42 PM · User-fgiunchedi, Operations, ops-codfw
Papaul updated the task description for T210447: codfw row A recable and add QFX.
Mon, Dec 3, 3:40 PM · ops-codfw, netops, Operations
Papaul updated the task description for T210447: codfw row A recable and add QFX.
Mon, Dec 3, 3:35 PM · ops-codfw, netops, Operations
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Mon, Dec 3, 2:42 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].

@Gehel all first 8 servers are all yours

Mon, Dec 3, 2:40 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].

@Gehel The first 8 servers (elastic2037-elastic2044) are ready. The only thing left is the first puppet run. When running puppet agent on those nodes I am getting the error below. I think you can fix that for me. Thank you.

Mon, Dec 3, 1:56 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Replaced all the parts that was shipped to me by Dell (main board, RAID controller, RAID controller interposer board.SAS cable) swapped CPU1 with CPU2 we have the same problem on the server. I email Dell last Friday, waiting for Dell to get back in touch with me.

Mon, Dec 3, 5:15 AM · User-fgiunchedi, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Mon, Dec 3, 5:10 AM · Patch-For-Review, Operations, ops-codfw

Fri, Nov 30

Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Fri, Nov 30, 9:44 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Fri, Nov 30, 9:30 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Fri, Nov 30, 9:29 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T210863: Reconfigure hardware and reimage restbase201[3-8].codfw.wmnet.

@fgiunchedi @Eevans I removed 2 SSD's from each server.

Fri, Nov 30, 8:04 PM · Patch-For-Review, User-Eevans, User-fgiunchedi, Services (watching), ops-codfw, Operations
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Fri, Nov 30, 4:56 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Fri, Nov 30, 4:41 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

@colewhite Thank you I saw that

Fri, Nov 30, 2:26 AM · User-fgiunchedi, Operations, ops-codfw

Thu, Nov 29

Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Thu, Nov 29, 10:29 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Thu, Nov 29, 9:47 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Thu, Nov 29, 8:34 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Thu, Nov 29, 8:16 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Did all the dell engineer recommended above. Waiting to proceed to step 10 .

Thu, Nov 29, 4:28 PM · User-fgiunchedi, Operations, ops-codfw
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

Please replace the remaining hardware sent.

Thu, Nov 29, 4:08 PM · User-fgiunchedi, Operations, ops-codfw

Wed, Nov 28

Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Wed, Nov 28, 6:23 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Wed, Nov 28, 4:13 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].

@Gehel In this case the racking proposal will not work since those racks are 1G rack. I will update the task description with the new racking proposal.

Wed, Nov 28, 4:12 PM · Patch-For-Review, Operations, ops-codfw
Papaul reassigned T209615: rack/setup/install restbase201[3-8].codfw.wmnet from Papaul to RobH.

Complete

Wed, Nov 28, 2:12 AM · User-fgiunchedi, Patch-For-Review, Services (watching), ops-codfw, Operations
Papaul updated the task description for T209615: rack/setup/install restbase201[3-8].codfw.wmnet.
Wed, Nov 28, 2:09 AM · User-fgiunchedi, Patch-For-Review, Services (watching), ops-codfw, Operations
Papaul updated subscribers of T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].

@Gehel My racking proposal was to rack some of those server in rack 4 of row A,B and D. @ayounsi and I will be working on replacing all the 1G switches in those rack soon with 10G switches; or the new servers came with both 1GB and 10GB NIC's. So my question to you is :
Are you going to use the 10G NIC or 1G NIC?

Wed, Nov 28, 1:27 AM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T209615: rack/setup/install restbase201[3-8].codfw.wmnet.
Wed, Nov 28, 1:13 AM · User-fgiunchedi, Patch-For-Review, Services (watching), ops-codfw, Operations
Papaul updated the task description for T209615: rack/setup/install restbase201[3-8].codfw.wmnet.
Wed, Nov 28, 1:02 AM · User-fgiunchedi, Patch-For-Review, Services (watching), ops-codfw, Operations
Papaul reassigned T209389: rack/setup/install sessionstore200[123].codfw.wmnet from Papaul to RobH.

This is complete at my end. Don't know the service owner so giving it back to you.

Wed, Nov 28, 1:02 AM · Services (watching), Core Platform Team Backlog (Watching / External), Operations, Core Platform Team (Session Management Service (CDP2))

Tue, Nov 27

Papaul updated the task description for T209615: rack/setup/install restbase201[3-8].codfw.wmnet.
Tue, Nov 27, 10:16 PM · User-fgiunchedi, Patch-For-Review, Services (watching), ops-codfw, Operations
Papaul updated the task description for T209615: rack/setup/install restbase201[3-8].codfw.wmnet.
Tue, Nov 27, 8:56 PM · User-fgiunchedi, Patch-For-Review, Services (watching), ops-codfw, Operations
Papaul updated the task description for T209389: rack/setup/install sessionstore200[123].codfw.wmnet.
Tue, Nov 27, 8:06 PM · Services (watching), Core Platform Team Backlog (Watching / External), Operations, Core Platform Team (Session Management Service (CDP2))
Papaul added a comment to T209615: rack/setup/install restbase201[3-8].codfw.wmnet.

papaul@asw-b-codfw> show interfaces ge-5/0/5 descriptions
Interface Admin Link Description
ge-5/0/5 up up restbase2013

Tue, Nov 27, 7:36 PM · User-fgiunchedi, Patch-For-Review, Services (watching), ops-codfw, Operations
Papaul added a comment to T209921: ms-be2047 spontaneous reboots.

I have replaced the system board on the system. The system is back online. I will be monitoring the system to see if we do have the same problem again. Also I have on site a new : H730P RAID controller card, SAS cable and RAID controller interposer board that Dell shipped to me.

Tue, Nov 27, 5:42 PM · User-fgiunchedi, Operations, ops-codfw
Papaul renamed T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4] from rack/setup/install elastic201[6-9], elastic202[0-9] and elastic203[0-3] to rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Tue, Nov 27, 3:51 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T210450: rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].
Tue, Nov 27, 3:10 AM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T209389: rack/setup/install sessionstore200[123].codfw.wmnet.
Tue, Nov 27, 2:31 AM · Services (watching), Core Platform Team Backlog (Watching / External), Operations, Core Platform Team (Session Management Service (CDP2))
Papaul added a comment to T209389: rack/setup/install sessionstore200[123].codfw.wmnet.
papaul@asw-b-codfw> show interfaces ge-4/0/1 descriptions 
Interface       Admin Link Description
ge-4/0/1        up    up   sessionstore2001
Tue, Nov 27, 1:11 AM · Services (watching), Core Platform Team Backlog (Watching / External), Operations, Core Platform Team (Session Management Service (CDP2))