Page MenuHomePhabricator

Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (234 w, 3 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Sat, Jun 15

Papaul added a comment to T222041: cp3037 is currently unreachable.

cp3037 is dead, it needs new mainboard since it is out of warranty better decommission it.

Sat, Jun 15, 10:39 AM · ops-esams, Operations, Traffic

Thu, Jun 13

Papaul updated the task description for T224250: Setup new msw1-codfw.
Thu, Jun 13, 9:02 PM · ops-codfw, netops, Operations

Tue, Jun 11

Papaul updated the task description for T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.
Tue, Jun 11, 2:27 PM · Cloud-Services, ops-codfw, Operations

Fri, Jun 7

Papaul updated subscribers of T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].

@ayounsi I am planning on installing those new servers in row c and row D and I don't have the "interface-range ganeti" in both of those rows Is it okay for me to go ahead and create "interface-range ganeti" on asw-c-codfw and asw-d-codfw?

Fri, Jun 7, 3:36 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].
Fri, Jun 7, 3:20 PM · Patch-For-Review, ops-codfw, Operations

Thu, Jun 6

Papaul updated the task description for T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.
Thu, Jun 6, 4:04 PM · Cloud-Services, ops-codfw, Operations

Wed, Jun 5

Papaul updated the task description for T209425: Decommission rdb2001, rdb2002.
Wed, Jun 5, 2:34 PM · Patch-For-Review, ops-codfw, User-jijiki, decommission, Operations
Papaul closed T209425: Decommission rdb2001, rdb2002 as Resolved.

This is complete

Wed, Jun 5, 2:34 PM · Patch-For-Review, ops-codfw, User-jijiki, decommission, Operations
Papaul closed T209425: Decommission rdb2001, rdb2002, a subtask of T206450: Reorganize our redis rdb1/rdb2 clusters, as Resolved.
Wed, Jun 5, 2:34 PM · Patch-For-Review, User-jijiki, User-Joe, Operations
Papaul lowered the priority of T213417: lvs2002: raid battery failure from Normal to Low.
Wed, Jun 5, 2:05 PM · Operations, ops-codfw, Traffic

Tue, Jun 4

Papaul added a comment to T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.

Both MD's are racked and Netbox updated.

Tue, Jun 4, 10:18 PM · Cloud-Services, ops-codfw, Operations
Papaul updated the task description for T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].
Tue, Jun 4, 9:58 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].
Tue, Jun 4, 8:27 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].
Tue, Jun 4, 12:40 PM · Patch-For-Review, ops-codfw, Operations
Papaul reassigned T223493: rack/setup/install kafka-main200[1-5] from Papaul to herron.
Tue, Jun 4, 12:39 PM · ops-codfw, Operations
Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

@herron fixed

Tue, Jun 4, 12:38 PM · ops-codfw, Operations

Mon, Jun 3

Papaul updated the task description for T224250: Setup new msw1-codfw.
Mon, Jun 3, 4:26 PM · ops-codfw, netops, Operations
Papaul added a comment to T224250: Setup new msw1-codfw.

Before upgrade

Mon, Jun 3, 4:26 PM · ops-codfw, netops, Operations
Papaul reassigned T223493: rack/setup/install kafka-main200[1-5] from Papaul to herron.

@herron You can merge the DHCP code and you should be good.

Mon, Jun 3, 3:36 PM · ops-codfw, Operations
Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

I would expect though that the DHCP requests would make it to the install servers, with or without entries in the dhcp config file.

Mon, Jun 3, 3:21 PM · ops-codfw, Operations
Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

kafak-main2002
after power drain

/admin1-> racadm serveraction powercycle
Server power operation initiated successfully
/admin1->
Mon, Jun 3, 3:14 PM · ops-codfw, Operations
Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].
  1. kafka-main200[2345] are not yet able to net boot

13:25 < papaul> herron: the problem is i added just kafka-main2001 to the DHCP file and

not the others that is why you can not boot the other servers

13:25 < papaul> since you wanted to do only the test on 2001

Mon, Jun 3, 2:45 PM · ops-codfw, Operations

Fri, May 31

Papaul added a comment to T222967: Interface errors on cr1-codfw: xe-5/3/1.

Dear Customer,

Fri, May 31, 2:52 PM · Operations, ops-codfw
Papaul added a comment to T222967: Interface errors on cr1-codfw: xe-5/3/1.

Email Telia again with another follow up email.

Fri, May 31, 2:29 PM · Operations, ops-codfw

Thu, May 30

Papaul updated the task description for T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].
Thu, May 30, 5:28 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

@herron also after the OS install please remember to change Netbox status to "staged"

Thu, May 30, 5:18 PM · ops-codfw, Operations
Papaul reassigned T223493: rack/setup/install kafka-main200[1-5] from Papaul to herron.

@herron All is done at my end what left to be done is just the OS install. Let me know if you have any questions

Thu, May 30, 5:16 PM · ops-codfw, Operations
Papaul updated the task description for T223493: rack/setup/install kafka-main200[1-5].
Thu, May 30, 5:15 PM · ops-codfw, Operations

Wed, May 29

Papaul added a comment to T224250: Setup new msw1-codfw.

console information:
scs-a1-codwf port 40

Wed, May 29, 4:48 PM · ops-codfw, netops, Operations
Papaul updated the task description for T224250: Setup new msw1-codfw.
Wed, May 29, 4:28 PM · ops-codfw, netops, Operations
Papaul triaged T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8] as Normal priority.
Wed, May 29, 4:13 PM · Patch-For-Review, ops-codfw, Operations
Papaul created T224603: rack/setup/ codfw: ganeti2009 - ganeti201[0-8].
Wed, May 29, 3:57 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.
Wed, May 29, 3:35 PM · Cloud-Services, ops-codfw, Operations
Papaul updated the task description for T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.
Wed, May 29, 3:31 PM · Cloud-Services, ops-codfw, Operations
jcrespo awarded T223492: rack/setup/install dbproxy200[1-4] a Love token.
Wed, May 29, 7:20 AM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul added a comment to T223492: rack/setup/install dbproxy200[1-4].

@Marostegui @jcrespo All your's you can take this task anytime.

Wed, May 29, 2:14 AM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul updated the task description for T223492: rack/setup/install dbproxy200[1-4].
Wed, May 29, 2:13 AM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul updated the task description for T223492: rack/setup/install dbproxy200[1-4].
Wed, May 29, 12:27 AM · Patch-For-Review, ops-codfw, Operations, DBA

Tue, May 28

Papaul updated the task description for T209425: Decommission rdb2001, rdb2002.
Tue, May 28, 11:07 PM · Patch-For-Review, ops-codfw, User-jijiki, decommission, Operations
Restricted Application added a project to T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet: Operations.
Tue, May 28, 10:46 PM · Cloud-Services, ops-codfw, Operations
Papaul updated the task description for T223492: rack/setup/install dbproxy200[1-4].
Tue, May 28, 8:00 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul added a comment to T222654: ms-be2043 'sdd' throwing lots of errors.

Return information below

Tue, May 28, 6:35 PM · User-fgiunchedi, ops-codfw, observability, media-storage, Operations
Papaul updated the task description for T223492: rack/setup/install dbproxy200[1-4].
Tue, May 28, 6:23 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul updated the task description for T223493: rack/setup/install kafka-main200[1-5].
Tue, May 28, 5:57 PM · ops-codfw, Operations
Papaul closed T224393: db2091 rebooted unexpectedly as Resolved.

Power drain and firmware upgrade.
Before

Tue, May 28, 5:55 PM · Patch-For-Review, Operations, DBA
Papaul reassigned T224456: Degraded RAID on db2035 from Papaul to jcrespo.

disk replacement complete

Tue, May 28, 3:43 PM · DBA, Operations, ops-codfw

Fri, May 24

Papaul added a comment to T222967: Interface errors on cr1-codfw: xe-5/3/1.

Dear Customer,

Fri, May 24, 7:40 PM · Operations, ops-codfw
Papaul added a comment to T222967: Interface errors on cr1-codfw: xe-5/3/1.

CY1 did test the X-connect and didn't find any problem. see
https://phabricator.wikimedia.org/T224196
Sending other follow up email to Telia

Fri, May 24, 6:18 PM · Operations, ops-codfw

Thu, May 23

Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

@herron can't find a partman recipe

Thu, May 23, 9:15 PM · ops-codfw, Operations
Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

Change 512236 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address for kafka-main2001

Thu, May 23, 8:54 PM · ops-codfw, Operations
Papaul added a comment to T223492: rack/setup/install dbproxy200[1-4].

oops, wrong ticket for the Gerrit comment

Thu, May 23, 8:54 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul updated the task description for T223492: rack/setup/install dbproxy200[1-4].
Thu, May 23, 8:32 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul updated the task description for T223493: rack/setup/install kafka-main200[1-5].
Thu, May 23, 3:04 PM · ops-codfw, Operations

Wed, May 22

Papaul closed T222383: pull decom hardware and ship to Harry/OIT @ SF office as Resolved.

You welcome.

Wed, May 22, 11:28 PM · ops-codfw, Operations
Papaul updated the task description for T223493: rack/setup/install kafka-main200[1-5].
Wed, May 22, 7:32 PM · ops-codfw, Operations
Papaul updated the task description for T223493: rack/setup/install kafka-main200[1-5].
Wed, May 22, 6:09 PM · ops-codfw, Operations
Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

@Ottomata thanks for the updates. Can you please make the necessaries modifications needed on raid10-gpt-srv-lvm-ext4.cfg .

Wed, May 22, 4:52 PM · ops-codfw, Operations

Tue, May 21

Papaul added a comment to T223493: rack/setup/install kafka-main200[1-5].

@Ottomata can you please provide me with the partman recipe for those systems and for the RAID config do you want user 256KB for RAID Stripe size?

Tue, May 21, 8:46 PM · ops-codfw, Operations
Papaul updated the task description for T209425: Decommission rdb2001, rdb2002.
Tue, May 21, 8:39 PM · Patch-For-Review, ops-codfw, User-jijiki, decommission, Operations
Papaul closed Unknown Object (Task), a subtask of T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019., as Resolved.
Tue, May 21, 8:36 PM · User-herron, Core Platform Team (Modern Event Platform (TEC2)), Core Platform Team Backlog (Watching / External), Services (watching), EventBus, Analytics, Operations
Papaul updated the task description for T223493: rack/setup/install kafka-main200[1-5].
Tue, May 21, 8:35 PM · ops-codfw, Operations
Papaul updated the task description for T223493: rack/setup/install kafka-main200[1-5].
Tue, May 21, 4:36 PM · ops-codfw, Operations

Mon, May 20

Papaul reassigned T222383: pull decom hardware and ship to Harry/OIT @ SF office from Papaul to HMarcus.

Please see below for tracking information.

Mon, May 20, 5:02 PM · ops-codfw, Operations

May 17 2019

Papaul added a comment to T222967: Interface errors on cr1-codfw: xe-5/3/1.

Dear Customer,

May 17 2019, 5:28 PM · Operations, ops-codfw
Papaul added a comment to T222967: Interface errors on cr1-codfw: xe-5/3/1.

Follow up email sent to Telia.

May 17 2019, 4:48 PM · Operations, ops-codfw

May 16 2019

Papaul added projects to T223493: rack/setup/install kafka-main200[1-5]: Analytics-Kanban, Patch-For-Review.
May 16 2019, 10:53 PM · ops-codfw, Operations
Papaul updated subscribers of T223493: rack/setup/install kafka-main200[1-5].
May 16 2019, 10:51 PM · ops-codfw, Operations
Papaul triaged T223493: rack/setup/install kafka-main200[1-5] as Normal priority.
May 16 2019, 10:50 PM · ops-codfw, Operations
Papaul created T223493: rack/setup/install kafka-main200[1-5].
May 16 2019, 10:49 PM · ops-codfw, Operations
Papaul triaged T223492: rack/setup/install dbproxy200[1-4] as Normal priority.
May 16 2019, 10:24 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul created T223492: rack/setup/install dbproxy200[1-4].
May 16 2019, 10:23 PM · Patch-For-Review, ops-codfw, Operations, DBA
Papaul closed T221134: reclaim to spares pool: restbase2007.codfw.wmnet, restbase2008.codfw.wmnet as Resolved.

Complete

May 16 2019, 8:33 PM · Patch-For-Review, DC-Ops, decommission
Papaul closed T221134: reclaim to spares pool: restbase2007.codfw.wmnet, restbase2008.codfw.wmnet, a subtask of T208087: Replace remaining Samsung SSDs, as Resolved.
May 16 2019, 8:33 PM · Patch-For-Review, Services (doing), Core Platform Team Kanban (Doing), Core Platform Team (Security, stability, performance and scalability (TEC1)), User-Eevans, Cassandra
Papaul updated the task description for T221134: reclaim to spares pool: restbase2007.codfw.wmnet, restbase2008.codfw.wmnet.
May 16 2019, 8:31 PM · Patch-For-Review, DC-Ops, decommission
Papaul closed T223468: audit offline codfw devices with rack location assigned as Resolved.

complete

May 16 2019, 8:25 PM · ops-codfw, ops-eqiad, Operations, Operations-Software-Development, netbox, DC-Ops
Papaul closed T223468: audit offline codfw devices with rack location assigned, a subtask of T223450: Triage and resolve all outstanding Netbox report errors, as Resolved.
May 16 2019, 8:25 PM · ops-codfw, ops-eqiad, Operations, Operations-Software-Development, netbox, DC-Ops
Papaul added a comment to T222383: pull decom hardware and ship to Harry/OIT @ SF office.

RAID memories (6x8) controller removed from
ms-be2001
ms-be2002
ms-be2006
ms-be2007
ms-be2008
ms-be2009
ms-be2010
ms-be2011

May 16 2019, 5:55 PM · ops-codfw, Operations
Papaul updated the task description for T209425: Decommission rdb2001, rdb2002.
May 16 2019, 4:56 PM · Patch-For-Review, ops-codfw, User-jijiki, decommission, Operations
Papaul renamed T209425: Decommission rdb2001, rdb2002 from Reclaim rdb2001, rdb2002 to Decommission rdb2001, rdb2002.
May 16 2019, 4:51 PM · Patch-For-Review, ops-codfw, User-jijiki, decommission, Operations
Papaul claimed T209425: Decommission rdb2001, rdb2002.
May 16 2019, 4:50 PM · Patch-For-Review, ops-codfw, User-jijiki, decommission, Operations
Papaul added a comment to T222654: ms-be2043 'sdd' throwing lots of errors.

@fgiunchedi so what do you want to do here? I still have the old disk with me. Do you want me to keep it with me and not ship it back to Dell for now?

May 16 2019, 4:25 PM · User-fgiunchedi, ops-codfw, observability, media-storage, Operations
Papaul closed T223455: mw2199 unreachable after reboot as Resolved.

@MoritzMuehlenhoff drained the power from the server and did a firmware upgrade as well. Tested the power cycle command all work great. Server is back up.

May 16 2019, 4:17 PM · Operations, DC-Ops
Papaul updated the task description for T221134: reclaim to spares pool: restbase2007.codfw.wmnet, restbase2008.codfw.wmnet.
May 16 2019, 3:31 PM · Patch-For-Review, DC-Ops, decommission
Papaul triaged T223455: mw2199 unreachable after reboot as Normal priority.
May 16 2019, 2:31 PM · Operations, DC-Ops
Papaul reassigned T222654: ms-be2043 'sdd' throwing lots of errors from Papaul to fgiunchedi.

@fgiunchedi disk replaced

May 16 2019, 2:30 PM · User-fgiunchedi, ops-codfw, observability, media-storage, Operations

May 15 2019

Papaul closed T218022: decommission: labtestservices2001.wikimedia.org as Resolved.

Complete

May 15 2019, 4:32 PM · Patch-For-Review, Operations, decommission, ops-codfw
Papaul closed T218022: decommission: labtestservices2001.wikimedia.org, a subtask of T217891: CloudVPS: rework codfw deployments, as Resolved.
May 15 2019, 4:32 PM · Cloud-VPS, cloud-services-team (Kanban)
Papaul updated the task description for T218022: decommission: labtestservices2001.wikimedia.org.
May 15 2019, 4:32 PM · Patch-For-Review, Operations, decommission, ops-codfw
Papaul added a comment to T209337: lvs2006 crashed into (what it seems) an unrecoverable state.

@ArielGlenn that system is out of warranty and the plan is to replace it with the systems in T196560

May 15 2019, 3:00 PM · Patch-For-Review, ops-codfw, Operations, Traffic
Papaul closed T218024: decommmision: labtestweb2001.wikimedia.org as Resolved.

complete

May 15 2019, 2:47 PM · Patch-For-Review, decommission, ops-codfw, Operations, cloud-services-team (Kanban)
Papaul closed T218024: decommmision: labtestweb2001.wikimedia.org, a subtask of T217891: CloudVPS: rework codfw deployments, as Resolved.
May 15 2019, 2:47 PM · Cloud-VPS, cloud-services-team (Kanban)
Papaul updated the task description for T218024: decommmision: labtestweb2001.wikimedia.org.
May 15 2019, 2:46 PM · Patch-For-Review, decommission, ops-codfw, Operations, cloud-services-team (Kanban)

May 14 2019

Papaul added a comment to T222967: Interface errors on cr1-codfw: xe-5/3/1.
  • Email sent to Telia
  • Telia open a case
  • Case information : Telia Carrier Case Reference 00980021
May 14 2019, 5:56 PM · Operations, ops-codfw
Papaul updated the task description for T218024: decommmision: labtestweb2001.wikimedia.org.
May 14 2019, 3:49 PM · Patch-For-Review, decommission, ops-codfw, Operations, cloud-services-team (Kanban)
Papaul closed T217398: elastic2038 DOWN (CPU/memory errors ) as Resolved.
  • Memory Replaced on DIMM B2
  • clear log
May 14 2019, 3:40 PM · Discovery-Search (Current work), Operations, ops-codfw
Papaul added a comment to T222654: ms-be2043 'sdd' throwing lots of errors.

Create Dispatch: Success
You have successfully submitted request SR990663287.

May 14 2019, 2:59 PM · User-fgiunchedi, ops-codfw, observability, media-storage, Operations

May 13 2019

Papaul added a comment to T217398: elastic2038 DOWN (CPU/memory errors ).

Create Dispatch: Success
You have successfully submitted request SR990577292.

May 13 2019, 1:37 PM · Discovery-Search (Current work), Operations, ops-codfw
Papaul added a comment to T217398: elastic2038 DOWN (CPU/memory errors ).

It looks like the error is showing now on DIMM B2, so we have a bad DIMM. I will go ahead and request a replacement.

May 13 2019, 2:53 AM · Discovery-Search (Current work), Operations, ops-codfw
Papaul claimed T217398: elastic2038 DOWN (CPU/memory errors ).
May 13 2019, 2:47 AM · Discovery-Search (Current work), Operations, ops-codfw

May 9 2019

Papaul closed T222753: db2114 hardware problem as Resolved.

@Marostegui @jcrespo main board replacement complete on db2114. The problem has been resolved. You can take over now.
System Board Fan1A 14% 3480 RPM 840 RPM N/A 480 RPM N/A
System Board Fan1B 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan2A 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan2B 14% 5040 RPM 840 RPM N/A 480 RPM N/A
System Board Fan3A 14% 3480 RPM 840 RPM N/A 480 RPM N/A
System Board Fan3B 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan4A 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan4B 14% 5160 RPM 840 RPM N/A 480 RPM N/A
System Board Fan5A 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan5B 14% 5160 RPM 840 RPM N/A 480 RPM N/A
System Board Fan6A 14% 3480 RPM 840 RPM N/A 480 RPM N/A
System Board Fan6B 14% 3840 RPM 840 RPM N/A 480 RPM N/A

May 9 2019, 4:08 PM · Operations, ops-codfw, DBA