Papaul (Papaul)
User

Projects (6)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (182 w, 5 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Yesterday

Papaul reassigned T196666: rack/setup/add to spares tracking 2 single cpu misc class systems from Papaul to RobH.

@RobH done at my end

Tue, Jun 19, 4:19 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196666: rack/setup/add to spares tracking 2 single cpu misc class systems.
Tue, Jun 19, 4:18 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196666: rack/setup/add to spares tracking 2 single cpu misc class systems.
Tue, Jun 19, 3:57 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196666: rack/setup/add to spares tracking 2 single cpu misc class systems.

Switch port information :
both servers are racked in D8
wmf6652 ge-8/0/3
wmf6653 ge-8/0/4

Tue, Jun 19, 3:56 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196666: rack/setup/add to spares tracking 2 single cpu misc class systems.
Tue, Jun 19, 3:53 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196666: rack/setup/add to spares tracking 2 single cpu misc class systems.
Tue, Jun 19, 3:29 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196493: rack/setup/install dns200[12].wikimedia.org.
Tue, Jun 19, 3:15 PM · Patch-For-Review, Traffic, DNS, ops-codfw, Operations
Papaul triaged T197697: switch port configuration for dns200[1-2] as Normal priority.
Tue, Jun 19, 3:13 PM · netops, Traffic, DNS, ops-codfw, Operations
Papaul updated the task description for T196493: rack/setup/install dns200[12].wikimedia.org.
Tue, Jun 19, 3:10 PM · Patch-For-Review, Traffic, DNS, ops-codfw, Operations

Mon, Jun 18

Papaul reassigned T196560: rack/setup/install LVS200[7-10] from Papaul to BBlack.

@BBlack Lvs2009 and lvs2010 are ready. For switch port information please see T196946. Once they are up, we can decommission lvs2004 in row B rack B2 so I can setup lvs2008. Once done please assign task back to me.

Mon, Jun 18, 5:17 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Mon, Jun 18, 5:13 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul updated the task description for T196666: rack/setup/add to spares tracking 2 single cpu misc class systems.
Mon, Jun 18, 4:28 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Mon, Jun 18, 4:13 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul added a comment to T196560: rack/setup/install LVS200[7-10].

I chat with @ayounsi, he confirmed that both servers were in the correct VLAN's. What i did on my end was to unplugged the other 3 NIC's form both server and leave only NIC1 and this fixed the issue.

Mon, Jun 18, 4:09 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul reassigned T197146: Disk predictive failure on db2052 from Papaul to Marostegui.

done

Mon, Jun 18, 2:53 PM · Operations, ops-codfw
Papaul reassigned T197146: Disk predictive failure on db2052 from Papaul to Marostegui.

Disk replaced

Mon, Jun 18, 2:31 PM · Operations, ops-codfw
Papaul triaged T197146: Disk predictive failure on db2052 as Normal priority.
Mon, Jun 18, 2:07 PM · Operations, ops-codfw

Fri, Jun 15

Papaul updated the task description for T196477: rack/setup/install backup2001.
Fri, Jun 15, 4:56 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated subscribers of T196477: rack/setup/install backup2001.

@MoritzMuehlenhoff we are missing in our installer network drivers for the NIC card on this system (QLogic 10GE 2P QL41112HxCU-DE Adapter )

Fri, Jun 15, 3:57 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T196560: rack/setup/install LVS200[7-10].

log on install2002 for lvs2010

Fri, Jun 15, 3:16 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul updated subscribers of T196560: rack/setup/install LVS200[7-10].

@ayounsi @BBlack I am getting the network error message below during install on both lvs2009 and lvs2010. Please advice. Thanks.

Fri, Jun 15, 3:14 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul reassigned T196665: rack/setup/install bast2002.wikimedia.org from Papaul to Dzahn.

@Dzahn all yours

Fri, Jun 15, 1:02 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196665: rack/setup/install bast2002.wikimedia.org.
Fri, Jun 15, 1:01 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196665: rack/setup/install bast2002.wikimedia.org.
Fri, Jun 15, 12:19 PM · Patch-For-Review, ops-codfw, Operations

Wed, Jun 13

Papaul updated the task description for T196666: rack/setup/add to spares tracking 2 single cpu misc class systems.
Wed, Jun 13, 5:19 PM · Patch-For-Review, ops-codfw, Operations
Papaul added a comment to T197147: Rack/Setup new codfw QFX5100 10G switch .

@ayounsi the name proposal is just temporally so i can add the switches in racktables and do the setup in the scs-a1/c1. After you are done with the configuration and we remove the old onces i will change the name back to asw-a/b/c/d.

Wed, Jun 13, 4:57 PM · netops, Operations
Papaul added a comment to T197146: Disk predictive failure on db2052.

@jcrespo disk replacement complete

Wed, Jun 13, 4:44 PM · Operations, ops-codfw
Papaul triaged T197147: Rack/Setup new codfw QFX5100 10G switch as Normal priority.
Wed, Jun 13, 4:36 PM · netops, Operations
Papaul updated the task description for T196946: switch port configuration for lvs200[7-10].
Wed, Jun 13, 4:19 PM · netops, ops-codfw, Traffic, Operations
Papaul updated the task description for T196664: rack/setup/install authdns2001.wikimedia.org.
Wed, Jun 13, 3:04 PM · ops-codfw, Operations
Papaul added a comment to T196946: switch port configuration for lvs200[7-10].

@ayounsi thanks

Wed, Jun 13, 2:13 PM · netops, ops-codfw, Traffic, Operations
Papaul reassigned T196946: switch port configuration for lvs200[7-10] from Papaul to ayounsi.

@ayounsi all fibers for lvs2010 and lvs2009 are already pulled according to the the first plan

LVS2009C2asw-c2asw-a2asw-b2asw-d2
LVS2010D2asw-d2asw-a2asw-b2asw-c2

see T196560 any reason why we have to change it ? maybe i was not clear on my request. I want for the port asw-c2 xe-2/0/44 for lvs2009 and asw-d2 xe-2/0/44 for lvs2010 to be setup up and enable on the switch side so I can star the install .

Wed, Jun 13, 1:15 PM · netops, ops-codfw, Traffic, Operations

Tue, Jun 12

Papaul updated the task description for T196493: rack/setup/install dns200[12].wikimedia.org.
Tue, Jun 12, 4:58 PM · Patch-For-Review, Traffic, DNS, ops-codfw, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Tue, Jun 12, 12:45 AM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Tue, Jun 12, 12:18 AM · Patch-For-Review, ops-codfw, Traffic, Operations

Mon, Jun 11

Papaul updated the task description for T196665: rack/setup/install bast2002.wikimedia.org.
Mon, Jun 11, 11:51 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196665: rack/setup/install bast2002.wikimedia.org.
Mon, Jun 11, 11:39 PM · Patch-For-Review, ops-codfw, Operations
Papaul triaged T196957: switch port configuration for bast2002 as Normal priority.
Mon, Jun 11, 11:35 PM · netops, ops-codfw, Operations
Papaul updated the task description for T196665: rack/setup/install bast2002.wikimedia.org.
Mon, Jun 11, 11:33 PM · Patch-For-Review, ops-codfw, Operations
Papaul placed T196946: switch port configuration for lvs200[7-10] up for grabs.
Mon, Jun 11, 11:01 PM · netops, ops-codfw, Traffic, Operations
Papaul triaged T196946: switch port configuration for lvs200[7-10] as Normal priority.
Mon, Jun 11, 8:09 PM · netops, ops-codfw, Traffic, Operations
Papaul triaged T196941: Rack/setup cr2-eqdfw as Normal priority.
Mon, Jun 11, 7:43 PM · netops, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Mon, Jun 11, 5:30 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul updated the task description for T196665: rack/setup/install bast2002.wikimedia.org.
Mon, Jun 11, 4:54 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196665: rack/setup/install bast2002.wikimedia.org.
Mon, Jun 11, 4:45 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Mon, Jun 11, 3:59 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul reassigned T196709: replace bad disk in db2059 from Papaul to Marostegui.

Disk replaced

Mon, Jun 11, 2:29 PM · DBA, Operations, ops-codfw

Fri, Jun 8

Papaul triaged T196782: Switch port configuration for backup2001 as Normal priority.
Fri, Jun 8, 8:51 PM · netops, ops-codfw, Operations
Papaul updated the task description for T196477: rack/setup/install backup2001.
Fri, Jun 8, 8:48 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196477: rack/setup/install backup2001.
Fri, Jun 8, 5:28 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196477: rack/setup/install backup2001.
Fri, Jun 8, 5:28 PM · Patch-For-Review, ops-codfw, Operations
Papaul updated the task description for T196483: rack/setup/install graphite2003.
Fri, Jun 8, 2:21 PM · ops-codfw, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Fri, Jun 8, 12:48 AM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul updated the task description for T196560: rack/setup/install LVS200[7-10].
Fri, Jun 8, 12:38 AM · Patch-For-Review, ops-codfw, Traffic, Operations

Thu, Jun 7

Papaul updated the task description for T196493: rack/setup/install dns200[12].wikimedia.org.
Thu, Jun 7, 8:06 PM · Patch-For-Review, Traffic, DNS, ops-codfw, Operations
Papaul updated the task description for T196493: rack/setup/install dns200[12].wikimedia.org.
Thu, Jun 7, 8:01 PM · Patch-For-Review, Traffic, DNS, ops-codfw, Operations
Papaul updated the task description for T196477: rack/setup/install backup2001.
Thu, Jun 7, 8:00 PM · Patch-For-Review, ops-codfw, Operations
Papaul reassigned T196339: pc2005 down from Papaul to jcrespo.

@jcrespo Dell Shipped a new main board and a new network card. I replaced first the network card to see if the network card was the problem and yes we had a bad network card. I reboot the server a couple of times to make sure we are not getting the error. The server looks good now. in case you need the MAC address of eth1 for the new card: 80:18:44:ED:98:12

Thu, Jun 7, 4:08 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul shifted T196560: rack/setup/install LVS200[7-10] from the Restricted Space space to the S1 Public space.
Thu, Jun 7, 2:54 AM · Patch-For-Review, ops-codfw, Traffic, Operations

Wed, Jun 6

Papaul reassigned T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech from Papaul to Jgreen.

@Jgreen all yours. let me know if you have any questions.

Wed, Jun 6, 9:27 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul updated the task description for T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Wed, Jun 6, 9:26 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul triaged T196560: rack/setup/install LVS200[7-10] as Normal priority.
Wed, Jun 6, 3:07 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Papaul triaged T196557: switch port configuration for frmon2001 as Normal priority.
Wed, Jun 6, 2:46 PM · netops, ops-codfw, Operations, fundraising-tech-ops

Tue, Jun 5

Papaul reassigned T196417: Rack/Setup frbast2001.frack.codfw.wmnet from Papaul to Jgreen.

@Jgreen this is done at my end. Once the switch port configuration is done it is all yours. Thanks

Tue, Jun 5, 7:57 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T196503: switch port configuration for frbast2001.
Tue, Jun 5, 7:53 PM · netops, ops-codfw, fundraising-tech-ops, Operations
Papaul created T196503: switch port configuration for frbast2001.
Tue, Jun 5, 7:53 PM · netops, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T196417: Rack/Setup frbast2001.frack.codfw.wmnet.
Tue, Jun 5, 7:46 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T196417: Rack/Setup frbast2001.frack.codfw.wmnet.
Tue, Jun 5, 7:35 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul claimed T196417: Rack/Setup frbast2001.frack.codfw.wmnet.
Tue, Jun 5, 7:35 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T196417: Rack/Setup frbast2001.frack.codfw.wmnet.
Tue, Jun 5, 6:44 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Tue, Jun 5, 6:44 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul updated the task description for T196417: Rack/Setup frbast2001.frack.codfw.wmnet.
Tue, Jun 5, 5:50 PM · Patch-For-Review, ops-codfw, fundraising-tech-ops, Operations
Papaul updated the task description for T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Tue, Jun 5, 5:49 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul renamed T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech from rack/setup/install Prometeuse/Grafana host for fr-tech to rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Tue, Jun 5, 5:48 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul updated the task description for T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Tue, Jun 5, 5:44 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul reassigned T196251: frdb2001 RAID disk failure from Papaul to Jgreen.

@Jgreen disk replacement complete

Tue, Jun 5, 4:29 PM · Operations, fundraising-tech-ops, ops-codfw
Papaul shifted T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech from the Restricted Space space to the S1 Public space.
Tue, Jun 5, 3:50 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul triaged T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech as Normal priority.
Tue, Jun 5, 3:48 PM · Patch-For-Review, ops-codfw, Operations, fundraising-tech-ops
Papaul reassigned T196246: Degraded RAID on db2047 from Papaul to Marostegui.

Disk replacement complete.

Tue, Jun 5, 3:31 PM · DBA, Operations, ops-codfw
Papaul added a comment to T196246: Degraded RAID on db2047.

db2064 thanks @jcrespo

Tue, Jun 5, 3:26 PM · DBA, Operations, ops-codfw
Papaul added a comment to T196339: pc2005 down.

@jcrespo you didn't create me more work, it is my work lol

Tue, Jun 5, 3:23 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T196246: Degraded RAID on db2047.

@Marostegui since db2067 is out of service, would you like for me to take one disk from it and replace the failed disk on db2047?

Tue, Jun 5, 3:20 PM · DBA, Operations, ops-codfw
Papaul added a comment to T196339: pc2005 down.

Good morning Papaul,

Tue, Jun 5, 3:17 PM · Patch-For-Review, DBA, Operations, ops-codfw
Papaul added a comment to T196339: pc2005 down.

Will be receiving a main board replacement .

Tue, Jun 5, 3:17 PM · Patch-For-Review, DBA, ops-codfw, Operations

Thu, May 31

Papaul closed T196045: elastic2018 not rebooting as Resolved.

@Gehel no follow up at my end

Thu, May 31, 5:30 PM · ops-codfw, Discovery-Search (Current work), DC-Ops, Operations
Papaul reassigned T196000: move/setup/install labtestnet2003(WMF6469) from Papaul to RobH.
Thu, May 31, 4:39 PM · Patch-For-Review, Cloud-VPS, Operations
Papaul updated the task description for T196000: move/setup/install labtestnet2003(WMF6469).
Thu, May 31, 4:38 PM · Patch-For-Review, Cloud-VPS, Operations
Papaul updated the task description for T196000: move/setup/install labtestnet2003(WMF6469).
Thu, May 31, 4:08 PM · Patch-For-Review, Cloud-VPS, Operations
Papaul added a comment to T196000: move/setup/install labtestnet2003(WMF6469).

moved wmf6469 from D5 to B1
new Switch port information
ge-1/0/15

Thu, May 31, 4:07 PM · Patch-For-Review, Cloud-VPS, Operations
Papaul added a comment to T196000: move/setup/install labtestnet2003(WMF6469).

We have already a hosts name labtestnet2002

Thu, May 31, 4:03 PM · Patch-For-Review, Cloud-VPS, Operations
Papaul reassigned T196045: elastic2018 not rebooting from Papaul to Gehel.

@Gehel for some reason, the server lost some settings like in the BIOS Serial Console & EMS EMS Console was COM1 , BOOT options was set to UEFI and USB options Internal SD Card Slot was enable. I changed back those settings the way it supposed to be. Power off and on the server 3 times using the ILO CLI . Server is back up

Thu, May 31, 3:00 PM · ops-codfw, Discovery-Search (Current work), DC-Ops, Operations
Papaul reassigned T196000: move/setup/install labtestnet2003(WMF6469) from Papaul to RobH.
Thu, May 31, 2:40 PM · Patch-For-Review, Cloud-VPS, Operations
Papaul claimed T196000: move/setup/install labtestnet2003(WMF6469).
Thu, May 31, 2:30 PM · Patch-For-Review, Cloud-VPS, Operations

Tue, May 29

Papaul reassigned T194835: mw2182 crash from Papaul to MoritzMuehlenhoff.

@MoritzMuehlenhoff

  • update IDRAC and BIOS
  • clean log
  • Swap CPU1 with CPU0
Tue, May 29, 3:30 PM · ops-codfw, Operations
Papaul reassigned T195626: db2059 disk on predictive failure from Papaul to Marostegui.

Another disk in place

Tue, May 29, 3:10 PM · ops-codfw, Operations, DBA
Papaul reassigned T195626: db2059 disk on predictive failure from Papaul to Marostegui.

Disk replacement complete

Tue, May 29, 2:57 PM · ops-codfw, Operations, DBA
Papaul claimed T194835: mw2182 crash.
Tue, May 29, 1:29 PM · ops-codfw, Operations
Papaul added a comment to T195306: Degraded RAID on elastic2020.

@Gehel is it okay to resolve this task?

Tue, May 29, 1:24 PM · Operations, ops-codfw
Papaul added a comment to T194835: mw2182 crash.

@MoritzMuehlenhoff yes we do have some decommissioned servers. This can be also a bad main board. What we can do first is to swap CPU position. Since the error is showing on CPU1 we can move CPU1 to CPU0 and CPU0 to CPU1 if we do have the error on CPU0 than the CPU1 is bad if we do have the error on CPU1 than we need to replace the main board.

Tue, May 29, 1:21 PM · ops-codfw, Operations

Fri, May 25

Papaul reassigned T194781: rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) from Papaul to Marostegui.

@Marostegui All done

Fri, May 25, 1:57 PM · Patch-For-Review, ops-codfw, DBA, Operations