Page MenuHomePhabricator

Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (259 w, 4 h)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Yesterday

Papaul updated the task description for T239733: codfw:rack/setup/install frdb2002.
Wed, Dec 4, 10:13 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T239733: codfw:rack/setup/install frdb2002.
apaul@fasw-c-codfw# run show interfaces ge-[0-1]/0/20 descriptions       
Interface       Admin Link Description
ge-0/0/20       up    up   frdb2002:eth0
ge-1/0/20       up    up   frdb2002:eth1
Wed, Dec 4, 9:49 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T239733: codfw:rack/setup/install frdb2002.
papaul@fasw-c-codfw# show | compare 
[edit interfaces interface-range vlan-fundraising]
     member "ge-[0-1]/0/19" { ... }
+    member "ge-[0-1]/0/20";
[edit interfaces]
+   ge-0/0/20 {
+       description frdb2002:eth0;
+   }
+   ge-1/0/20 {
+       description frdb2002:eth1;
+   }
Wed, Dec 4, 9:47 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T239733: codfw:rack/setup/install frdb2002.
Virtual Disk 0: RAID10, 5.237TB, Ready
Wed, Dec 4, 9:32 PM · Patch-For-Review, Operations, ops-codfw
Papaul moved T233185: Decommission db2067.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Wed, Dec 4, 5:45 PM · Operations, DC-Ops, ops-codfw, decommission
Papaul moved T239732: codfw: rack/setup/install puppetnaster2003.codfw.wmnet from Backlog to Racking Tasks on the ops-codfw board.
Wed, Dec 4, 5:45 PM · Operations, ops-codfw
Papaul moved T239733: codfw:rack/setup/install frdb2002 from Backlog to Racking Tasks on the ops-codfw board.
Wed, Dec 4, 5:45 PM · Patch-For-Review, Operations, ops-codfw
Papaul moved T239805: ms-fe2007 NIC failure from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Dec 4, 5:45 PM · Operations, ops-codfw
Papaul closed T239758: mw2259 down and mgmt does not exist? as Resolved.

Reset the IDRAC server is back up

Wed, Dec 4, 5:41 PM · DC-Ops, ops-codfw, serviceops, Operations
Papaul added a comment to T239805: ms-fe2007 NIC failure.

@fgiunchedi the 10G NiC is dead

Wed, Dec 4, 5:29 PM · Operations, ops-codfw
Papaul updated the task description for T238726: Decommission db2062.codfw.wmnet.
Wed, Dec 4, 3:59 PM · Operations, DC-Ops, ops-codfw, decommission
Papaul added a comment to T233185: Decommission db2067.codfw.wmnet.
[edit interfaces interface-range vlan-private1-d-codfw]
-    member ge-6/0/15;
[edit interfaces interface-range disabled]
     member ge-6/0/9 { ... }
+    member ge-6/0/15;
[edit interfaces]
-   ge-6/0/15 {
-       description db2067;
-       enable;
-   }
Wed, Dec 4, 1:59 AM · Operations, DC-Ops, ops-codfw, decommission

Tue, Dec 3

Papaul closed T239009: Degraded RAID on ganeti2002 as Resolved.

We can resolve this task for now and re-open if that drive fails

Tue, Dec 3, 6:44 PM · Operations, ops-codfw
Papaul updated the task description for T239733: codfw:rack/setup/install frdb2002.
Tue, Dec 3, 6:10 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T239733: codfw:rack/setup/install frdb2002.
Tue, Dec 3, 6:02 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T239733: codfw:rack/setup/install frdb2002.
Tue, Dec 3, 6:01 PM · Patch-For-Review, Operations, ops-codfw
Papaul added a comment to T239733: codfw:rack/setup/install frdb2002.

@Jgreen I need some information for this server.

Tue, Dec 3, 5:58 PM · Patch-For-Review, Operations, ops-codfw
Papaul triaged T239733: codfw:rack/setup/install frdb2002 as Medium priority.
Tue, Dec 3, 5:52 PM · Patch-For-Review, Operations, ops-codfw
Papaul created T239733: codfw:rack/setup/install frdb2002.
Tue, Dec 3, 5:51 PM · Patch-For-Review, Operations, ops-codfw
Papaul updated the task description for T239732: codfw: rack/setup/install puppetnaster2003.codfw.wmnet.
Tue, Dec 3, 5:44 PM · Operations, ops-codfw
Papaul triaged T239732: codfw: rack/setup/install puppetnaster2003.codfw.wmnet as Medium priority.
Tue, Dec 3, 5:39 PM · Operations, ops-codfw
Papaul created T239732: codfw: rack/setup/install puppetnaster2003.codfw.wmnet.
Tue, Dec 3, 5:38 PM · Operations, ops-codfw
Papaul added a comment to T239009: Degraded RAID on ganeti2002.

@akosiaris after removing and putting the failed disk back the disk came back online with no errors

Tue, Dec 3, 4:47 PM · Operations, ops-codfw

Mon, Dec 2

Papaul moved T239249: codfw: rack/setup/install mc203[7,8,9].codfw.wmnet from Backlog to Racking Tasks on the ops-codfw board.
Mon, Dec 2, 4:11 PM · Operations, ops-codfw
Papaul moved T238726: Decommission db2062.codfw.wmnet from Procurement to Decommission on the ops-codfw board.
Mon, Dec 2, 4:11 PM · Operations, DC-Ops, ops-codfw, decommission
Papaul moved T238726: Decommission db2062.codfw.wmnet from Backlog to Procurement on the ops-codfw board.
Mon, Dec 2, 4:10 PM · Operations, DC-Ops, ops-codfw, decommission
Papaul added a comment to T232126: Decommission old mw2231/WMF6435 replaced with WMF6403.

@Volans Thanks

Mon, Dec 2, 4:10 PM · Operations, ops-codfw
Papaul added a comment to T238726: Decommission db2062.codfw.wmnet.
papaul@asw-b-codfw# show | compare 
[edit interfaces interface-range vlan-private1-b-codfw]
-    member ge-5/0/40;
[edit interfaces interface-range disabled]
     member ge-8/0/17 { ... }
+    member ge-5/0/40;
[edit interfaces]
-   ge-5/0/40 {
-       description db2062;
-       enable;
-   }
Mon, Dec 2, 4:07 PM · Operations, DC-Ops, ops-codfw, decommission
Papaul updated the task description for T238726: Decommission db2062.codfw.wmnet.
Mon, Dec 2, 4:07 PM · Operations, DC-Ops, ops-codfw, decommission
Papaul updated the task description for T238726: Decommission db2062.codfw.wmnet.
Mon, Dec 2, 4:02 PM · Operations, DC-Ops, ops-codfw, decommission
Papaul closed T239345: scs-[a1-c1]-codfw redundancy power test as Resolved.

Resolving this task since redundancy power is working in codfw

Mon, Dec 2, 2:00 AM · ops-eqdfw, Operations, ops-codfw
Papaul closed T235125: Move kafka200[123] to logstash202[012] as Resolved.

Update Netxbox

Mon, Dec 2, 1:59 AM · DC-Ops, Operations, ops-codfw
Papaul closed T232126: Decommission old mw2231/WMF6435 replaced with WMF6403 as Resolved.

I update Netbox

Mon, Dec 2, 1:52 AM · Operations, ops-codfw
Papaul updated subscribers of T239009: Degraded RAID on ganeti2002.

@akosiaris please see @wiki_willy comment above.

Mon, Dec 2, 1:46 AM · Operations, ops-codfw

Sat, Nov 30

Papaul added a comment to T232126: Decommission old mw2231/WMF6435 replaced with WMF6403.

@Volans confirm that 10.193.1.118 is indeed the mgmt IP address for mw2231 and 10.193.2.251 is no longer in use.

Sat, Nov 30, 7:09 PM · Operations, ops-codfw
Papaul added a comment to T232126: Decommission old mw2231/WMF6435 replaced with WMF6403.

@Volans do we have a template in place when a server name changes from X to Y to also update DNS in netbox. The last steps that I recalled were
1- change switch port description
2- change physical label

Sat, Nov 30, 6:52 PM · Operations, ops-codfw
Papaul closed T221068: decom ms-be201[345] as Resolved.
Sat, Nov 30, 6:44 PM · decommission, ops-codfw, SRE-swift-storage, User-fgiunchedi, Operations
Papaul closed T238526: Decommission db2061.codfw.wmnet, a subtask of T228258: Decommission db2043-db2070, as Resolved.
Sat, Nov 30, 6:40 PM · Operations, DBA
Papaul closed T238526: Decommission db2061.codfw.wmnet as Resolved.
Sat, Nov 30, 6:40 PM · Operations, ops-codfw, decommission

Wed, Nov 27

Papaul added a comment to T239345: scs-[a1-c1]-codfw redundancy power test.

Looks like redundancy power is working on both scs's in codfw. What thing to do in esams is to try to change the outlet in which the scs is plugged in on ps2-oe16-esams and ps1-oe16-esams. we did have the same power issue on the mr1 while we were there.

Wed, Nov 27, 6:38 PM · ops-eqdfw, Operations, ops-codfw
Papaul updated the task description for T239249: codfw: rack/setup/install mc203[7,8,9].codfw.wmnet.
Wed, Nov 27, 4:35 PM · Operations, ops-codfw
Papaul triaged T239345: scs-[a1-c1]-codfw redundancy power test as Medium priority.
Wed, Nov 27, 4:31 PM · ops-eqdfw, Operations, ops-codfw
RobH awarded T239345: scs-[a1-c1]-codfw redundancy power test a Like token.
Wed, Nov 27, 4:06 PM · ops-eqdfw, Operations, ops-codfw
Papaul renamed T239345: scs-[a1-c1]-codfw redundancy power test from scs-[a1-c8]-codfw redundancy power test to scs-[a1-c1]-codfw redundancy power test.
Wed, Nov 27, 3:58 PM · ops-eqdfw, Operations, ops-codfw
Papaul added a comment to T239345: scs-[a1-c1]-codfw redundancy power test.

Test on scs-a1-codfw
Test1: turn off power switch on psu1 and turn on power switch on psu2
device is still on
Test2: turn off power switch on psu2 and turn on power switch on psu1
device is still

Wed, Nov 27, 3:58 PM · ops-eqdfw, Operations, ops-codfw
Papaul created T239345: scs-[a1-c1]-codfw redundancy power test.
Wed, Nov 27, 3:38 PM · ops-eqdfw, Operations, ops-codfw

Tue, Nov 26

Papaul added a comment to T184066: rack/setup/install ps[12]-oe1[456]-esams.

qfx5100-spare1, psu 0 {#20156} to ps2-oe15-esams:17
qfx5100-spare2, psu 0 {#20157} to ps2-oe15-esams:16
qfx5100-spare1, psu 1 {#20159} to ps1-oe15-esams:2
qfx5100-spare2, psu 1 {#20158} to ps1-oe15-esams:3

Tue, Nov 26, 5:36 PM · Operations, ops-esams
Papaul triaged T239249: codfw: rack/setup/install mc203[7,8,9].codfw.wmnet as Medium priority.
Tue, Nov 26, 5:15 PM · Operations, ops-codfw
Papaul created T239249: codfw: rack/setup/install mc203[7,8,9].codfw.wmnet.
Tue, Nov 26, 5:14 PM · Operations, ops-codfw

Mon, Nov 25

Papaul reassigned T239042: db2125 crashed from Papaul to Marostegui.

complete
Before
BIOS Version 2.2.11
iDRAC Firmware Version 3.34.34.34

Mon, Nov 25, 5:20 PM · ops-codfw, Operations, DBA

Fri, Nov 22

Papaul closed T237913: Decommission db2048.codfw.wmnet, a subtask of T228258: Decommission db2043-db2070, as Resolved.
Fri, Nov 22, 11:19 PM · Operations, DBA
Papaul closed T237913: Decommission db2048.codfw.wmnet as Resolved.

Complete

Fri, Nov 22, 11:19 PM · Patch-For-Review, ops-codfw, decommission, Operations
Papaul updated the task description for T237913: Decommission db2048.codfw.wmnet.
Fri, Nov 22, 11:19 PM · Patch-For-Review, ops-codfw, decommission, Operations
Papaul closed T238526: Decommission db2061.codfw.wmnet, a subtask of T228258: Decommission db2043-db2070, as Resolved.
Fri, Nov 22, 11:18 PM · Operations, DBA
Papaul closed T238526: Decommission db2061.codfw.wmnet as Resolved.

Complete

Fri, Nov 22, 11:18 PM · Operations, ops-codfw, decommission
Papaul updated the task description for T238526: Decommission db2061.codfw.wmnet.
Fri, Nov 22, 11:18 PM · Operations, ops-codfw, decommission
Papaul closed T238233: decommission alnilam.frack.codfw.wmnet as Resolved.

Complete

Fri, Nov 22, 4:25 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Papaul updated the task description for T238233: decommission alnilam.frack.codfw.wmnet.
Fri, Nov 22, 4:25 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Papaul placed T236216: rack/setup/install ganeti300[123] up for grabs.
Fri, Nov 22, 4:26 AM · Operations, ops-esams
Papaul updated the task description for T238526: Decommission db2061.codfw.wmnet.
Fri, Nov 22, 1:58 AM · Operations, ops-codfw, decommission

Wed, Nov 20

Papaul updated the task description for T238233: decommission alnilam.frack.codfw.wmnet.
Wed, Nov 20, 8:13 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Papaul updated the task description for T237913: Decommission db2048.codfw.wmnet.
Wed, Nov 20, 8:06 PM · Patch-For-Review, ops-codfw, decommission, Operations
Papaul added a comment to T238526: Decommission db2061.codfw.wmnet.
papaul@asw-d-codfw# show | compare 
[edit interfaces interface-range vlan-private1-d-codfw]
-    member ge-6/0/9;
[edit interfaces interface-range disabled]
     member ge-6/0/1 { ... }
+    member ge-6/0/9;
[edit interfaces]
-   ge-6/0/9 {
-       description db2061;
-       enable;
-   }
Wed, Nov 20, 3:50 PM · Operations, ops-codfw, decommission
Papaul closed T238642: Update label and switch port for wezen -> centrallog2001, a subtask of T224564: Reimage wezen to Buster (and rename to centrallog2001), as Resolved.
Wed, Nov 20, 3:47 PM · User-fgiunchedi, observability, Operations
Papaul closed T238642: Update label and switch port for wezen -> centrallog2001 as Resolved.

complete

Wed, Nov 20, 3:47 PM · ops-codfw, User-fgiunchedi, observability, Operations
Papaul added a comment to T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.

You welcome

Wed, Nov 20, 3:56 AM · Cloud-Services, Operations, ops-codfw

Tue, Nov 19

Papaul added a comment to T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.

https://netbox.wikimedia.org/dcim/devices/2147/
https://netbox.wikimedia.org/dcim/devices/2148/

Tue, Nov 19, 11:25 PM · Cloud-Services, Operations, ops-codfw
Papaul added a comment to T238642: Update label and switch port for wezen -> centrallog2001.
papaul@asw-d-codfw# show | compare 
[edit interfaces ge-5/0/13]
-   description wezen;
+   description centrallog2001;
Tue, Nov 19, 3:26 PM · ops-codfw, User-fgiunchedi, observability, Operations
Papaul triaged T238642: Update label and switch port for wezen -> centrallog2001 as Medium priority.
Tue, Nov 19, 3:24 PM · ops-codfw, User-fgiunchedi, observability, Operations
Papaul added a comment to T224528: rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet.

@Andrew this task has been resolved but both cloudbackup2001 and 2002 are showing "staged" in Netbox for status and both array are showing "planned" too.

Tue, Nov 19, 3:49 AM · Cloud-Services, Operations, ops-codfw

Thu, Nov 14

Papaul added a comment to T238233: decommission alnilam.frack.codfw.wmnet.
papaul@fasw-c-codfw# show | compare 
[edit interfaces interface-range disabled]
     member "ge-[0-1]/0/8" { ... }
+    member "ge-[0-1]/0/9";
[edit interfaces interface-range vlan-fundraising]
-    member "ge-[0-1]/0/9";
[edit interfaces]
-   ge-0/0/9 {
-       description alnilam;
-   }
-   ge-1/0/9 {
-       description alnilam;
-   }
Thu, Nov 14, 5:19 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Papaul updated the task description for T237913: Decommission db2048.codfw.wmnet.
Thu, Nov 14, 5:09 PM · Patch-For-Review, ops-codfw, decommission, Operations
Papaul added a comment to T237913: Decommission db2048.codfw.wmnet.
papaul@asw-a-codfw# show | compare 
[edit interfaces interface-range vlan-private1-a-codfw]
-    member ge-1/0/0;
[edit interfaces interface-range disabled]
     member ge-6/0/16 { ... }
+    member ge-1/0/0;
[edit interfaces]
-   ge-1/0/0 {
-       description db2048;
-   }
Thu, Nov 14, 5:09 PM · Patch-For-Review, ops-codfw, decommission, Operations

Wed, Nov 13

Papaul reassigned T237905: Upgrade db2072 firmware and bios from Papaul to Marostegui.

Before
BIOS Version
2.4.3
Firmware Version
2.40

Wed, Nov 13, 4:43 PM · Operations, ops-codfw, DBA
Papaul added a comment to T237702: (codfw):rack/setup/install db213[2-5].

@Marostegui @jcrespo please fell free to take over the task.

Wed, Nov 13, 12:38 AM · Operations, DBA, ops-codfw
Papaul updated the task description for T237702: (codfw):rack/setup/install db213[2-5].
Wed, Nov 13, 12:37 AM · Operations, DBA, ops-codfw
Papaul updated the task description for T237702: (codfw):rack/setup/install db213[2-5].
Wed, Nov 13, 12:04 AM · Operations, DBA, ops-codfw

Tue, Nov 12

Papaul moved T237913: Decommission db2048.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Tue, Nov 12, 11:41 PM · Patch-For-Review, ops-codfw, decommission, Operations
Papaul updated the task description for T237702: (codfw):rack/setup/install db213[2-5].
Tue, Nov 12, 10:35 PM · Operations, DBA, ops-codfw
Papaul added a comment to T237702: (codfw):rack/setup/install db213[2-5].
papaul@asw-a-codfw# show | compare 
[edit interfaces interface-range vlan-private1-a-codfw]
     member xe-7/0/9 { ... }
+    member ge-5/0/1;
[edit interfaces interface-range disabled]
-    member ge-5/0/1;
[edit interfaces]
+   ge-5/0/1 {
+       description db2132;
+   }
Tue, Nov 12, 10:32 PM · Operations, DBA, ops-codfw
Papaul updated the task description for T237702: (codfw):rack/setup/install db213[2-5].
Tue, Nov 12, 5:43 PM · Operations, DBA, ops-codfw
Papaul updated the task description for T237702: (codfw):rack/setup/install db213[2-5].
Tue, Nov 12, 4:21 PM · Operations, DBA, ops-codfw

Fri, Nov 8

Papaul closed T209738: decom einsteinium, a subtask of T202782: upgrade icinga server to stretch and replace einsteinium, as Resolved.
Fri, Nov 8, 7:45 PM · Patch-For-Review, observability, Operations
Papaul closed T209738: decom einsteinium as Resolved.
Fri, Nov 8, 7:45 PM · ops-eqiad, decommission, Operations
Papaul closed Unknown Object (Task), a subtask of T220170: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment, as Resolved.
Fri, Nov 8, 7:15 PM · Goal, DBA
Papaul closed T233289: Decommission ms-be1027 as Resolved.

Complete

Fri, Nov 8, 7:12 PM · decommission, Operations, ops-eqiad
Papaul updated the task description for T233289: Decommission ms-be1027.
Fri, Nov 8, 7:10 PM · decommission, Operations, ops-eqiad
Papaul closed T216491: Decommission dbstore1002 as Resolved.

Complete

Fri, Nov 8, 7:02 PM · Patch-For-Review, decommission, ops-eqiad, Operations, Analytics
Papaul updated the task description for T216491: Decommission dbstore1002.
Fri, Nov 8, 7:02 PM · Patch-For-Review, decommission, ops-eqiad, Operations, Analytics
Papaul closed T206524: Decommission analytics1003 as Resolved.

Complete

Fri, Nov 8, 6:57 PM · ops-eqiad, Operations, decommission, DC-Ops, Analytics
Papaul closed T206524: Decommission analytics1003, a subtask of T205509: Replace the Analytics Hadoop coordinator - Hive/Oozie/etc... (hardware refresh), as Resolved.
Fri, Nov 8, 6:57 PM · User-Elukey, Analytics-Kanban, Analytics
Papaul updated the task description for T206524: Decommission analytics1003.
Fri, Nov 8, 6:57 PM · ops-eqiad, Operations, decommission, DC-Ops, Analytics
Papaul added a comment to T206524: Decommission analytics1003.
papaul@asw2-c-eqiad# show | compare 
[edit interfaces]
-   ge-4/0/18 {
-       description "analytics1003 - no-bw-mon";
-   }
Fri, Nov 8, 6:01 PM · ops-eqiad, Operations, decommission, DC-Ops, Analytics
Papaul updated the task description for T233289: Decommission ms-be1027.
Fri, Nov 8, 5:49 PM · decommission, Operations, ops-eqiad
Papaul updated the task description for T233289: Decommission ms-be1027.
Fri, Nov 8, 5:49 PM · decommission, Operations, ops-eqiad
Papaul added a comment to T233289: Decommission ms-be1027.
papaul@asw2-d-eqiad# show | compare 
[edit interfaces]
-   xe-7/0/13 {
-       description ms-be1027;
-       enable;
-   }
Fri, Nov 8, 5:46 PM · decommission, Operations, ops-eqiad
Papaul updated the task description for T209738: decom einsteinium.
Fri, Nov 8, 5:44 PM · ops-eqiad, decommission, Operations
Papaul added a comment to T209738: decom einsteinium.
papaul@asw2-d-eqiad# show | compare 
[edit interfaces]
-   ge-3/0/7 {
-       description einsteinium;
-       enable;
-   }
Fri, Nov 8, 5:43 PM · ops-eqiad, decommission, Operations
Papaul updated the task description for T216491: Decommission dbstore1002.
Fri, Nov 8, 5:41 PM · Patch-For-Review, decommission, ops-eqiad, Operations, Analytics
Papaul added a comment to T216491: Decommission dbstore1002.
papaul@asw2-d-eqiad# show | compare 
[edit interfaces]
-   ge-1/0/7 {
-       description dbstore1002;
-       enable;
-   }
Fri, Nov 8, 5:40 PM · Patch-For-Review, decommission, ops-eqiad, Operations, Analytics