Page MenuHomePhabricator

Papaul (Papaul)
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Dec 18 2014, 3:39 PM (270 w, 3 d)
Availability
Available
LDAP User
Papaul
MediaWiki User
Unknown

Recent Activity

Sat, Feb 22

Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
Servers Rack D3Ready for service
mw2366Yes
mw2367Yes
mw2368Yes
mw2369Yes
mw2370Yes
mw2371Yes
mw2372Yes
mw2373Yes
mw2374Yes
mw2375Yes
mw2376Yes
Sat, Feb 22, 1:11 AM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul updated the task description for T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
Sat, Feb 22, 12:59 AM · ops-codfw, cloud-services-team (Hardware), Operations, DC-Ops, decommission

Fri, Feb 21

Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Fri, Feb 21, 7:35 PM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul added a comment to T245192: Investigate Juniper storm control.

Reading from librenms on ms1-eqiad

Fri, Feb 21, 4:26 AM · Operations, Wikimedia-Incident, netops

Thu, Feb 20

Papaul updated the task description for T244783: (no date provided) rack/setup/install ganeti20[19-24].
Thu, Feb 20, 8:18 PM · ops-codfw, Operations, DC-Ops
Papaul reassigned T245566: (ASAP) rack/setup/install frdb2001 from Papaul to Jgreen.

@Jgreen All yours

Thu, Feb 20, 8:16 PM · fundraising-tech-ops, Patch-For-Review, Operations, ops-codfw, DC-Ops
Papaul closed T245714: es2022 power supply issues as Resolved.

A reboot fixed the problem

Thu, Feb 20, 3:55 PM · ops-codfw, Operations, DBA
Papaul closed T245714: es2022 power supply issues, a subtask of T243052: Productionize es1020-es1025, es2020-es2025, as Resolved.
Thu, Feb 20, 3:55 PM · DBA
Papaul closed T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted as Resolved.

We replaced the PDU, all good

Thu, Feb 20, 3:25 AM · Operations, ops-codfw
Papaul updated the task description for T245566: (ASAP) rack/setup/install frdb2001.
Thu, Feb 20, 3:15 AM · fundraising-tech-ops, Patch-For-Review, Operations, ops-codfw, DC-Ops
Papaul added a comment to T245566: (ASAP) rack/setup/install frdb2001.
papaul@fasw-c-codfw# show | compare
[edit interfaces interface-range vlan-fundraising]
     member "ge-[0-1]/0/20" { ... }
+    member "ge-[0-1]/0/23";
[edit interfaces interface-range disabled]
-    member ge-0/0/23;
-    member ge-1/0/23;
[edit interfaces]
+   ge-0/0/23 {
+       description frdb2001:eth0;
+   }
+   ge-1/0/23 {
+       description frdb2001:eth1;
+   }
Thu, Feb 20, 3:13 AM · fundraising-tech-ops, Patch-For-Review, Operations, ops-codfw, DC-Ops
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Thu, Feb 20, 3:00 AM · fundraising-tech-ops, ops-codfw, Operations
Papaul added a comment to T244169: codfw: rack/setup/install 3 new payments server for frack.
papaul@fasw-c-codfw# show | compare
[edit interfaces interface-range disabled]
-    member "ge-[0-1]/0/8";
-    member "ge-[0-1]/0/9";
-    member "ge-[0-1]/0/10";
[edit interfaces interface-range vlan-payments]
     member "ge-[0-1]/0/4" { ... }
+    member "ge-[0-1]/0/8";
+    member "ge-[0-1]/0/9";
+    member "ge-[0-1]/0/10";
[edit interfaces]
+   ge-0/0/8 {
+       description payments2001-a:eth0;
+   }
+   ge-0/0/9 {
+       description payments2002-a:eth0;
+   }
+   ge-0/0/10 {
+       description payments2003-a:eth0;
+   }
+   ge-1/0/8 {
+       description payments2001-a:eth1;
+   }
+   ge-1/0/9 {
+       description payments2002-a:eth1;
+   }
+   ge-1/0/10 {
+       description payments2003-a:eth1;
+   }
Thu, Feb 20, 2:59 AM · fundraising-tech-ops, ops-codfw, Operations

Wed, Feb 19

Papaul updated the task description for T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
Wed, Feb 19, 9:58 PM · ops-codfw, cloud-services-team (Hardware), Operations, DC-Ops, decommission
Papaul updated the task description for T243329: decommission labstore2001.codfw.wmnet and labstore2002.codfw.wmnet.
Wed, Feb 19, 9:57 PM · ops-codfw, Operations, DC-Ops, decommission
Papaul closed T244196: codfw: Delete cloud interface-range as Resolved.

Complete

Wed, Feb 19, 9:57 PM · Operations, ops-codfw, netops
Papaul updated the task description for T244950: codfw:fundraising single-cpu misc servers.
Wed, Feb 19, 6:46 PM · Operations, fundraising-tech-ops, ops-codfw
Papaul updated the task description for T245566: (ASAP) rack/setup/install frdb2001.
Wed, Feb 19, 6:33 PM · fundraising-tech-ops, Patch-For-Review, Operations, ops-codfw, DC-Ops
Papaul updated the task description for T245566: (ASAP) rack/setup/install frdb2001.
Wed, Feb 19, 4:10 PM · fundraising-tech-ops, Patch-For-Review, Operations, ops-codfw, DC-Ops
Papaul closed Unknown Object (Task), a subtask of T238395: Fundraising Analytics Infrastructure and Setup, as Resolved.
Wed, Feb 19, 4:10 PM · fundraising-tech-ops, Fundraising-Backlog
Papaul closed Unknown Object (Task), a subtask of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, as Resolved.
Wed, Feb 19, 4:10 PM · fundraising-tech-ops
Papaul added a comment to T244196: codfw: Delete cloud interface-range.

@ayounsi since i deleted the interface range do you want me to delete also the VLAN cloud-support1-b-codfw

Wed, Feb 19, 2:45 AM · Operations, ops-codfw, netops
Papaul added a comment to T244196: codfw: Delete cloud interface-range.
papaul@asw-b-codfw# show | compare
[edit interfaces]
-   interface-range vlan-cloud-support1-b-codfw {
-       member ge-8/0/9;
-       mtu 9192;
-       unit 0 {
-           family ethernet-switching {
-               interface-mode access;
-               vlan {
-                   members cloud-support1-b-codfw;
-               }
-           }
-       }
-   }
Wed, Feb 19, 2:41 AM · Operations, ops-codfw, netops
Papaul closed T244743: decommission bellatrix.frack.codfw.wmnet as Resolved.

complete

Wed, Feb 19, 2:32 AM · Operations, DC-Ops, decommission
Papaul updated the task description for T244743: decommission bellatrix.frack.codfw.wmnet.
Wed, Feb 19, 2:32 AM · Operations, DC-Ops, decommission
Papaul moved T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Feb 19, 1:42 AM · Operations, ops-codfw
Papaul moved T245566: (ASAP) rack/setup/install frdb2001 from Backlog to Racking Tasks on the ops-codfw board.
Wed, Feb 19, 1:42 AM · fundraising-tech-ops, Patch-For-Review, Operations, ops-codfw, DC-Ops
Papaul reassigned T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet from Papaul to Gehel.

@Gehel All yours let me know if you have any questions.

Wed, Feb 19, 1:40 AM · Discovery-Search (Current work), Operations, ops-codfw
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Wed, Feb 19, 1:40 AM · Discovery-Search (Current work), Operations, ops-codfw
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Wed, Feb 19, 1:08 AM · Discovery-Search (Current work), Operations, ops-codfw
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Wed, Feb 19, 12:03 AM · Discovery-Search (Current work), Operations, ops-codfw

Tue, Feb 18

Papaul added a comment to T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted.

I open a request ticket (TICKET NO.1578279) with CY1 to assistance me on unplugging the old PDU and plugging the new one tomorrow the 19th at 10:30 Dallas time

Tue, Feb 18, 10:04 PM · Operations, ops-codfw
Papaul claimed T245192: Investigate Juniper storm control.
Tue, Feb 18, 8:24 PM · Operations, Wikimedia-Incident, netops
Papaul assigned T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted to wiki_willy.

@wiki_willy ps1-a8 is not stable to still in production. I notice that there is a clicking noise coming from the PDU and the readings are not stable it keeps flapping.

Tue, Feb 18, 6:34 PM · Operations, ops-codfw
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Tue, Feb 18, 6:15 PM · Discovery-Search (Current work), Operations, ops-codfw
Papaul closed T245458: asw-a-codfw:FPC8 PEM0 flapping as Resolved.

PEM 0 has no problem the problem is from PS1-a8 where PEM 0 is plug into. maybe problem from last week upgrade i have no readings on PS1-a8 right now.
closing this task

Tue, Feb 18, 4:29 PM · ops-codfw, Operations

Fri, Feb 14

Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Fri, Feb 14, 12:01 AM · Discovery-Search (Current work), Operations, ops-codfw

Thu, Feb 13

Papaul renamed T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet from (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org to (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Thu, Feb 13, 11:50 PM · Discovery-Search (Current work), Operations, ops-codfw
Papaul added a comment to T243715: elastic2043 has hardware errors that trigger reboots.

BIOS Version 1.5.6
iDRAC Firmware Version 3.21.21.21

Thu, Feb 13, 5:16 PM · Discovery, Operations, ops-codfw
Papaul added a comment to T243715: elastic2043 has hardware errors that trigger reboots.

"i have a case open with Dell. See below for case reference. According to Dell like always they will not give you a solution to the problem like they don't know also what the issue is keep trying possible solutions like upgrade BIOS and IDRAC.

Thu, Feb 13, 4:21 PM · Discovery, Operations, ops-codfw
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
servers Rack A3OSready for serviceservers Rack A6OSready for service
mw2291Stretchyesmw2301Stretchyes
mw2292Stretchyesmw2302Stretchyes
mw2293Stretchyesmw2303Stretchyes
mw2294Stretchyesmw2304Stretchyes
mw2295Stretchyesmw2305Stretchyes
mw2296Stretchyesmw2306Stretchyes
mw2297Stretchyesmw2307Stretchyes
mw2298Stretchyesmw2308Stretchyes
mw2299Stretchyesmw2309Stretchyes
mw2300Stretchyes
Thu, Feb 13, 3:56 AM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Thu, Feb 13, 12:18 AM · Patch-For-Review, ops-codfw, serviceops, Operations

Wed, Feb 12

Papaul updated the task description for T243112: codfw: rack/setup/install parse200[1-20].codfw.wmnet.
Wed, Feb 12, 7:32 PM · Operations, ops-codfw
Papaul triaged T244950: codfw:fundraising single-cpu misc servers as Medium priority.
Wed, Feb 12, 6:28 PM · Operations, fundraising-tech-ops, ops-codfw
Papaul moved T244196: codfw: Delete cloud interface-range from Backlog to Non-Urgent on the ops-codfw board.
Wed, Feb 12, 5:45 PM · Operations, ops-codfw, netops
Papaul moved T244950: codfw:fundraising single-cpu misc servers from Backlog to Racking Tasks on the ops-codfw board.
Wed, Feb 12, 3:20 AM · Operations, fundraising-tech-ops, ops-codfw
Papaul reassigned T244950: codfw:fundraising single-cpu misc servers from Papaul to Jgreen.

@Jgreen Please rename the name of each server and assign back to me.
Thanks.

Wed, Feb 12, 3:19 AM · Operations, fundraising-tech-ops, ops-codfw
Papaul created T244950: codfw:fundraising single-cpu misc servers.
Wed, Feb 12, 3:19 AM · Operations, fundraising-tech-ops, ops-codfw
Papaul updated the task description for T244743: decommission bellatrix.frack.codfw.wmnet.
Wed, Feb 12, 3:03 AM · Operations, DC-Ops, decommission

Tue, Feb 11

Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Tue, Feb 11, 5:42 PM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Tue, Feb 11, 4:50 PM · fundraising-tech-ops, ops-codfw, Operations
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Tue, Feb 11, 4:47 PM · fundraising-tech-ops, ops-codfw, Operations
Papaul renamed T244783: (no date provided) rack/setup/install ganeti20[19-24] from (no date provided) rack/setup/install ganeti20[16-21] to (no date provided) rack/setup/install ganeti20[19-24].
Tue, Feb 11, 1:42 AM · ops-codfw, Operations, DC-Ops

Mon, Feb 10

Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Mon, Feb 10, 8:51 PM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul reassigned T244783: (no date provided) rack/setup/install ganeti20[19-24] from Papaul to RobH.

@RobH we have already ganeti2009-ganeti2014 in codfw

Mon, Feb 10, 8:02 PM · ops-codfw, Operations, DC-Ops
Papaul updated the task description for T244743: decommission bellatrix.frack.codfw.wmnet.
Mon, Feb 10, 6:56 PM · Operations, DC-Ops, decommission
Papaul added a comment to T244743: decommission bellatrix.frack.codfw.wmnet.
papaul@fasw-c-codfw# show | compare
[edit interfaces interface-range disabled]
     member "ge-[0-1]/0/12" { ... }
+    member "ge-[0-1]/0/7";
[edit interfaces interface-range vlan-administration]
-    member "ge-[0-1]/0/7";
[edit interfaces]
-   ge-0/0/7 {
-       description bellatrix;
-   }
-   ge-1/0/7 {
-       description bellatrix;
-   }
Mon, Feb 10, 6:55 PM · Operations, DC-Ops, decommission
Papaul added a comment to T244743: decommission bellatrix.frack.codfw.wmnet.

@Jgreen haha i was waiting for the task thanks

Mon, Feb 10, 4:43 PM · Operations, DC-Ops, decommission

Thu, Feb 6

Papaul reassigned T242301: codfw: rack/setup/install wdqs200[7-8].codfw.wmnet from Papaul to Gehel.

@Gehel All yours

Thu, Feb 6, 7:32 PM · ops-codfw, Operations
Papaul updated the task description for T242301: codfw: rack/setup/install wdqs200[7-8].codfw.wmnet.
Thu, Feb 6, 7:31 PM · ops-codfw, Operations
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Thu, Feb 6, 6:54 PM · fundraising-tech-ops, ops-codfw, Operations
Papaul added a comment to T244438: codfw: new mw servers not getting an IP when default to Stretch.

Did the re-install on mw2311 it works . Thanks

Thu, Feb 6, 3:47 PM · serviceops-radar, Operations, ops-codfw
Papaul triaged T244438: codfw: new mw servers not getting an IP when default to Stretch as Medium priority.
Thu, Feb 6, 1:23 AM · serviceops-radar, Operations, ops-codfw
Papaul created T244438: codfw: new mw servers not getting an IP when default to Stretch.
Thu, Feb 6, 1:23 AM · serviceops-radar, Operations, ops-codfw

Wed, Feb 5

Papaul updated subscribers of T241852: rack/setup/install new codfw mw systems.

All the servers on the table above are running Buster I had a chat with @MoritzMuehlenhoff and he mentioned that we need to install Stretch on those servers so I have to update the dhcp file and re-image those servers

Wed, Feb 5, 4:07 PM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul added a comment to T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.

Talked to @Gehel on IRC those servers will be in the private VLAN and not in the public VLAN with Stretch as OS.

Wed, Feb 5, 3:56 PM · Discovery-Search (Current work), Operations, ops-codfw
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
serversready for service
mw2310yes
mw2311yes
mw2312yes
mw2313yes
mw2314yes
mw2315yes
mw2316yes
mw2317yes
mw2318yes
mw2319yes
mw2320yes
mw2321yes
mw2322yes
mw2323yes
mw2324yes
Wed, Feb 5, 3:17 AM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Wed, Feb 5, 12:47 AM · Patch-For-Review, ops-codfw, serviceops, Operations

Tue, Feb 4

Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Tue, Feb 4, 7:33 PM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul added a comment to T244196: codfw: Delete cloud interface-range.

okay thanks will do that

Tue, Feb 4, 4:10 PM · Operations, ops-codfw, netops
Papaul closed T243715: elastic2043 has hardware errors that trigger reboots as Resolved.

Upgrade CPLD to version 1.0.10 no more errors. system is back up running

Tue, Feb 4, 5:07 AM · Discovery, Operations, ops-codfw
Papaul moved T243715: elastic2043 has hardware errors that trigger reboots from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Tue, Feb 4, 4:31 AM · Discovery, Operations, ops-codfw
Papaul claimed T243715: elastic2043 has hardware errors that trigger reboots.
Tue, Feb 4, 4:31 AM · Discovery, Operations, ops-codfw
Papaul closed T243646: ms-be2030.mgmt is down as Resolved.

This is has been fixed

Tue, Feb 4, 4:28 AM · ops-codfw, Operations
Papaul closed T205712: (OoW) wtp2020: correctable memory errors as Resolved.

Closing this task since we haven;t seen any errors since November

Tue, Feb 4, 4:25 AM · Operations, ops-codfw
Papaul moved T244169: codfw: rack/setup/install 3 new payments server for frack from Backlog to Racking Tasks on the ops-codfw board.
Tue, Feb 4, 4:23 AM · fundraising-tech-ops, ops-codfw, Operations
Papaul closed T242990: decommission alnitak.frack.codfw.wmnet as Resolved.

complete

Tue, Feb 4, 4:23 AM · ops-codfw, Operations, DC-Ops, decommission
Papaul closed T242990: decommission alnitak.frack.codfw.wmnet, a subtask of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, as Resolved.
Tue, Feb 4, 4:23 AM · fundraising-tech-ops
Papaul updated the task description for T242990: decommission alnitak.frack.codfw.wmnet.
Tue, Feb 4, 4:22 AM · ops-codfw, Operations, DC-Ops, decommission
Papaul closed T242983: decommission frdb2001.frack.codfw.wmnet as Resolved.

complete

Tue, Feb 4, 4:22 AM · ops-codfw, Operations, DC-Ops, decommission
Papaul closed T242983: decommission frdb2001.frack.codfw.wmnet, a subtask of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, as Resolved.
Tue, Feb 4, 4:22 AM · fundraising-tech-ops
Papaul updated the task description for T242983: decommission frdb2001.frack.codfw.wmnet.
Tue, Feb 4, 4:22 AM · ops-codfw, Operations, DC-Ops, decommission
Papaul updated the task description for T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
Tue, Feb 4, 4:01 AM · ops-codfw, cloud-services-team (Hardware), Operations, DC-Ops, decommission
Papaul added a comment to T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.

I did not remove ge-8/0/9 from interface-range vlan-cloud-support1-b-codfw see T244196

Tue, Feb 4, 4:00 AM · ops-codfw, cloud-services-team (Hardware), Operations, DC-Ops, decommission
Papaul created T244196: codfw: Delete cloud interface-range.
Tue, Feb 4, 3:58 AM · Operations, ops-codfw, netops
Papaul added a comment to T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
papaul@asw-b-codfw# show | compare 
[edit interfaces interface-range vlan-cloud-support1-b-codfw]
-    member ge-8/0/1;
-    member ge-8/0/9;
[edit interfaces interface-range disabled]
     member ge-1/0/1 { ... }
+    member ge-8/0/1;
+    member ge-8/0/9;
[edit interfaces]
-   ge-8/0/1 {
-       description labstore2004;
-       enable;
-   }
-   ge-8/0/9 {
-       description labstore2003;
-       enable;
-   }
Tue, Feb 4, 3:29 AM · ops-codfw, cloud-services-team (Hardware), Operations, DC-Ops, decommission
Papaul updated the task description for T243329: decommission labstore2001.codfw.wmnet and labstore2002.codfw.wmnet.
Tue, Feb 4, 3:27 AM · ops-codfw, Operations, DC-Ops, decommission
Papaul added a comment to T243329: decommission labstore2001.codfw.wmnet and labstore2002.codfw.wmnet.
apaul@asw-b-codfw# show | compare 
[edit interfaces interface-range vlan-cloud-support1-b-codfw]
-    member-range ge-1/0/0 to ge-1/0/1;
[edit interfaces interface-range disabled]
     member ge-5/0/40 { ... }
+    member ge-1/0/0;
+    member ge-1/0/1;
[edit interfaces]
-   ge-1/0/0 {
-       description labstore2001;
-       enable;
-   }
-   ge-1/0/1 {
-       description labstore2002;
-       enable;
-   }
Tue, Feb 4, 3:26 AM · ops-codfw, Operations, DC-Ops, decommission

Mon, Feb 3

Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Mon, Feb 3, 8:28 PM · fundraising-tech-ops, Operations, ops-codfw
Papaul triaged T244169: codfw: rack/setup/install 3 new payments server for frack as Medium priority.
Mon, Feb 3, 8:08 PM · fundraising-tech-ops, Operations, ops-codfw
Papaul created T244169: codfw: rack/setup/install 3 new payments server for frack.
Mon, Feb 3, 8:08 PM · fundraising-tech-ops, Operations, ops-codfw

Thu, Jan 30

Papaul closed T240876: hw troubleshooting: hardware RAID predictive failure for bellatrix.frack.codfw.wmnet as Declined.

No need for this task since we have a replacing server setup on T237440

Thu, Jan 30, 7:16 PM · ops-codfw, Operations, fundraising-tech-ops, DC-Ops
Papaul closed T222464: PDUs with Infeed < 0.5Amps as Resolved.

I think we don't need alerting when the readings are low since some racks in codfw are not fully populated. For example rack d8 has only 7 servers.
Let us only alert when the values are high. Thanks
Resolving this task

Thu, Jan 30, 7:12 PM · Operations, ops-codfw
Papaul closed T243936: labtestpuppetmaster2001 showing Failed in Netbox as Resolved.

Done

Thu, Jan 30, 5:45 PM · cloud-services-team (Kanban)
Papaul created T243936: labtestpuppetmaster2001 showing Failed in Netbox.
Thu, Jan 30, 1:33 AM · cloud-services-team (Kanban)

Wed, Jan 29

Papaul updated subscribers of T243927: Export Netbox Stats for DCops to a visualization tool.
Wed, Jan 29, 11:02 PM · SRE-tools, netbox, DC-Ops, User-crusnov

Jan 25 2020

Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Jan 25 2020, 2:38 AM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.

11 servers in row D racked and Netnox updated

Jan 25 2020, 2:36 AM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Jan 25 2020, 2:16 AM · Patch-For-Review, ops-codfw, serviceops, Operations
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.

@jijiki I think we are off with the server distribution

Jan 25 2020, 2:14 AM · Patch-For-Review, ops-codfw, serviceops, Operations