Page MenuHomePhabricator

wiki_willy
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Apr 16 2019, 9:00 PM (39 w, 4 d)
Availability
Available
LDAP User
Wpao
MediaWiki User
Unknown

Recent Activity

Yesterday

wiki_willy reassigned T243088: esams: normalize the power outlet assignments from wiki_willy to RobH.

Assigning back over to @RobH after our conversation via IRC

Fri, Jan 17, 6:42 PM · Operations, ops-esams

Mon, Jan 13

wiki_willy added a comment to T237055: Terminate OE10,11,12,13 Racks.

Draft of termination letter completed by Jim from Legal. Pending review via email, before mailing out to Iron Mountain. Thanks, Willy

Mon, Jan 13, 1:35 AM · ops-esams, Operations

Thu, Jan 9

wiki_willy reassigned T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet from Joe to Jclark-ctr.
Thu, Jan 9, 5:09 PM · serviceops, ops-eqiad, Operations
wiki_willy reassigned T241849: (No Need By Date Provided) rack/setup/install mw[1385-1413].eqiad.wmnet from Joe to Jclark-ctr.
Thu, Jan 9, 5:07 PM · serviceops, ops-eqiad, Operations

Wed, Jan 8

wiki_willy assigned T242265: rack/setup/install frlog2001.frack.codfw.wmnet to Papaul.
Wed, Jan 8, 10:29 PM · Operations, ops-codfw, fundraising-tech-ops

Tue, Jan 7

wiki_willy renamed T241794: (Need By Jan 25) rack/setup/install snapshot1010.eqiad.wmnet from (No Need By Date) rack/setup/install snapshot1010.eqiad.wmnet to (Need By Jan 25) rack/setup/install snapshot1010.eqiad.wmnet.
Tue, Jan 7, 4:02 PM · ops-eqiad, Dumps-Generation, Operations

Mon, Jan 6

wiki_willy added a comment to T241884: Degraded RAID on cloudvirt1024.

@Jclark-ctr - if no errors pop up after firmware and bios upgrades, maybe it a bad raid controller.

Mon, Jan 6, 6:26 PM · cloud-services-team (Hardware), Patch-For-Review, ops-eqiad, Operations
wiki_willy moved T241313: cloudvirt1013: server down for no reason (power issue?) from Backlog to Cloud Tasks on the ops-eqiad board.
Mon, Jan 6, 6:25 PM · cloud-services-team (Hardware), Operations, ops-eqiad, DC-Ops
wiki_willy assigned T241313: cloudvirt1013: server down for no reason (power issue?) to Jclark-ctr.
Mon, Jan 6, 6:24 PM · cloud-services-team (Hardware), Operations, ops-eqiad, DC-Ops
wiki_willy moved T241882: cloudvirt1016 crash from Backlog to Cloud Tasks on the ops-eqiad board.
Mon, Jan 6, 6:13 PM · cloud-services-team (Hardware), ops-eqiad, DC-Ops, Operations
wiki_willy added a comment to T241882: cloudvirt1016 crash.

@Jclark-ctr looks like this one is warranty until May 2020, so you can just RMA this with Dell. Thanks, Willy

Mon, Jan 6, 6:13 PM · cloud-services-team (Hardware), ops-eqiad, DC-Ops, Operations
wiki_willy assigned T241882: cloudvirt1016 crash to Jclark-ctr.
Mon, Jan 6, 6:11 PM · cloud-services-team (Hardware), ops-eqiad, DC-Ops, Operations
wiki_willy assigned T241494: Degraded RAID on cloudvirt1014 to Jclark-ctr.
Mon, Jan 6, 4:48 PM · ops-eqiad, Operations
wiki_willy assigned T241884: Degraded RAID on cloudvirt1024 to Jclark-ctr.
Mon, Jan 6, 4:47 PM · cloud-services-team (Hardware), Patch-For-Review, ops-eqiad, Operations

Fri, Jan 3

wiki_willy added a comment to T238305: servers freeze across the caching cluster.

In going through all the affected systems in this task, I'd like to treat db2125 and backup2001 separately, since they seem like one-offs and could very well be hardware issues. (db2125 was 1 of 10 systems in the batch and backup2001 was ordered 1.5yrs ago) . Even the two cp1077 and cp1087 systems in eqiad have been around since May 2018, so those could also be related or unrelated to the cp crashes in esams as well.

Fri, Jan 3, 7:31 PM · Traffic, Operations
wiki_willy renamed T241850: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet from rack/setup/install new eqiad kubenetes systems to (No Need By Date Provided) rack/setup/install new eqiad kubenetes systems.
Fri, Jan 3, 7:01 PM · serviceops, ops-eqiad, Operations
wiki_willy renamed T241849: (No Need By Date Provided) rack/setup/install mw[1385-1413].eqiad.wmnet from rack/setup/install new eqiad mw systems to (No Need By Date Provided) rack/setup/install new eqiad mw systems.
Fri, Jan 3, 7:00 PM · serviceops, ops-eqiad, Operations
wiki_willy renamed T241790: (No Need By Date Provided) rack/setup/install restbase202[123] from rack/setup/install restbase202[123] to (No Need By Date Provided) rack/setup/install restbase202[123].
Fri, Jan 3, 6:54 PM · Core Platform Team Workboards (Clinic Duty Team), ops-codfw, Operations
wiki_willy renamed T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org from codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org to (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org.
Fri, Jan 3, 6:53 PM · Operations, ops-codfw

Thu, Jan 2

wiki_willy renamed T239732: (No Need By Date Provided) codfw: rack/setup/install puppetmaster2003.codfw.wmnet from codfw: rack/setup/install puppetmaster2003.codfw.wmnet to (No Need By Date Provided) codfw: rack/setup/install puppetmaster2003.codfw.wmnet.
Thu, Jan 2, 11:43 PM · Operations, ops-codfw
wiki_willy renamed T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet from codfw: rack/setup/install mc203[7,8,9].codfw.wmnet to (No Need By Date Provided) codfw: rack/setup/install mc203[7,8,9].codfw.wmnet.
Thu, Jan 2, 11:43 PM · Operations, ops-codfw
wiki_willy renamed T241336: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet from codfw: rack/setup/install es202[0-5].codfw.wmnet to (No Need By Date Provided) codfw: rack/setup/install es202[0-5].codfw.wmnet.
Thu, Jan 2, 11:41 PM · Operations, DBA, ops-codfw
wiki_willy renamed T240882: (No Need By Date Provided) rack/setup/install logstash202[6-9].codfw.wmnet from rack/setup/install logstash202[6-9].codfw.wmnet to (No Need By Date Provided) rack/setup/install logstash202[6-9].codfw.wmnet.
Thu, Jan 2, 11:40 PM · Patch-For-Review, Operations, ops-codfw, Wikimedia-Logstash
wiki_willy renamed T241796: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet from rack/setup/install mc-gp200[123].codfw.wmnet to (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet.
Thu, Jan 2, 11:39 PM · ops-codfw, serviceops, Operations
wiki_willy renamed T240881: (No Need By Date) rack/setup/install logstash102[6-9].eqiad.wmnet from rack/setup/install logstash102[6-9].eqiad.wmnet to (No Need By Date) rack/setup/install logstash102[6-9].eqiad.wmnet.
Thu, Jan 2, 11:37 PM · Operations, ops-eqiad, Wikimedia-Logstash
wiki_willy renamed T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet from eqiad: rack/setup/install es102[0-5].eqiad.wmnet to (No Need By Date) eqiad: rack/setup/install es102[0-5].eqiad.wmnet.
Thu, Jan 2, 11:36 PM · DBA, Operations, ops-eqiad
wiki_willy renamed T241794: (Need By Jan 25) rack/setup/install snapshot1010.eqiad.wmnet from rack/setup/install snapshot1010.eqiad.wmnet to (No Need By Date) rack/setup/install snapshot1010.eqiad.wmnet.
Thu, Jan 2, 11:35 PM · ops-eqiad, Dumps-Generation, Operations
wiki_willy renamed T241791: (No Need By Date) rack/setup/install relforge100[34] from rack/setup/install relforge100[34] to (No Need By Date) rack/setup/install relforge100[34].
Thu, Jan 2, 11:34 PM · ops-eqiad, Discovery-Search (Current work), Operations
wiki_willy renamed T241784: (No Need By Date) rack/setup/install restbase1029, restbase1029, restbase1030 from rack/setup/install restbase1029, restbase1029, restbase1030 to (No Need By Date) rack/setup/install restbase1029, restbase1029, restbase1030.
Thu, Jan 2, 11:33 PM · Core Platform Team Workboards (Clinic Duty Team), ops-eqiad, Operations
wiki_willy renamed T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet from rack/setup/install mc-gp100[123].eqiad.wmnet to (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Thu, Jan 2, 11:32 PM · serviceops, ops-eqiad, Operations

Sat, Dec 28

wiki_willy moved T238624: Decommission db1061.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Sat, Dec 28, 9:06 PM · Operations, DC-Ops, ops-eqiad, decommission
wiki_willy moved T239188: Decommission db1062.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Sat, Dec 28, 9:06 PM · Operations, DC-Ops, ops-eqiad, decommission
wiki_willy moved T240187: mw1280 crashed logging correctable memory errors from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Sat, Dec 28, 9:05 PM · Operations, ops-eqiad
wiki_willy moved T241494: Degraded RAID on cloudvirt1014 from Backlog to Cloud Tasks on the ops-eqiad board.
Sat, Dec 28, 9:05 PM · ops-eqiad, Operations
wiki_willy moved T241506: Degraded RAID on db1100 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Sat, Dec 28, 9:04 PM · DBA, ops-eqiad, Operations
wiki_willy assigned T241506: Degraded RAID on db1100 to Jclark-ctr.

@Jclark-ctr - when you come in next, can you open up a RMA for this one? Thanks, Willy

Sat, Dec 28, 9:01 PM · DBA, ops-eqiad, Operations

Mon, Dec 23

wiki_willy added a comment to T237055: Terminate OE10,11,12,13 Racks.

Followed up with Jim last week on this, and sorted out a few questions he had with the existing contract and what we would like to include in the termination letter. Will update again, when more progress is made. Thanks, Willy

Mon, Dec 23, 5:58 PM · ops-esams, Operations
wiki_willy moved T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet from Backlog to Racking Tasks on the ops-eqiad board.
Mon, Dec 23, 5:53 PM · DBA, Operations, ops-eqiad
wiki_willy assigned T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet to Jclark-ctr.
Mon, Dec 23, 5:53 PM · DBA, Operations, ops-eqiad
wiki_willy added a parent task for T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet: Unknown Object (Task).
Mon, Dec 23, 5:52 PM · DBA, Operations, ops-eqiad

Fri, Dec 20

wiki_willy assigned T241155: Relabel labmon1001.eqiad.wmnet to cloudmetrics1001eqiad.wmnet and labmon1002.eqiad.wmnet to cloudmetrics1002eqiad.wmnet to Cmjohnson.
Fri, Dec 20, 11:39 PM · DC-Ops, ops-eqiad, Operations, cloud-services-team
wiki_willy added a comment to T228099: rack/setup/install ganeti500[123].eqsin.wmnet.

@RobH - I'm pretty sure we took care of all the onsite work via T229243 with DreamICC, but can you confirm and check off the boxes in this task, up to the current step we're on? Thanks, Willy

Fri, Dec 20, 5:34 PM · Operations

Dec 17 2019

wiki_willy assigned T239880: Replacement hardware for buster/stretch upgrade of contint1001 and contint2001 to RobH.
Dec 17 2019, 10:52 PM · Continuous-Integration-Infrastructure (phase-out-jessie), DC-Ops, hardware-requests, Operations

Dec 13 2019

wiki_willy assigned T239805: ms-fe2007 NIC failure to Papaul.
Dec 13 2019, 11:29 PM · User-fgiunchedi, ops-codfw, Operations

Dec 12 2019

wiki_willy reassigned T240534: Degraded RAID on db1123 from wiki_willy to Jclark-ctr.

@Jclark-ctr - looks like this one is still under warranty, so you should be able to just RMA it. Thanks, Willy

Dec 12 2019, 5:05 PM · DBA, ops-eqiad, Operations

Dec 10 2019

wiki_willy assigned T184066: rack/setup/install ps[12]-oe1[456]-esams to RobH.
Dec 10 2019, 9:03 PM · Operations, ops-esams
wiki_willy assigned T237009: Add missing labels for equipment and cables to RobH.
Dec 10 2019, 8:43 PM · DC-Ops, ops-esams, Operations

Dec 9 2019

wiki_willy assigned T240187: mw1280 crashed logging correctable memory errors to Jclark-ctr.

Looks like it's out of warranty. We can purchase a replacement DIMM, but will need the correct specs to place the order.

Dec 9 2019, 8:38 PM · Operations, ops-eqiad

Dec 6 2019

wiki_willy closed T239957: Degraded RAID on cloudelastic1002 as Resolved.
Dec 6 2019, 9:58 PM · Discovery-Search (Current work), Discovery, ops-eqiad, Operations

Dec 5 2019

wiki_willy assigned T239957: Degraded RAID on cloudelastic1002 to Jclark-ctr.

Looks like this one is a duplicate of T230088

Dec 5 2019, 11:16 PM · Discovery-Search (Current work), Discovery, ops-eqiad, Operations
wiki_willy renamed T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet from (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet to (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.
Dec 5 2019, 5:46 PM · serviceops, Operations, ops-eqiad
wiki_willy added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

Thanks for confirming @jijiki - I'll let @Jclark-ctr provide an ETA on them, when he gets in a bit later today. Thanks, Willy

Dec 5 2019, 5:30 PM · serviceops, Operations, ops-eqiad
wiki_willy added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

Hi @jijiki - we received these on Sept 25, but just recently received the racking instructions from your team on Nov 25, so there might be some shuffling around we need to do, before getting these racked to meet the recommendations. When do you need these handed over to you by? And we can try to meet that need by date. Thanks, Willy

Dec 5 2019, 5:24 PM · serviceops, Operations, ops-eqiad

Dec 4 2019

wiki_willy assigned T239758: mw2259 down and mgmt does not exist? to Papaul.
Dec 4 2019, 1:56 AM · DC-Ops, ops-codfw, serviceops, Operations

Dec 3 2019

wiki_willy added a comment to T239009: Degraded RAID on ganeti2002.

Per Papaul's request, we'll procure the replacement replacement drive ahead of time before it fails again via T239738 . Thanks, Willy

Dec 3 2019, 6:36 PM · Operations, ops-codfw
wiki_willy added a subtask for T239009: Degraded RAID on ganeti2002: Unknown Object (Task).
Dec 3 2019, 6:32 PM · Operations, ops-codfw

Dec 2 2019

wiki_willy reassigned T239569: cloudstore1008 crash - Memory error from wiki_willy to Jclark-ctr.

Re-assigning to @Jclark-ctr , who can coordinate with @JHedden during their weekly sync up on Tuesdays. Thanks, Willy

Dec 2 2019, 11:34 PM · Operations, ops-eqiad, cloud-services-team (Kanban), Cloud-Services
wiki_willy assigned T239597: Hardware asset tag Netbox/DNS mgmt inconsistencies to Jclark-ctr.
Dec 2 2019, 8:39 PM · ops-eqiad, Operations, DC-Ops

Nov 27 2019

wiki_willy assigned T239365: Degraded RAID on an-worker1089 to Jclark-ctr.

@Jclark-ctr - looks like the system is under warranty until October 2021, so we'll be able to RMA it. Thanks, Willy

Nov 27 2019, 9:57 PM · Analytics, ops-eqiad, Operations
wiki_willy closed T239211: HP SSD Failure Firmware Fix as Resolved.

Confirmed by Dasher (email below), that we're not impacted by the critical firmware update bulletins. Resolving task.

Nov 27 2019, 5:33 PM · ops-eqiad, ops-codfw, Operations, DC-Ops
wiki_willy edited projects for T239211: HP SSD Failure Firmware Fix, added: ops-codfw, ops-eqiad; removed procurement.
Nov 27 2019, 5:06 PM · ops-eqiad, ops-codfw, Operations, DC-Ops
wiki_willy added a project to T239211: HP SSD Failure Firmware Fix: procurement.

Email sent over to Dasher/HP to see if they can cross-check if any of our previously purchased drives are impacted. Thanks, Willy

Nov 27 2019, 5:05 PM · ops-eqiad, ops-codfw, Operations, DC-Ops
wiki_willy claimed T239211: HP SSD Failure Firmware Fix.
Nov 27 2019, 4:51 PM · ops-eqiad, ops-codfw, Operations, DC-Ops

Nov 26 2019

wiki_willy assigned T239009: Degraded RAID on ganeti2002 to Papaul.

Looks like the warranty expired on March 19, 2018...and we're coming up on the 5yr mark. The hardware refresh is scheduled for Q3, so let's confirm with the service owner that we can go without replacing the part on this. Thanks, Willy

Nov 26 2019, 8:51 PM · Operations, ops-codfw
wiki_willy assigned T220503: Decommission neodymium to Jclark-ctr.
Nov 26 2019, 8:36 PM · decommission, ops-eqiad, Operations

Nov 25 2019

wiki_willy assigned T239045: analytics1057's BBU is faulty to Jclark-ctr.
Nov 25 2019, 9:49 PM · Analytics, Operations, ops-eqiad
wiki_willy assigned T239098: Duplicate cable label in cr1-eqiad/cr2-eqiad to Jclark-ctr.
Nov 25 2019, 9:48 PM · DC-Ops, ops-eqiad, Operations
wiki_willy added a comment to T239139: rack/setup/install frdb1003.frack.eqiad.wmnet.

@Jgreen - I gave @Jclark-ctr a heads up on this, so he'll starting working on it, when he gets in a bit later today. Thanks, Willy

Nov 25 2019, 7:14 PM · Operations, fundraising-tech-ops, ops-eqiad

Nov 22 2019

wiki_willy added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

Thanks @jijiki , much appreciated

Nov 22 2019, 10:08 PM · serviceops, Operations, ops-eqiad
wiki_willy renamed T234069: rack/setup/install frban2001.frack.codfw.wmnet from rack/setup/install frban2001.frack.codfw.wmnet to (No Need By Date Provided) rack/setup/install frban2001.frack.codfw.wmnet.
Nov 22 2019, 9:27 PM · Operations, fundraising-tech-ops, ops-codfw
wiki_willy renamed T228919: (No Need By Date Provided) replace scs-a8-eqiad from replace scs-a8-eqiad to (No Need By Date Provided) replace scs-a8-eqiad.
Nov 22 2019, 9:01 PM · Operations, ops-eqiad
wiki_willy renamed T232137: (ASAP) rack/setup/install frnetmon1001.frack.eqiad.wmnet from rack/setup/install frnetmon1001.frack.eqiad.wmnet to (No Need By Date Provided) rack/setup/install frnetmon1001.frack.eqiad.wmnet.
Nov 22 2019, 9:01 PM · fundraising-tech-ops, Operations, ops-eqiad
wiki_willy renamed T234068: (ASAP) rack/setup/install frban1001.frack.eqiad.wmnet from rack/setup/install frban1001.eqiad.wmnet to (No Need By Date Provided) rack/setup/install frban1001.eqiad.wmnet.
Nov 22 2019, 9:00 PM · Operations, fundraising-tech-ops, ops-eqiad
wiki_willy renamed T235685: (No Need By Date Provided) rack/setup/install cloudvirt-wdqs100[123].eqiad.wmnet from rack/setup/install cloudvirt-wdqs100[123].eqiad.wmnet to (No Need By Date Provided) rack/setup/install cloudvirt-wdqs100[123].eqiad.wmnet.
Nov 22 2019, 8:59 PM · cloud-services-team (Hardware), Operations, ops-eqiad
wiki_willy renamed T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet from rack/setup/install ms-be105[7-9].eqiad.wmnet to (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet.
Nov 22 2019, 8:59 PM · ops-eqiad, User-fgiunchedi, Operations
wiki_willy renamed T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet from rack/setup/install mw13[49-84].eqiad.wmnet to (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet.
Nov 22 2019, 8:57 PM · serviceops, Operations, ops-eqiad

Nov 21 2019

wiki_willy added a comment to T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.

@Joe et al - just following up on this, since we've had these servers onsite for a while now. Also, once you have any specific racking instructions for us, please keep in mind that we'll need some time to complete all the tasks associated with getting these hosts set up, so it should be reflected in your requested "need by date" as well.

Nov 21 2019, 10:59 PM · serviceops, Operations, ops-eqiad

Nov 18 2019

wiki_willy added a comment to T232654: eqiad: three clouvirt-wdqs servers for WDQS testing.

Hey @Andrew - I think this is the racking task for these - https://phabricator.wikimedia.org/T235685. Servers arrived on November 5. Thanks, Willy

Nov 18 2019, 7:40 PM · DC-Ops, Operations, hardware-requests

Nov 15 2019

wiki_willy added a comment to T227632: Document PDU models.

@faidon - I'll check with the team on this one, and see if we can get an estimate for completion. Thanks, Willy

Nov 15 2019, 8:21 PM · netbox, Operations, ops-codfw, ops-eqiad

Nov 13 2019

wiki_willy closed T226778: Install new PDUs in rows A/B (Top level tracking task) as Resolved.

Resolving parent task for PDU upgrades. Much appreciated to @Cmjohnson and @Jclark-ctr for taking care of these. Thanks, Willy

Nov 13 2019, 5:16 PM · DC-Ops, Operations, ops-eqiad

Nov 7 2019

wiki_willy assigned T236497: cp3056 hardware issue to RobH.
Nov 7 2019, 7:20 PM · DC-Ops, ops-esams, Operations, Traffic
wiki_willy added a comment to T237582: frqueue1001 system battery needs replacement.

Thanks @Jgreen - much appreciated.

Nov 7 2019, 5:52 PM · ops-eqiad, Operations
wiki_willy added a comment to T237582: frqueue1001 system battery needs replacement.

@Jgreen - Child task T237651 created to order the part. For any hardware repair requests going forward, can you follow the template here - https://phabricator.wikimedia.org/maniphest/task/edit/form/55/ . There's a few things in the description that we're trying to get in advance to help us with scheduling downtime, prioritization, etc. Much appreciated.

Nov 7 2019, 5:20 PM · ops-eqiad, Operations
wiki_willy added a subtask for T237582: frqueue1001 system battery needs replacement: Unknown Object (Task).
Nov 7 2019, 5:16 PM · ops-eqiad, Operations
wiki_willy assigned T237582: frqueue1001 system battery needs replacement to Jclark-ctr.

@Jgreen - looks like the warranty ended for the server a few months ago in May. Let me know if you're looking to decommission this server soon or if you would like us to purchase the replacement part.

Nov 7 2019, 12:00 AM · ops-eqiad, Operations

Nov 5 2019

wiki_willy added a comment to T236187: decom cobalt.

@Dzahn - sounds good to me.

Nov 5 2019, 11:58 PM · serviceops, Operations

Nov 4 2019

wiki_willy closed Unknown Object (Task), a subtask of T228606: Degraded RAID on elastic1046, as Resolved.
Nov 4 2019, 11:21 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations
wiki_willy closed T228606: Degraded RAID on elastic1046 as Resolved.
Nov 4 2019, 11:18 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations
wiki_willy added a comment to T228606: Degraded RAID on elastic1046.

Thanks @Gehel , thanks @Jclark-ctr - I'll go ahead and resolve this task. Thanks, Willy

Nov 4 2019, 11:18 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations
wiki_willy assigned T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC) to Cmjohnson.
Nov 4 2019, 6:28 PM · DC-Ops, Operations, ops-eqiad
wiki_willy renamed T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC) from b7-eqiad pdu refresh (Tuesday 11/5 @10am UTC) to b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).
Nov 4 2019, 4:19 PM · DC-Ops, Operations, ops-eqiad

Nov 1 2019

wiki_willy assigned T237133: analytics1062 lost one of its power supplies to Jclark-ctr.

@Jclark-ctr - looks like this one is from last Thursday's PDU upgrade. Can you check if it's maybe a loose cord? If not, we'll have to RMA it (server under warranty thru March 2020) . Thanks, Willy

Nov 1 2019, 10:35 PM · Analytics, Operations, Analytics-Cluster, ops-eqiad

Oct 31 2019

wiki_willy added a comment to T237055: Terminate OE10,11,12,13 Racks.

Emailed Jim Buatti last Tuesday to provide overview of what we're trying to do with the contract (have OE14,15,16 renew in May 2021 and terminate OE10,11,12,13 in Nov 2020 with clause to term early if another customer can be found to lease the racks), and received confirmation that he'll help us put something together. Thanks, Willy

Oct 31 2019, 9:45 PM · Operations, ops-esams
wiki_willy added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T237055: Terminate OE10,11,12,13 Racks.
Oct 31 2019, 9:42 PM · Patch-For-Review, DC-Ops, Operations, ops-esams
wiki_willy added a parent task for T237055: Terminate OE10,11,12,13 Racks: T235805: ESAMS Refresh/Rebuild (October 2019).
Oct 31 2019, 9:42 PM · Operations, ops-esams
wiki_willy created T237055: Terminate OE10,11,12,13 Racks.
Oct 31 2019, 9:41 PM · Operations, ops-esams
wiki_willy reassigned T227543: b8-eqiad pdu refresh (Thursday 10/31 @11am UTC) from Cmjohnson to RobH.

Cable reseated (clip was bent) by @Jclark-ctr - reassigning back to @RobH for configuration.

Oct 31 2019, 7:51 PM · DC-Ops, Operations, ops-eqiad
wiki_willy reassigned T236437: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet from Joe to Jclark-ctr.

Assigning to @Jclark-ctr since he's going to be taking care of the install, but @Joe - let us know if there are any specific racking instructions for these. Thanks, Willy

Oct 31 2019, 7:41 PM · serviceops, Operations, ops-eqiad

Oct 30 2019

wiki_willy added a comment to T227867: mw1239 memory errors .

@jijiki - just following up to see if this is still an issue or if we can resolve this. Thanks, Willy

Oct 30 2019, 8:14 PM · ops-eqiad, DC-Ops, Operations, serviceops
wiki_willy reassigned T234076: (Need by Aug 1) rack/setup/install dumpsdata1003.eqiad.wmnet from ArielGlenn to Cmjohnson.

Reassigning to @Cmjohnson for Ariel's RAID question

Oct 30 2019, 8:12 PM · Operations