Page MenuHomePhabricator

Jclark-ctr (John Clark)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Jul 24 2019, 8:11 PM (16 w, 6 d)
Availability
Available
LDAP User
Jclark-ctr
MediaWiki User
Jclark-ctr [ Global Accounts ]

Recent Activity

Today

Jclark-ctr added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

followed up with dell regarding results spoke with Madhusudan.Rao@dell.com on phone. he will follow up with Kapeel Pawaskar

Wed, Nov 20, 1:55 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)

Yesterday

Jclark-ctr added a comment to T230575: Degraded RAID on cloudvirt1018.

Replaced Failed drive

Tue, Nov 19, 10:19 PM · ops-eqiad, Operations
Jclark-ctr added a comment to T236331: Degraded RAID on cloudvirt1018.

Replaced Failed Drive

Tue, Nov 19, 10:16 PM · ops-eqiad, Operations

Mon, Nov 18

Jclark-ctr reassigned T237438: rack/setup/install ms-be105[7-9].eqiad.wmnet from Jclark-ctr to RobH.

@Cmjohnson entered mgmt password for ms-be1057 again

Mon, Nov 18, 9:55 PM · Patch-For-Review, Operations, ops-eqiad

Wed, Nov 13

Jclark-ctr added a comment to T236601: Degraded RAID on elastic1039.

@Gehel replaced drive

Wed, Nov 13, 4:54 PM · Discovery-Search (Current work), ops-eqiad, Operations
Jclark-ctr closed T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC), a subtask of T226778: Install new PDUs in rows A/B (Top level tracking task), as Resolved.
Wed, Nov 13, 4:40 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr closed T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC) as Resolved.

confirmed link and errors cleared from icinga

Wed, Nov 13, 4:40 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr updated the task description for T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).
Wed, Nov 13, 4:39 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr closed Unknown Object (Task), a subtask of T228606: Degraded RAID on elastic1046, as Resolved.
Wed, Nov 13, 4:02 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations
Jclark-ctr closed T237133: analytics1062 lost one of its power supplies as Resolved.

alert cleared no errors in icinga

Wed, Nov 13, 3:54 PM · Analytics, Operations, Analytics-Cluster, ops-eqiad

Tue, Nov 12

Jclark-ctr added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

Verified performance mode in bios . loaded stress test multiple errors on start up sent errors to dell Requested Support Engineer for replacement

Tue, Nov 12, 8:09 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Jclark-ctr added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

@Andrew want to confirm this box is not in use right now. Need to perform additional test for dell

Tue, Nov 12, 7:04 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)

Thu, Nov 7

Jclark-ctr closed T237133: analytics1062 lost one of its power supplies as Resolved.
Thu, Nov 7, 10:15 PM · Analytics, Operations, Analytics-Cluster, ops-eqiad
Jclark-ctr added a comment to T237133: analytics1062 lost one of its power supplies.

Received new psu and replaced.

Thu, Nov 7, 10:14 PM · Analytics, Operations, Analytics-Cluster, ops-eqiad
Jclark-ctr added a comment to T237438: rack/setup/install ms-be105[7-9].eqiad.wmnet.

Rob Finished configuring bios and mgmt

Thu, Nov 7, 12:26 AM · Patch-For-Review, Operations, ops-eqiad

Wed, Nov 6

Jclark-ctr updated the task description for T237438: rack/setup/install ms-be105[7-9].eqiad.wmnet.
Wed, Nov 6, 10:41 PM · Patch-For-Review, Operations, ops-eqiad
Jclark-ctr added a comment to T237567: mw1247: IPMI Sensor Status UNKNOWN internal IPMI error.

preformed flea power drain

Wed, Nov 6, 9:53 PM · ops-eqiad, Operations
Jclark-ctr added a comment to T236739: frdb1001 has suffered a raid event resulting in /dev/sda going read only.

Replaced Raid controller and bbu

Wed, Nov 6, 9:49 PM · fundraising-tech-ops, Fundraising-Backlog, DC-Ops
Jclark-ctr added a comment to T237438: rack/setup/install ms-be105[7-9].eqiad.wmnet.

@fgiunchedi The host was put into rack prior to this ticket being made i can move to row D today

Wed, Nov 6, 1:17 PM · Patch-For-Review, Operations, ops-eqiad

Tue, Nov 5

Jclark-ctr added a comment to T237438: rack/setup/install ms-be105[7-9].eqiad.wmnet.

host racked and cabled.

Tue, Nov 5, 11:36 PM · Patch-For-Review, Operations, ops-eqiad
Jclark-ctr updated the task description for T237438: rack/setup/install ms-be105[7-9].eqiad.wmnet.
Tue, Nov 5, 11:35 PM · Patch-For-Review, Operations, ops-eqiad
Jclark-ctr updated the task description for T237438: rack/setup/install ms-be105[7-9].eqiad.wmnet.
Tue, Nov 5, 11:35 PM · Patch-For-Review, Operations, ops-eqiad
Jclark-ctr added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

@JHedden . Found host did not have ip address in it. Reentered address and mgnt password

Tue, Nov 5, 10:15 PM · Operations, cloud-services-team (Kanban), Cloud-Services
Jclark-ctr reassigned T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC) from Cmjohnson to RobH.

Connect to port > 15

Tue, Nov 5, 1:24 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr updated the task description for T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).
Tue, Nov 5, 1:23 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).

finished pdu refresh

Tue, Nov 5, 1:22 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).

starting pdu refresh .

Tue, Nov 5, 12:01 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr closed T229284: add all remaining new pdus to netbox, a subtask of T226778: Install new PDUs in rows A/B (Top level tracking task), as Resolved.
Tue, Nov 5, 12:00 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr closed T229284: add all remaining new pdus to netbox as Resolved.
Tue, Nov 5, 12:00 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr updated the task description for T229284: add all remaining new pdus to netbox.
Tue, Nov 5, 11:59 AM · DC-Ops, Operations, ops-eqiad

Mon, Nov 4

Jclark-ctr added a comment to T237133: analytics1062 lost one of its power supplies.

unfortunately not a loose power cord Submitted Tech direct ticket for replacement psu Service Request 1001998096

Mon, Nov 4, 10:22 PM · Analytics, Operations, Analytics-Cluster, ops-eqiad
Jclark-ctr added a comment to T228606: Degraded RAID on elastic1046.

Replaced drive sda . @Gehel

Mon, Nov 4, 10:17 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations
Jclark-ctr added a comment to T228606: Degraded RAID on elastic1046.

received replacement drive @elukey @Gehel.

Mon, Nov 4, 10:01 PM · Patch-For-Review, Discovery-Search (Current work), ops-eqiad, Operations

Fri, Nov 1

Jclark-ctr updated the task description for T202040: Decommission radon.
Fri, Nov 1, 10:40 PM · decommission, ops-eqiad, Traffic, Operations
Jclark-ctr updated the task description for T206524: Decommission analytics1003.
Fri, Nov 1, 10:40 PM · ops-eqiad, Operations, decommission, DC-Ops, Analytics
Jclark-ctr updated the task description for T209357: Return graphite100[13] to spares pool (or decom).
Fri, Nov 1, 10:38 PM · Patch-For-Review, ops-eqiad, decommission, User-fgiunchedi, Operations
Jclark-ctr updated the task description for T209738: decom einsteinium.
Fri, Nov 1, 10:38 PM · ops-eqiad, decommission, Operations
Jclark-ctr updated the task description for T216491: Decommission dbstore1002.
Fri, Nov 1, 10:37 PM · Patch-For-Review, decommission, ops-eqiad, Operations, Analytics
Jclark-ctr updated the task description for T223217: Decommission db1064.
Fri, Nov 1, 10:37 PM · Patch-For-Review, Operations, ops-eqiad, decommission
Jclark-ctr updated the task description for T226689: decommission db1068.
Fri, Nov 1, 10:35 PM · DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr updated the task description for T227560: decommission db1065.
Fri, Nov 1, 10:34 PM · ops-eqiad, Operations, DC-Ops, decommission
Jclark-ctr added a comment to T229381: decommission db1071.eqiad.wmnet.

wiped drives added to google decom sheet

Fri, Nov 1, 10:34 PM · Patch-For-Review, DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr updated the task description for T229381: decommission db1071.eqiad.wmnet.
Fri, Nov 1, 10:33 PM · Patch-For-Review, DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr updated the task description for T232564: Decommission db1063.eqiad.wmnet.
Fri, Nov 1, 10:32 PM · DC-Ops, decommission, ops-eqiad, Operations
Jclark-ctr updated the task description for T233289: Decommission ms-be1027.
Fri, Nov 1, 10:30 PM · decommission, Operations, ops-eqiad
Jclark-ctr added a comment to T233289: Decommission ms-be1027.

removed all drives to be degaused hardware failure will not boot to wipe drives

Fri, Nov 1, 10:29 PM · decommission, Operations, ops-eqiad

Thu, Oct 31

Jclark-ctr added a comment to T227543: b8-eqiad pdu refresh (Thursday 10/31 @11am UTC).

Reseated cable fixed bent clip.

Thu, Oct 31, 7:55 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr reassigned T227543: b8-eqiad pdu refresh (Thursday 10/31 @11am UTC) from Cmjohnson to RobH.
Thu, Oct 31, 12:27 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227543: b8-eqiad pdu refresh (Thursday 10/31 @11am UTC).

finished pdu refresh, netbox updated,

Thu, Oct 31, 12:26 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227543: b8-eqiad pdu refresh (Thursday 10/31 @11am UTC).

Starting pdu refresh eqiad b8

Thu, Oct 31, 10:57 AM · DC-Ops, Operations, ops-eqiad

Wed, Oct 30

Jclark-ctr reassigned T230746: (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet from Jclark-ctr to Christopher.
Wed, Oct 30, 12:52 PM · Discovery-Search (Current work), Patch-For-Review, Operations
Jclark-ctr added a comment to T230746: (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet.

Host Switchport
elastic1053 33
elastic1054 30
elastic1055 26
elastic1056 23
elastic1057 25
elastic1058 20
elastic1059 21
elastic1060 11
elastic1061 12
elastic1062 28
elastic1063 39
elastic1064 35
elastic1065 4
elastic1066 7
elastic1067 9

Wed, Oct 30, 2:19 AM · Discovery-Search (Current work), Patch-For-Review, Operations
Jclark-ctr updated the task description for T230746: (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet.
Wed, Oct 30, 2:19 AM · Discovery-Search (Current work), Patch-For-Review, Operations
Jclark-ctr added a comment to T230746: (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet.

updated netbox, Finished Idrac and bios setup

Wed, Oct 30, 2:17 AM · Discovery-Search (Current work), Patch-For-Review, Operations

Tue, Oct 29

Jclark-ctr added a comment to T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).

netbox updated/ serial connected

Tue, Oct 29, 1:19 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr reassigned T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC) from Cmjohnson to RobH.
Tue, Oct 29, 1:18 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr updated the task description for T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).
Tue, Oct 29, 1:18 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).

Finished with swapping of pdu`s

Tue, Oct 29, 12:05 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227538: b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).

starting pdu refresh

Tue, Oct 29, 10:59 AM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T230746: (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet.

All host racked and powered, console and network finished for all. Will update netbox and finish idrac settings tomorrow

Tue, Oct 29, 12:05 AM · Discovery-Search (Current work), Patch-For-Review, Operations

Fri, Oct 25

Jclark-ctr added a comment to T236331: Degraded RAID on cloudvirt1018.

@Bstorm Is this host in use? when can we schedule a good time for me to troubleshoot?

Fri, Oct 25, 8:19 PM · ops-eqiad, Operations

Thu, Oct 24

Jclark-ctr added a comment to T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC).

Finished pdu refresh

Thu, Oct 24, 12:48 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr reassigned T227142: a6-eqiad pdu refresh (Tuesday 10/22 @11am UTC) from Jclark-ctr to RobH.

@RobH Corrected cable issue on pdu

Thu, Oct 24, 12:47 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC).

Starting Pdu Replacement

Thu, Oct 24, 11:08 AM · DC-Ops, Operations, ops-eqiad

Tue, Oct 22

Jclark-ctr reassigned T227142: a6-eqiad pdu refresh (Tuesday 10/22 @11am UTC) from Cmjohnson to RobH.

finished PDU Maintenance . Netbox updated with new PDU

Tue, Oct 22, 12:23 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T227142: a6-eqiad pdu refresh (Tuesday 10/22 @11am UTC).

starting PDU Maintenance

Tue, Oct 22, 11:02 AM · DC-Ops, Operations, ops-eqiad

Mon, Oct 21

Jclark-ctr added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

From Dell Support they have not been able to find any hardware errors from tsr report

Mon, Oct 21, 7:40 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)

Oct 17 2019

Jclark-ctr added a comment to T227133: a8-eqiad pdu refresh (Thursday 10/17 @11am UTC).

pdu swap completed all host online netbox update

Oct 17 2019, 12:34 PM · DC-Ops, Operations, ops-eqiad

Oct 16 2019

Jclark-ctr added a comment to T230289: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only.

Cleared Foreign state on ofline drives. offline drives now list as ready

Oct 16 2019, 10:00 PM · cloud-services-team, ops-eqiad, Operations
Jclark-ctr added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

Dell EMC SR # 1000122167 || Service Tag: 31R9KH2 || Server Crashes under Load

Oct 16 2019, 9:25 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Jclark-ctr added a comment to T235406: maps1002: Failed power supply.

@wiki_willy @MoritzMuehlenhoff Found loose power cable reseated cable psu is on now.

Oct 16 2019, 1:09 PM · Discovery-Search (Current work), Operations, ops-eqiad

Oct 15 2019

Jclark-ctr claimed T233289: Decommission ms-be1027.
Oct 15 2019, 9:08 PM · decommission, Operations, ops-eqiad
Jclark-ctr claimed T203520: decommission thulium.frack.eqiad.wmnet.
Oct 15 2019, 9:06 PM · decommission, Operations, ops-eqiad
Jclark-ctr claimed T202040: Decommission radon.
Oct 15 2019, 9:05 PM · decommission, ops-eqiad, Traffic, Operations
Jclark-ctr claimed T191362: decom promethium/WMF3571.
Oct 15 2019, 9:05 PM · decommission, Operations, DC-Ops, ops-eqiad
Jclark-ctr claimed T203861: decom radium.
Oct 15 2019, 9:03 PM · Patch-For-Review, ops-eqiad, decommission, Operations
Jclark-ctr claimed T209642: Remove labnodepool1001.eqiad.wmnet.
Oct 15 2019, 9:02 PM · DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr claimed T209738: decom einsteinium.
Oct 15 2019, 9:01 PM · ops-eqiad, decommission, Operations
Jclark-ctr claimed T189921: decom californium.
Oct 15 2019, 8:59 PM · Patch-For-Review, ops-eqiad, decommission, DC-Ops, Operations
Jclark-ctr claimed T206524: Decommission analytics1003.
Oct 15 2019, 8:58 PM · ops-eqiad, Operations, decommission, DC-Ops, Analytics
Jclark-ctr claimed T216491: Decommission dbstore1002.
Oct 15 2019, 8:58 PM · Patch-For-Review, decommission, ops-eqiad, Operations, Analytics
Jclark-ctr claimed T211826: decommission oxygen.eqiad.wmnet.
Oct 15 2019, 8:56 PM · Operations, ops-eqiad, DC-Ops, decommission
Jclark-ctr claimed T222109: decommission frav1001.frack.eqiad.wmnet.
Oct 15 2019, 8:55 PM · decommission, Operations, fundraising-tech-ops, ops-eqiad, DC-Ops
Jclark-ctr claimed T223217: Decommission db1064.
Oct 15 2019, 8:54 PM · Patch-For-Review, Operations, ops-eqiad, decommission
Jclark-ctr claimed T226689: decommission db1068.
Oct 15 2019, 8:52 PM · DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr claimed T227560: decommission db1065.
Oct 15 2019, 8:51 PM · ops-eqiad, Operations, DC-Ops, decommission
Jclark-ctr claimed T227166: decommission db1069.
Oct 15 2019, 8:49 PM · ops-eqiad, Operations, DC-Ops, decommission
Jclark-ctr claimed T229381: decommission db1071.eqiad.wmnet.
Oct 15 2019, 8:49 PM · Patch-For-Review, DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr claimed T228956: decommission db1072.eqiad.wmnet.
Oct 15 2019, 8:47 PM · DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr claimed T232564: Decommission db1063.eqiad.wmnet.
Oct 15 2019, 8:45 PM · DC-Ops, decommission, ops-eqiad, Operations
Jclark-ctr claimed T231892: Decommission db1073.eqiad.wmnet.
Oct 15 2019, 8:44 PM · DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr claimed T231967: Decommission dbproxy1005.eqiad.wmnet.
Oct 15 2019, 8:16 PM · DC-Ops, ops-eqiad, decommission, Operations
Jclark-ctr reassigned T226782: a1-eqiad pdu refresh (Tuesday 10/15 @11am UTC) from Jclark-ctr to RobH.

the PDU swap is over. Nothing lost while swapping PDU. Everything is cabled and they're linked together. Netbox is updated. still needs PDU configuration completed .

Oct 15 2019, 12:19 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr updated the task description for T226782: a1-eqiad pdu refresh (Tuesday 10/15 @11am UTC).
Oct 15 2019, 12:11 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr claimed T226782: a1-eqiad pdu refresh (Tuesday 10/15 @11am UTC).
Oct 15 2019, 12:10 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr added a comment to T226782: a1-eqiad pdu refresh (Tuesday 10/15 @11am UTC).

Starting pdu upgrade

Oct 15 2019, 11:00 AM · DC-Ops, Operations, ops-eqiad

Oct 14 2019

Jclark-ctr closed T233273: labsdb1009 broken PSU, a subtask of T226782: a1-eqiad pdu refresh (Tuesday 10/15 @11am UTC), as Resolved.
Oct 14 2019, 1:23 PM · DC-Ops, Operations, ops-eqiad
Jclark-ctr closed T233273: labsdb1009 broken PSU, a subtask of T233248: Power issue in eqiad A1, as Resolved.
Oct 14 2019, 1:23 PM · Operations, ops-eqiad
Jclark-ctr closed T233273: labsdb1009 broken PSU as Resolved.
Oct 14 2019, 1:23 PM · Operations, DC-Ops, ops-eqiad, DBA
Jclark-ctr added a comment to T233273: labsdb1009 broken PSU.

Replaced failed PSU

Oct 14 2019, 1:10 PM · Operations, DC-Ops, ops-eqiad, DBA