Page MenuHomePhabricator

Cmjohnson (cmjohnson)
User

Projects (11)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 10:22 PM (248 w, 2 d)
Availability
Available
IRC Nick
cmjohnson1
LDAP User
Cmjohnson
MediaWiki User
Unknown

Recent Activity

Yesterday

Cmjohnson closed T233265: Check for faulty optic asw-c-eqiad to cr1-eqiad as Resolved.

It's been nearly 24 hours and there are 0 errors. resolving the task

Thu, Sep 19, 9:03 PM · Operations, netops, ops-eqiad, DC-Ops
Cmjohnson reassigned T220505: Decommission iron from Cmjohnson to Jclark-ctr.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Thu, Sep 19, 8:58 PM · Cloud-VPS, ops-eqiad, decommission, Operations
Cmjohnson reassigned T221244: decommission astatine from Cmjohnson to Jclark-ctr.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Thu, Sep 19, 8:57 PM · ops-eqiad, DC-Ops, decommission, Operations
Cmjohnson reassigned T216749: Decommission labsdb1004.eqiad.wmnet and labsdb1005.eqiad.wmnet from Cmjohnson to Jclark-ctr.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Thu, Sep 19, 8:57 PM · ops-eqiad, Operations, decommission, Data-Services, cloud-services-team (Kanban)
Cmjohnson reassigned T221391: decommission phab1002/WMF4727 from Cmjohnson to Jclark-ctr.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Thu, Sep 19, 8:56 PM · Patch-For-Review, Operations, ops-eqiad, DC-Ops, decommission
Cmjohnson reassigned T221817: Decommission labcontrol1001 & labcontrol1002 from Cmjohnson to Jclark-ctr.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Thu, Sep 19, 8:56 PM · ops-eqiad, decommission, Operations
Cmjohnson closed T221818: Decommission labnet1001 & labnet1002 as Resolved.

these were added to the tracking sheet

Thu, Sep 19, 8:55 PM · Patch-For-Review, ops-eqiad, decommission, Operations
Cmjohnson closed T221857: Decommission labservices1001 & labservices1002 as Resolved.
Thu, Sep 19, 8:54 PM · Patch-For-Review, ops-eqiad, decommission, Operations
Cmjohnson updated the task description for T221857: Decommission labservices1001 & labservices1002.
Thu, Sep 19, 8:54 PM · Patch-For-Review, ops-eqiad, decommission, Operations
Cmjohnson assigned T224268: Decommission rhenium to Jclark-ctr.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Thu, Sep 19, 8:46 PM · Operations, ops-eqiad, decommission
Cmjohnson reassigned T226517: Decommission old Kafka analytics brokers: kafka1012,kafka1013,kafka1014,kafka1020,kafka1022,kafka1023 from RobH to Jclark-ctr.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Thu, Sep 19, 8:44 PM · ops-eqiad, DC-Ops, Analytics, decommission, Operations
Cmjohnson closed T220590: Decom ms-be101[345] as Resolved.
Thu, Sep 19, 8:43 PM · Patch-For-Review, ops-eqiad, decommission, User-fgiunchedi, media-storage, Operations
Cmjohnson updated the task description for T220590: Decom ms-be101[345].
Thu, Sep 19, 8:43 PM · Patch-For-Review, ops-eqiad, decommission, User-fgiunchedi, media-storage, Operations
Cmjohnson reassigned T191357: decom silver/WMF3434 from Cmjohnson to Jclark-ctr.

@Jclark-ctr wipe, remove the servers, update netbox and the google sheet. Please assign back to me once everything is complete

Thu, Sep 19, 8:23 PM · decommission, Operations, DC-Ops, ops-eqiad
Cmjohnson assigned T228768: Decommission dbproxy1004 and dbproxy1009 to Jclark-ctr.

@Jclark-ctr wipe, remove the servers, update netbox and the google sheet. Please assign back to me once everything is complete

Thu, Sep 19, 8:22 PM · ops-eqiad, decommission, Operations, Analytics-EventLogging, Analytics
Cmjohnson assigned T226715: decommission restbase10(0[7-9]|1[0-5]) to Jclark-ctr.
Thu, Sep 19, 8:21 PM · Operations, ops-eqiad, DC-Ops, decommission
Cmjohnson updated subscribers of T226715: decommission restbase10(0[7-9]|1[0-5]).

@Jclark-ctr wipe, remove the servers, update netbox and the google sheet. Please assign back to me once everything is complete

Thu, Sep 19, 8:21 PM · Operations, ops-eqiad, DC-Ops, decommission
Cmjohnson added a comment to T229557: decommission lithium.

@Jclark-ctr please wipe, remove, update tracking and netbox.

Thu, Sep 19, 6:19 PM · Operations, ops-eqiad, DC-Ops, decommission
Cmjohnson reassigned T229557: decommission lithium from Cmjohnson to Jclark-ctr.
Thu, Sep 19, 6:08 PM · Operations, ops-eqiad, DC-Ops, decommission
Cmjohnson added a comment to T233289: Unable to power on ms-be1027.

John checked on this first thing this morning, first thing. The power light was blinking green but are not getting any power. I had him reseat and drain flea power. That did not work. He then took it down to minimum hardware 1 DIMM and 1 CPU and the servers will still not power on. We've had this issue before and is usually resolved with a mainboard swap but this sever is out of warranty.

Thu, Sep 19, 4:44 PM · User-fgiunchedi, Operations, ops-eqiad
Cmjohnson created T233302: Verify switch port connections.
Thu, Sep 19, 12:16 PM · Operations, ops-eqiad
Cmjohnson merged task T229612: asw2-c-eqiad:xe-2/0/45 inbound interface errors into T233265: Check for faulty optic asw-c-eqiad to cr1-eqiad.
Thu, Sep 19, 12:12 PM · netops, Operations, ops-eqiad
Cmjohnson merged T229612: asw2-c-eqiad:xe-2/0/45 inbound interface errors into T233265: Check for faulty optic asw-c-eqiad to cr1-eqiad.
Thu, Sep 19, 12:12 PM · Operations, netops, ops-eqiad, DC-Ops
Cmjohnson moved T231638: db1074 crashed: Broken BBU from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Thu, Sep 19, 12:10 PM · ops-eqiad, Operations, DBA
Cmjohnson moved T231967: Decommission dbproxy1005.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Thu, Sep 19, 12:08 PM · DC-Ops, ops-eqiad, decommission, Operations
Cmjohnson moved T231892: Decommission db1073.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Thu, Sep 19, 12:08 PM · DC-Ops, ops-eqiad, decommission, Operations
Cmjohnson moved T232564: Decommission db1063.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Thu, Sep 19, 12:08 PM · DC-Ops, decommission, ops-eqiad, Operations
Cmjohnson moved T233289: Unable to power on ms-be1027 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Thu, Sep 19, 12:08 PM · User-fgiunchedi, Operations, ops-eqiad
Cmjohnson moved T233273: labsdb1009 broken PSU from Backlog to Blocked on the ops-eqiad board.
Thu, Sep 19, 12:08 PM · Operations, DC-Ops, ops-eqiad, DBA
Cmjohnson updated subscribers of T233273: labsdb1009 broken PSU.

This server is out of warranty and @RobH has created a procurement task.

Thu, Sep 19, 12:08 PM · Operations, DC-Ops, ops-eqiad, DBA

Wed, Sep 18

Cmjohnson closed T233248: Power issue in eqiad A1 as Resolved.

resolving this task

Wed, Sep 18, 11:15 PM · Operations, ops-eqiad
Cmjohnson moved T232137: rack/setup/install frnetmon1001 from Backlog to Racking Tasks on the ops-eqiad board.
Wed, Sep 18, 11:15 PM · fundraising-tech-ops, ops-eqiad, Operations
Cmjohnson added a comment to T233248: Power issue in eqiad A1.

John replaced side A pdu with a new PDU.

Wed, Sep 18, 11:15 PM · Operations, ops-eqiad
Cmjohnson moved T233248: Power issue in eqiad A1 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Sep 18, 11:14 PM · Operations, ops-eqiad
Cmjohnson moved T233265: Check for faulty optic asw-c-eqiad to cr1-eqiad from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Sep 18, 11:14 PM · Operations, netops, ops-eqiad, DC-Ops
Cmjohnson added a comment to T233265: Check for faulty optic asw-c-eqiad to cr1-eqiad.

swapped both optics on cr1-eqiad and asw2-c xe-2/045. Giving it 24 hours to see if any errors return

Wed, Sep 18, 11:14 PM · Operations, netops, ops-eqiad, DC-Ops
Cmjohnson created T233265: Check for faulty optic asw-c-eqiad to cr1-eqiad.
Wed, Sep 18, 9:56 PM · Operations, netops, ops-eqiad, DC-Ops

Tue, Sep 17

Cmjohnson reassigned T209357: Return graphite100[13] to spares pool (or decom) from Cmjohnson to Jclark-ctr.

please wipe these especially 1001 to make some space for ms-be servers

Tue, Sep 17, 12:27 PM · ops-eqiad, decommission, User-fgiunchedi, Operations

Fri, Sep 13

Cmjohnson added a comment to T227335: backup1001 can't address the disk shelf's drives.

Actually we need to close this task and open a separate task about the
disk. Different issue should get a different task.

Fri, Sep 13, 1:09 PM · ops-eqiad, Operations, DC-Ops

Thu, Sep 12

Cmjohnson added a comment to T228606: Degraded RAID on elastic1046.

I did notice that ssds are different types
The new ssd is a DC3320 series
The old ssd is a DC3610 series

Thu, Sep 12, 3:50 PM · Discovery-Search (Current work), ops-eqiad, Operations
Cmjohnson added a comment to T228606: Degraded RAID on elastic1046.

@wiki_willy not really but I reseated it anyway. As far as I can tell in bios everything looks normal. I did swap the 2 disks. @Gehel try again please.

Thu, Sep 12, 3:47 PM · Discovery-Search (Current work), ops-eqiad, Operations

Tue, Sep 10

Cmjohnson updated the task description for T230746: (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet.
Tue, Sep 10, 1:12 PM · Patch-For-Review, Operations, ops-eqiad
Cmjohnson added a comment to T227541: b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC).

The PDU has been swapped and the new pdus are in netbox. @RobH can you help with the setup for serial console please.

Tue, Sep 10, 12:44 PM · DC-Ops, Operations, ops-eqiad
Cmjohnson updated the task description for T227541: b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC).
Tue, Sep 10, 12:42 PM · DC-Ops, Operations, ops-eqiad
Cmjohnson closed Unknown Object (Task), a subtask of T221636: Replace elastic1017-1031, as Resolved.
Tue, Sep 10, 12:06 PM · Discovery-Search (Current work), Operations, hardware-requests
Cmjohnson closed Unknown Object (Task), a subtask of T219768: Get a third dumpsdata server, as Resolved.
Tue, Sep 10, 12:05 PM · hardware-requests, Operations, Dumps-Generation

Mon, Sep 9

Cmjohnson reassigned T227335: backup1001 can't address the disk shelf's drives from Cmjohnson to Jclark-ctr.

this got lost in the shuffle....will work on it this week . @Jclark-ctr can you contact HPE support and open a ticket please.

Mon, Sep 9, 3:49 PM · ops-eqiad, Operations, DC-Ops

Fri, Sep 6

Cmjohnson updated the task description for T228102: rack/setup/install cloudcephmon100[123].
Fri, Sep 6, 5:48 PM · cloud-services-team (Kanban), Operations, Cloud-Services, ops-eqiad

Thu, Sep 5

Cmjohnson added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

@Ottomata the on-site work is done, They will need updated production DNS but all are moved and connected.

Thu, Sep 5, 7:31 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
Cmjohnson reassigned T229871: relocate/reimage cloudvirt1023 with 10G interfaces from Cmjohnson to Andrew.

@Andrew the new mac is in an earlier update. The server is moved, connected to the new port and raid cfg completed...needs the dhcp file updated and ready for you to re-image.

Thu, Sep 5, 7:16 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson updated the task description for T229871: relocate/reimage cloudvirt1023 with 10G interfaces.
Thu, Sep 5, 7:15 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

B0:26:28:29:6A:E0

Thu, Sep 5, 7:01 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson reassigned T229872: relocate/reimage cloudvirt1022 with 10G interfaces from Cmjohnson to Andrew.

@Andrew this is ready for you to re-image

Thu, Sep 5, 6:50 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson updated the task description for T229872: relocate/reimage cloudvirt1022 with 10G interfaces.
Thu, Sep 5, 6:50 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson reassigned T229873: relocate/reimage cloudvirt1021 with 10G interfaces from Cmjohnson to Andrew.

@Andrew this is ready for you to re-image

Thu, Sep 5, 6:49 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson updated the task description for T229873: relocate/reimage cloudvirt1021 with 10G interfaces.
Thu, Sep 5, 6:49 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)

Wed, Sep 4

Cmjohnson added a comment to T228102: rack/setup/install cloudcephmon100[123].

@Jclark-ctr Please set up the idrac and add the mgmt dns. Let me know if you have any issues or questions. I also need the switch ports.

Wed, Sep 4, 12:10 AM · cloud-services-team (Kanban), Operations, Cloud-Services, ops-eqiad

Tue, Sep 3

Cmjohnson added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

@Ottomata All the servers are moved and all of them but cloudvirtan1003 are connected to the switch in the correct vlan. @Jclark-ctr if you are still around can you verify that cloudvirtan is connected to switch in rack d7 xe-7/0/20, please.

Tue, Sep 3, 11:58 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
Cmjohnson added a comment to T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..

@Ottomata Do you still need the 2nd port now that you're not doing the cloud thing? If so which vlan?

Tue, Sep 3, 5:38 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics

Fri, Aug 30

Cmjohnson added a comment to T231638: db1074 crashed: Broken BBU.

@wiki_willy negative, we do not have any spare BBUs lying around.

Fri, Aug 30, 5:25 PM · ops-eqiad, Operations, DBA
Cmjohnson added a comment to T230289: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only.

updated the idrac and raid f/w

Fri, Aug 30, 5:08 PM · cloud-services-team, ops-eqiad, Operations

Thu, Aug 29

Cmjohnson reassigned T228102: rack/setup/install cloudcephmon100[123] from RobH to Jclark-ctr.

@Jclark-ctr please rack 1 each in B2/B4/B7 please and update netbox

Thu, Aug 29, 4:41 PM · cloud-services-team (Kanban), Operations, Cloud-Services, ops-eqiad
Cmjohnson added a comment to T224188: rack/setup/install (3) new osd ceph nodes.

@Jclark-ctr please rack 1 each in B2/B4/B7 please and update netbox

Thu, Aug 29, 4:40 PM · ops-eqiad, Operations, cloud-services-team (Kanban), Cloud-Services
Cmjohnson moved T231525: cp1085 - IPMI not working from Procurement to Hardware Failure / Troubleshoot on the ops-eqiad board.
Thu, Aug 29, 4:36 PM · ops-eqiad, Traffic, Operations
Cmjohnson moved T230746: (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet from Backlog to Racking Tasks on the ops-eqiad board.
Thu, Aug 29, 4:36 PM · Patch-For-Review, Operations, ops-eqiad
Cmjohnson moved T231525: cp1085 - IPMI not working from Backlog to Procurement on the ops-eqiad board.
Thu, Aug 29, 4:35 PM · ops-eqiad, Traffic, Operations
Cmjohnson added a comment to T231525: cp1085 - IPMI not working.

looks like the mgmt is locked out and this server will require a hard reboot and flea power drain. please let me know when it's safe to turn the server off for 5-10 mins.

Thu, Aug 29, 4:28 PM · ops-eqiad, Traffic, Operations

Tue, Aug 27

Cmjohnson reassigned T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. from Cmjohnson to Jclark-ctr.

@Jclark-ctr Can you move these servers as evenly as you can into rows B2/B4 and B7, cable with 10G DAC cables and the mgmt cable please and update netbox and this task with their location and the port numbers you connected the servers.

Tue, Aug 27, 8:00 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
Cmjohnson moved T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. from Cloud Tasks to Hardware Failure / Troubleshoot on the ops-eqiad board.
Tue, Aug 27, 7:57 PM · Analytics-Kanban, ops-eqiad, Operations, netops, Analytics
Cmjohnson added a comment to T229871: relocate/reimage cloudvirt1023 with 10G interfaces.

@Andrew This server will require a physical move to B2, B4 or B7. I will do this one last, working on cabling 1021/1022 and updating the raid cfg so you can re-image

Tue, Aug 27, 7:57 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson reassigned T229872: relocate/reimage cloudvirt1022 with 10G interfaces from Cmjohnson to Jclark-ctr.

@Jclark-ctr Can you run 10G DAC cables in rack B7. Connect to the 10G ports on the server but do not plug into the switch. Be sure to label each cable.

Tue, Aug 27, 7:56 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson reassigned T229873: relocate/reimage cloudvirt1021 with 10G interfaces from Cmjohnson to Jclark-ctr.

@Jclark-ctr Can you run 10G DAC cables in rack B4. Connect to the 10G ports on the server but do not plug into the switch. Be sure to label each cable.

Tue, Aug 27, 7:55 PM · ops-eqiad, DC-Ops, Operations, Epic, cloud-services-team (Kanban)
Cmjohnson moved T231199: Degraded RAID on db1063 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Tue, Aug 27, 7:51 PM · DBA, ops-eqiad, Operations
Cmjohnson reassigned T230575: Degraded RAID on cloudvirt1018 from Cmjohnson to wiki_willy.

The reason for the task being declined. I verified that the failed disk is indeed 1.9TB but is a SSD. The original order and showing on the disk caddy label is for an Intel 1.6TB SSD S3610. Assigning to @wiki_willy

Tue, Aug 27, 7:51 PM · ops-eqiad, Operations
Cmjohnson added a comment to T231199: Degraded RAID on db1063.

@Marostegui Replaced the disk with one of the few remaining used spares. I did notice 2 more disks are starting to fail....you may want to speed up the decom process.

Tue, Aug 27, 7:50 PM · DBA, ops-eqiad, Operations

Fri, Aug 23

Cmjohnson added a comment to T228606: Degraded RAID on elastic1046.

I replaced the failed disk

Fri, Aug 23, 3:31 PM · Discovery-Search (Current work), ops-eqiad, Operations
Cmjohnson added a comment to T230575: Degraded RAID on cloudvirt1018.

The ticket was declined by Dell....stating that the disk we have installed are not original to the server. this requires me to investigate

Fri, Aug 23, 2:58 PM · ops-eqiad, Operations
Cmjohnson closed T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory as Resolved.

Finished the idrac setup. on-site work is complete

Fri, Aug 23, 2:57 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)

Thu, Aug 22

Cmjohnson updated the task description for T221818: Decommission labnet1001 & labnet1002.
Thu, Aug 22, 5:32 PM · Patch-For-Review, ops-eqiad, decommission, Operations
Cmjohnson added a comment to T221818: Decommission labnet1001 & labnet1002.

@Jclark-ctr did you add this to the tracking sheet?

Thu, Aug 22, 5:28 PM · Patch-For-Review, ops-eqiad, decommission, Operations

Wed, Aug 21

Cmjohnson added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

Board arrived DOA...need another one

Wed, Aug 21, 6:17 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)
Cmjohnson added a comment to T230289: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only.

The disk was replaced but from what I can tell is that the raid configuration is not accepting the new disk. When I am in the raid utility it shows that all the disks are good but the raid is missing a disk. This may need the raid config manually updated and a re-install. Let me know

Wed, Aug 21, 5:15 PM · cloud-services-team, ops-eqiad, Operations
Cmjohnson added a comment to T230289: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only.

@Bstorm can you try rebooting the server and see if the disks get back to the correct order. I know that works for analytics. Please try that...i do have a disk but I'm not sure which disk is bad

Wed, Aug 21, 4:56 PM · cloud-services-team, ops-eqiad, Operations

Aug 20 2019

Cmjohnson raised the priority of T221818: Decommission labnet1001 & labnet1002 from Normal to High.
Aug 20 2019, 7:13 PM · Patch-For-Review, ops-eqiad, decommission, Operations
Cmjohnson reassigned T221818: Decommission labnet1001 & labnet1002 from Cmjohnson to Jclark-ctr.

@Jclark-ctr Please wipe and remove these servers from the rack and update the task -- assign it back to me once done please.

Aug 20 2019, 7:13 PM · Patch-For-Review, ops-eqiad, decommission, Operations
Cmjohnson reassigned T189921: decom californium from Cmjohnson to Jclark-ctr.

Can you wipe this server and remove from the rack as soon as you can. Need the space.

Aug 20 2019, 7:02 PM · Patch-For-Review, ops-eqiad, decommission, DC-Ops, Operations
Cmjohnson added a comment to T217556: Decommission old eqiad logstash hardware hosts logstash100[456].

@Jclark-ctr has this ben done? We need the space in rack B2 so please make this a priority item. Thanks!

Aug 20 2019, 6:44 PM · observability, decommission, DC-Ops, ops-eqiad, User-herron, Operations, Wikimedia-Logstash
Cmjohnson raised the priority of T220505: Decommission iron from Normal to High.
Aug 20 2019, 6:43 PM · Cloud-VPS, ops-eqiad, decommission, Operations
Cmjohnson updated the task description for T220505: Decommission iron.
Aug 20 2019, 6:43 PM · Cloud-VPS, ops-eqiad, decommission, Operations
Cmjohnson moved T228956: decommission db1072.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Aug 20 2019, 6:36 PM · DC-Ops, ops-eqiad, decommission, Operations
Cmjohnson added a comment to T227025: (Need By: August 31) rack/setup/install (3) new zookeeper nodes.

@elukey the site specific portion is complete if you want to take over from here

Aug 20 2019, 3:14 PM · User-Elukey, Operations, ops-eqiad
Cmjohnson updated the task description for T227025: (Need By: August 31) rack/setup/install (3) new zookeeper nodes.
Aug 20 2019, 3:13 PM · User-Elukey, Operations, ops-eqiad
Cmjohnson moved T230682: Degraded RAID on db1063 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Aug 20 2019, 2:50 PM · DBA, ops-eqiad, Operations
Cmjohnson added a comment to T230682: Degraded RAID on db1063.

@Marostegui I had a used disk on-site and replace it....it's currently in rebuild

Aug 20 2019, 2:50 PM · DBA, ops-eqiad, Operations
Cmjohnson added a comment to T229452: db1114 crashed due to memory issues (server under warranty).

Swapped the DIMM B3 with A3 and B7 with A7. Powered on and cleared log. Let's see if the errors return or change,

Aug 20 2019, 2:46 PM · ops-eqiad, Operations, DBA
Cmjohnson added a comment to T230289: Degraded RAID on cloudvirt1024 -- Filesystem mounted read-only.

A ticket has been placed with Dell

Aug 20 2019, 2:37 PM · cloud-services-team, ops-eqiad, Operations
Cmjohnson added a comment to T230575: Degraded RAID on cloudvirt1018.

Another ticket has been placed with Dell

Aug 20 2019, 2:37 PM · ops-eqiad, Operations
Cmjohnson moved T230575: Degraded RAID on cloudvirt1018 from Backlog to Cloud Tasks on the ops-eqiad board.
Aug 20 2019, 2:24 PM · ops-eqiad, Operations

Aug 16 2019

Cmjohnson added a comment to T220853: VMs on cloudvirt1015 crashing - bad mainboard/memory.

Dell approved my ticket. I talked to the technician today and he will be
out Monday morning to replace the motherboard.

Aug 16 2019, 2:57 PM · Operations, ops-eqiad, DC-Ops, User-Zppix, cloud-services-team (Kanban)

Aug 15 2019

Cmjohnson added a comment to T230518: elastic1017 lost network after reboot.

I will add that this server is out of warranty and would require a motherboard replacement if it is the nic. We typically do not do this after the warranty period and the host should be decommissioned.

Aug 15 2019, 5:42 PM · ops-eqiad, DC-Ops, Operations, Discovery-Search (Current work)