Page MenuHomePhabricator

Cmjohnson (cmjohnson)
User

Projects (11)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 10:22 PM (267 w, 17 h)
Availability
Available
IRC Nick
cmjohnson1
LDAP User
Cmjohnson
MediaWiki User
Unknown

Recent Activity

Yesterday

Cmjohnson reassigned T228926: rack/setup/instal (4) CI ganeti nodes from Cmjohnson to akosiaris.

@akosiaris I am assigning this to you, the initial puppet run has been completed. I removed the ops-eqiad tag

Tue, Jan 28, 10:01 PM · Operations
Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Tue, Jan 28, 9:59 PM · Operations
Cmjohnson reassigned T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet from Cmjohnson to elukey.

@elukey these are ready for you to implement. I am removing ops-eqiad tag.

Tue, Jan 28, 9:51 PM · serviceops, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Tue, Jan 28, 9:49 PM · serviceops, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Tue, Jan 28, 4:47 PM · serviceops, Operations
Cmjohnson added a comment to T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.

All three have the OS installed. Still need the initial puppet run

Tue, Jan 28, 4:47 PM · serviceops, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Tue, Jan 28, 4:39 PM · serviceops, Operations

Mon, Jan 27

Cmjohnson added a comment to T228926: rack/setup/instal (4) CI ganeti nodes.

the OS has been installed on 4 of these, initial puppet run has not been done

Mon, Jan 27, 4:46 PM · Operations
Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Mon, Jan 27, 4:46 PM · Operations
Cmjohnson added a comment to T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.

The mac addresses were out of order...updated the file and attempting to run the installer

Mon, Jan 27, 4:46 PM · serviceops, Operations
Cmjohnson added a comment to T242511: Degraded RAID on ms-be1039.

@godog I replaced the disk, please see what you need to do to add it back to the raid. Thanks!

Mon, Jan 27, 4:04 PM · SRE-swift-storage, ops-eqiad, Operations

Thu, Jan 23

Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Thu, Jan 23, 7:20 PM · Operations
Cmjohnson added a comment to T241359: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet.

@Marostegui I do not know yet, there are several racking/setup tasks that I am trying to get through. I need to check with @Jclark-ctr and see if they're even in racks yet.

Thu, Jan 23, 6:51 PM · DBA, ops-eqiad, Operations
Cmjohnson added a comment to T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.

not hitting the installer...still working on them

Thu, Jan 23, 6:50 PM · serviceops, Operations
Cmjohnson added a comment to T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Thu, Jan 23, 6:47 PM · serviceops, Operations

Wed, Jan 22

Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Wed, Jan 22, 10:21 PM · Operations
Cmjohnson claimed T228926: rack/setup/instal (4) CI ganeti nodes.
Wed, Jan 22, 10:17 PM · Operations
Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Wed, Jan 22, 10:16 PM · Operations
Cmjohnson updated subscribers of T239250: setup/install cescout1001.eqiad.wmnet.

@ssingh Do you know what type of partitioning this gets? maybe @RobH knows?

Wed, Jan 22, 9:43 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Wed, Jan 22, 8:40 PM · serviceops, Operations

Tue, Jan 21

Cmjohnson added a comment to T242511: Degraded RAID on ms-be1039.

the disk was not delivered...HPE stated

Tue, Jan 21, 10:34 PM · SRE-swift-storage, ops-eqiad, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Tue, Jan 21, 8:01 PM · serviceops, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Tue, Jan 21, 7:55 PM · serviceops, Operations
Cmjohnson removed a project from T234068: (ASAP) rack/setup/install frban1001.frack.eqiad.wmnet: ops-eqiad.
Tue, Jan 21, 6:30 PM · Operations, fundraising-tech-ops
Cmjohnson reassigned T234068: (ASAP) rack/setup/install frban1001.frack.eqiad.wmnet from Cmjohnson to Jgreen.
Tue, Jan 21, 6:30 PM · Operations, fundraising-tech-ops
Cmjohnson updated the task description for T234068: (ASAP) rack/setup/install frban1001.frack.eqiad.wmnet.
Tue, Jan 21, 6:29 PM · Operations, fundraising-tech-ops
Cmjohnson added a comment to T234068: (ASAP) rack/setup/install frban1001.frack.eqiad.wmnet.

these are cabled, running into an issue on the switch cfg removing from disabled vlan. connected fasw1-c ge-0/0/0 (eth 1) and fasw2-c ge-1/0/0 (eth2)

Tue, Jan 21, 6:02 PM · Operations, fundraising-tech-ops
Cmjohnson updated the task description for T234068: (ASAP) rack/setup/install frban1001.frack.eqiad.wmnet.
Tue, Jan 21, 6:02 PM · Operations, fundraising-tech-ops
Cmjohnson closed T237582: frqueue1001 system battery needs replacement as Resolved.

Replaced the CMOS battery, reset the date/time in bios. resolving

Tue, Jan 21, 5:26 PM · ops-eqiad, Operations

Thu, Jan 16

Cmjohnson added a comment to T242511: Degraded RAID on ms-be1039.

a new disk has been sent

Thu, Jan 16, 7:19 PM · SRE-swift-storage, ops-eqiad, Operations

Wed, Jan 15

Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Wed, Jan 15, 8:05 PM · Operations
Cmjohnson added a comment to T242511: Degraded RAID on ms-be1039.

AHS log sent to HPE as per their request

Wed, Jan 15, 7:32 PM · SRE-swift-storage, ops-eqiad, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Wed, Jan 15, 7:14 PM · serviceops, Operations
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Wed, Jan 15, 3:21 PM · serviceops, Operations

Tue, Jan 14

Cmjohnson renamed T239250: setup/install cescout1001.eqiad.wmnet from setup/install censcout1001.eqiad.wmnet to setup/install cescout1001.eqiad.wmnet.
Tue, Jan 14, 9:01 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson renamed T239250: setup/install cescout1001.eqiad.wmnet from setup/install censorship1001.eqiad.wmnet to setup/install censcout1001.eqiad.wmnet.
Tue, Jan 14, 9:00 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson added a comment to T239250: setup/install cescout1001.eqiad.wmnet.

updated hostname to cescout1001

Tue, Jan 14, 9:00 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson closed T241155: Relabel labmon1001.eqiad.wmnet to cloudmetrics1001eqiad.wmnet and labmon1002.eqiad.wmnet to cloudmetrics1002eqiad.wmnet as Resolved.

updated physical label and switch label

Tue, Jan 14, 8:48 PM · DC-Ops, ops-eqiad, Operations, cloud-services-team
Cmjohnson updated the task description for T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
Tue, Jan 14, 8:41 PM · serviceops, Operations
Cmjohnson claimed T228924: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet.

The mgmt passwords have been updated. Expect these to be ready this week

Tue, Jan 14, 8:22 PM · ops-eqiad, vm-requests, Operations
Cmjohnson added a comment to T228926: rack/setup/instal (4) CI ganeti nodes.

The mgmt passwords have been updated.

Tue, Jan 14, 8:21 PM · Operations
Cmjohnson added a comment to T236327: replace onboard NIC in kafka-jumbo100[1-6].

@Jclark-ctr where are you with these?

Tue, Jan 14, 7:05 PM · ops-eqiad, Operations, Analytics, User-Elukey
Cmjohnson claimed T237582: frqueue1001 system battery needs replacement.
Tue, Jan 14, 7:00 PM · ops-eqiad, Operations
Cmjohnson added a comment to T237582: frqueue1001 system battery needs replacement.

@Jgreen I have the batteries...when can we schedule to do this?

Tue, Jan 14, 6:59 PM · ops-eqiad, Operations
Cmjohnson moved T242511: Degraded RAID on ms-be1039 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Tue, Jan 14, 6:57 PM · SRE-swift-storage, ops-eqiad, Operations
Cmjohnson claimed T242511: Degraded RAID on ms-be1039.
Tue, Jan 14, 6:57 PM · SRE-swift-storage, ops-eqiad, Operations
Cmjohnson added a comment to T242511: Degraded RAID on ms-be1039.

I am working on this now, a ticket has been created with HPE

Tue, Jan 14, 6:57 PM · SRE-swift-storage, ops-eqiad, Operations

Tue, Jan 7

Cmjohnson moved T241155: Relabel labmon1001.eqiad.wmnet to cloudmetrics1001eqiad.wmnet and labmon1002.eqiad.wmnet to cloudmetrics1002eqiad.wmnet from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Tue, Jan 7, 4:37 PM · DC-Ops, ops-eqiad, Operations, cloud-services-team

Dec 10 2019

Cmjohnson closed T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet as Resolved.

Thanks, fixed!

Dec 10 2019, 5:14 PM · ops-eqiad, User-fgiunchedi, Operations

Nov 26 2019

Cmjohnson moved T238297: decommission db1067.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Nov 26 2019, 6:09 PM · DC-Ops, decommission, ops-eqiad, Operations
Cmjohnson moved T239217: Degraded RAID on dbstore1003 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 26 2019, 6:08 PM · Analytics, ops-eqiad, Operations
Cmjohnson assigned T239217: Degraded RAID on dbstore1003 to Jclark-ctr.

I created a self-dispatch ticket. You have successfully submitted request SR1004377941. Assigning to @Jclark-ctr since I will be out of the area.

Nov 26 2019, 6:08 PM · Analytics, ops-eqiad, Operations
Cmjohnson closed T239110: Asset tag remaining cablemgmt in eqiad as Resolved.

@faidon All should have asset tags now and netbox updated.

Nov 26 2019, 5:52 PM · ops-eqiad, Operations, DC-Ops
Cmjohnson added a comment to T228924: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet.

@Jclark-ctr can you fix the mgmt password for these please.

Nov 26 2019, 12:06 AM · ops-eqiad, vm-requests, Operations

Nov 25 2019

Cmjohnson closed Unknown Object (Task), a subtask of T237582: frqueue1001 system battery needs replacement, as Resolved.
Nov 25 2019, 9:16 PM · ops-eqiad, Operations
Cmjohnson added a comment to T239110: Asset tag remaining cablemgmt in eqiad.

Yes, I misread the task....I will update and resolve once completed

Nov 25 2019, 8:59 PM · ops-eqiad, Operations, DC-Ops
Cmjohnson closed T239110: Asset tag remaining cablemgmt in eqiad as Resolved.

asset tags have been added to all and netbox updated

Nov 25 2019, 7:11 PM · ops-eqiad, Operations, DC-Ops
Cmjohnson added a comment to T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet.

@fgiunchedi These are ready for you for implementation. I removed the ops-eqiad tag. if you have an issue please assign to me and add the ops-eqiad tag back

Nov 25 2019, 7:05 PM · ops-eqiad, User-fgiunchedi, Operations
Cmjohnson removed a project from T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet: ops-eqiad.
Nov 25 2019, 7:04 PM · ops-eqiad, User-fgiunchedi, Operations
Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Nov 25 2019, 5:49 PM · Operations
Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Nov 25 2019, 5:47 PM · Operations
Cmjohnson added a comment to T228926: rack/setup/instal (4) CI ganeti nodes.

production dns was added here https://gerrit.wikimedia.org/r/#/c/operations/dns/+/552812/

Nov 25 2019, 5:47 PM · Operations
Cmjohnson closed T236601: Degraded RAID on elastic1039 as Resolved.

Resolving this task for the failed raid, @Gehel you may want to create a new one for the re-image.

Nov 25 2019, 1:25 PM · Discovery-Search (Current work), ops-eqiad, Operations
Cmjohnson added a comment to T227867: mw1239 memory errors .

mw1239 is well out of warranty and is over 5 years old. Historically we decom these host at this stage in their life. We also have a several new MW servers waiting to be racked and will be on-line soon. If you want DIMM, we need to purchase a new stick.

Nov 25 2019, 1:24 PM · ops-eqiad, DC-Ops, Operations, serviceops

Nov 18 2019

Cmjohnson moved T230575: Degraded RAID on cloudvirt1018 from Cloud Tasks to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 18 2019, 4:55 PM · ops-eqiad, Operations
Cmjohnson claimed T232137: (ASAP) rack/setup/install frnetmon1001.frack.eqiad.wmnet.

I will work on network setup

Nov 18 2019, 4:40 PM · fundraising-tech-ops, ops-eqiad, Operations
Cmjohnson reassigned T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet from RobH to Jclark-ctr.

@Jclark-ctr the mgmt password for ms-be1057 is still not working, can you try again.

Nov 18 2019, 4:16 PM · ops-eqiad, User-fgiunchedi, Operations

Nov 15 2019

Cmjohnson updated the task description for T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet.
Nov 15 2019, 4:08 PM · ops-eqiad, User-fgiunchedi, Operations

Nov 14 2019

Cmjohnson added a comment to T236331: Degraded RAID on cloudvirt1018.

Looks like this was approved this time around, @Jclark-ctr please keep an eye out for the disk in receiving

Nov 14 2019, 7:02 PM · ops-eqiad, Operations

Nov 13 2019

Cmjohnson reassigned T224188: rack/setup/install (3) new osd ceph nodes from Cmjohnson to JHedden.

@JHedden What is the status of these servers, it looks like most everything is finished but the checkboxes are not complete. I believe the ops-eqiad work is finished so I am removing the tag. Please resolve this task when you see fit.

Nov 13 2019, 5:54 PM · Operations, cloud-services-team (Kanban), Cloud-Services
Cmjohnson reassigned T228926: rack/setup/instal (4) CI ganeti nodes from Cmjohnson to Jclark-ctr.

@Jclark-ctr Can you get the network ports, please add to the task. Thanks

Nov 13 2019, 5:50 PM · Operations
Cmjohnson updated the task description for T228926: rack/setup/instal (4) CI ganeti nodes.
Nov 13 2019, 5:49 PM · Operations
Cmjohnson updated the task description for T234068: (ASAP) rack/setup/install frban1001.frack.eqiad.wmnet.
Nov 13 2019, 5:27 PM · Operations, fundraising-tech-ops
Cmjohnson updated the task description for T232137: (ASAP) rack/setup/install frnetmon1001.frack.eqiad.wmnet.
Nov 13 2019, 5:23 PM · fundraising-tech-ops, ops-eqiad, Operations
Cmjohnson closed T215332: reimage WMF6937/mw1298, a subtask of T192457: Reallocate former image scalers, as Resolved.
Nov 13 2019, 5:12 PM · Patch-For-Review, serviceops, Operations
Cmjohnson closed T215332: reimage WMF6937/mw1298, a subtask of T195623: request to assign wmf6937 (mw1298, former imagescaler) (now: wmf4727) as phab1002, as Resolved.
Nov 13 2019, 5:12 PM · Patch-For-Review, Operations, hardware-requests
Cmjohnson closed T215332: reimage WMF6937/mw1298, a subtask of T196019: setup/install phab1002(WMF4727), as Resolved.
Nov 13 2019, 5:12 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson closed T215332: reimage WMF6937/mw1298 as Resolved.

The label has been done

Nov 13 2019, 5:12 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson updated the task description for T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet.
Nov 13 2019, 5:01 PM · ops-eqiad, User-fgiunchedi, Operations
Cmjohnson added a comment to T236327: replace onboard NIC in kafka-jumbo100[1-6].

These are projected to be in eqiad Dec 5th

Nov 13 2019, 4:23 PM · ops-eqiad, Operations, Analytics, User-Elukey
Cmjohnson closed T235124: Move kafka100[123] to logstash102[012] as Resolved.

labels updated

Nov 13 2019, 4:23 PM · DC-Ops, Operations, ops-eqiad
Cmjohnson added a comment to T230575: Degraded RAID on cloudvirt1018.

I created another dispatch ticket with Dell and will try and resolve this through them. If not a phone call will have to be made.

Nov 13 2019, 4:16 PM · ops-eqiad, Operations
Cmjohnson added a comment to T236601: Degraded RAID on elastic1039.

@Gehel The disk is here, Can this be done anytime or does this need to be coordinated?

Nov 13 2019, 3:57 PM · Discovery-Search (Current work), ops-eqiad, Operations
Cmjohnson moved T146455: Decommission labsdb1002 from Backlog to Decommission on the ops-eqiad board.
Nov 13 2019, 3:56 PM · hardware-requests, Patch-For-Review, ops-eqiad, Operations
Cmjohnson moved T215332: reimage WMF6937/mw1298 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:56 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson moved T233071: Decommission db1066.eqiad.wmnet from Backlog to Decommission on the ops-eqiad board.
Nov 13 2019, 3:55 PM · DC-Ops, ops-eqiad, decommission, Operations
Cmjohnson moved T233578: hw troubleshooting: Memory correctable errors -EDAC- for elastic1029.eqiad.wmnet from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:55 PM · Operations, ops-eqiad, DC-Ops
Cmjohnson added a comment to T233578: hw troubleshooting: Memory correctable errors -EDAC- for elastic1029.eqiad.wmnet.

@Mathew.onipe elastic1029 is over 2 years out of warranty. The DIMM can be reseated, does this server need scheduled downtime or can it be taken down anytime?

Nov 13 2019, 3:55 PM · Operations, ops-eqiad, DC-Ops
Cmjohnson moved T235124: Move kafka100[123] to logstash102[012] from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:53 PM · DC-Ops, Operations, ops-eqiad
Cmjohnson moved T236327: replace onboard NIC in kafka-jumbo100[1-6] from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:53 PM · ops-eqiad, Operations, Analytics, User-Elukey
Cmjohnson moved T236331: Degraded RAID on cloudvirt1018 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:53 PM · ops-eqiad, Operations
Cmjohnson moved T236601: Degraded RAID on elastic1039 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:53 PM · Discovery-Search (Current work), ops-eqiad, Operations
Cmjohnson moved T237133: analytics1062 lost one of its power supplies from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:53 PM · Analytics, Operations, Analytics-Cluster, ops-eqiad
Cmjohnson moved T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet from Cloud Tasks to Racking Tasks on the ops-eqiad board.
Nov 13 2019, 3:53 PM · ops-eqiad, User-fgiunchedi, Operations
Cmjohnson moved T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet from Backlog to Cloud Tasks on the ops-eqiad board.
Nov 13 2019, 3:52 PM · ops-eqiad, User-fgiunchedi, Operations
Cmjohnson moved T237582: frqueue1001 system battery needs replacement from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Nov 13 2019, 3:52 PM · ops-eqiad, Operations
Cmjohnson reassigned T228102: rack/setup/install cloudcephmon100[123] from Cmjohnson to Bstorm.

@Bstorm assigning to you to update netbox once the systems are online. I am removing the ops-eqiad tag. If you have an issue please add tag back. Thanks

Nov 13 2019, 3:52 PM · cloud-services-team (Kanban), Operations, Cloud-Services
Cmjohnson updated the task description for T228102: rack/setup/install cloudcephmon100[123].
Nov 13 2019, 3:50 PM · cloud-services-team (Kanban), Operations, Cloud-Services
Cmjohnson added a comment to T228102: rack/setup/install cloudcephmon100[123].

The mgmt issue seems to have been resolved, was this done by @Jclark-ctr I do not see an update

Nov 13 2019, 3:48 PM · cloud-services-team (Kanban), Operations, Cloud-Services

Nov 11 2019

Cmjohnson added a comment to T237438: (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet.

@Jclark-ctr I am not able to login to mgmt, can you verify that the IP, Gateway and subnet are correct.

Nov 11 2019, 2:18 PM · ops-eqiad, User-fgiunchedi, Operations