Cmjohnson (cmjohnson)
User

Projects (12)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 10:22 PM (162 w, 1 h)
Availability
Available
IRC Nick
cmjohnson1
LDAP User
Cmjohnson
MediaWiki User
Unknown

Recent Activity

Thu, Jan 18

Cmjohnson created T185226: Decommission host erbium.
Thu, Jan 18, 4:29 PM · hardware-requests, Operations, Patch-For-Review

Tue, Jan 16

Cmjohnson added a comment to T184888: Replace codfw x1 master (db2033) (WAS: Failed BBU on db2033 (x1 master)).

@RobH no I do not have any spares at this time.

Tue, Jan 16, 4:59 PM · Patch-For-Review, DBA

Fri, Jan 12

Cmjohnson updated the task description for T184293: rack/setup/install lvs101[3-6].
Fri, Jan 12, 6:22 PM · Patch-For-Review, ops-eqiad, Operations, Traffic
Cmjohnson moved T184293: rack/setup/install lvs101[3-6] from Up next to Being worked on on the ops-eqiad board.
Fri, Jan 12, 6:00 PM · Patch-For-Review, ops-eqiad, Operations, Traffic

Thu, Jan 11

Cmjohnson closed T183895: Decommission mw1180-1200 as Resolved.
Thu, Jan 11, 6:14 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson closed T183895: Decommission mw1180-1200, a subtask of T165519: rack and setup mw1307-1348 , as Resolved.
Thu, Jan 11, 6:14 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson moved T184160: db1059 BBU issues from Up next to Being worked on on the ops-eqiad board.
Thu, Jan 11, 6:14 PM · ops-eqiad, Operations, DBA
Cmjohnson added a comment to T184160: db1059 BBU issues.

Swapped the bbu....leaving this open to confirm everything is okay.

Thu, Jan 11, 6:14 PM · ops-eqiad, Operations, DBA
Cmjohnson moved T165519: rack and setup mw1307-1348 from Being worked on to Blocked on the ops-eqiad board.
Thu, Jan 11, 6:13 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson added a comment to T183895: Decommission mw1180-1200.

removed from rack and racktables updated.

Thu, Jan 11, 6:13 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson updated the task description for T183895: Decommission mw1180-1200.
Thu, Jan 11, 6:13 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson moved T184722: Hardware check on mw1271 from Backlog to Being worked on on the ops-eqiad board.
Thu, Jan 11, 6:13 PM · Operations, ops-eqiad
Cmjohnson added a comment to T184722: Hardware check on mw1271.

I swapped the DIMM from A1 to B1 to see if the error persists on the DIMM bank or if it stays with the DIMM.

Thu, Jan 11, 6:12 PM · Operations, ops-eqiad
Cmjohnson added a comment to T184722: Hardware check on mw1271.

The server has a DIMM error on A1

Thu, Jan 11, 6:04 PM · Operations, ops-eqiad
Cmjohnson assigned T165519: rack and setup mw1307-1348 to elukey.

assigning this to @elukey to complete installs.

Thu, Jan 11, 6:01 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson added a comment to T165519: rack and setup mw1307-1348 .

the final 10 servers have been racked. 9 of 10 are now ready to be installed. There is an issue with the idrac setup on mw1340 but will be addressed today.
The 9 are ready for install if you want to tackle now or wait for the mw1340

Thu, Jan 11, 4:32 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson updated the task description for T165519: rack and setup mw1307-1348 .
Thu, Jan 11, 4:30 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad

Wed, Jan 10

Cmjohnson updated the task description for T183895: Decommission mw1180-1200.
Wed, Jan 10, 3:55 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson added a comment to T184160: db1059 BBU issues.

@Marostegui II have a used spare battery we can swap this out with. LMK when you want to schedule this

Wed, Jan 10, 3:50 PM · ops-eqiad, Operations, DBA
Cmjohnson closed T181784: Decommission db104[67] as Resolved.
Wed, Jan 10, 3:49 PM · Analytics, hardware-requests, Operations, ops-eqiad
Cmjohnson closed T181378: Decommission db1021 as Resolved.
Wed, Jan 10, 3:49 PM · Patch-For-Review, hardware-requests, ops-eqiad, Operations, DBA
Cmjohnson closed T181378: Decommission db1021, a subtask of T134476: Decommission old coredb machines (<=db1050), as Resolved.
Wed, Jan 10, 3:49 PM · Patch-For-Review, Goal, Operations, DBA
Cmjohnson closed T174763: Decommission db1026 as Resolved.
Wed, Jan 10, 3:48 PM · hardware-requests, Patch-For-Review, ops-eqiad, Operations, DBA
Cmjohnson closed T174806: Decommission db1045 as Resolved.
Wed, Jan 10, 3:48 PM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Cmjohnson closed T175264: Decommission db1049, a subtask of T134476: Decommission old coredb machines (<=db1050), as Resolved.
Wed, Jan 10, 3:48 PM · Patch-For-Review, Goal, Operations, DBA
Cmjohnson closed T175264: Decommission db1049 as Resolved.
Wed, Jan 10, 3:48 PM · hardware-requests, ops-eqiad, Operations, DBA
Cmjohnson closed T173570: Decommission db1015 as Resolved.
Wed, Jan 10, 3:48 PM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Cmjohnson closed T173570: Decommission db1015, a subtask of T134476: Decommission old coredb machines (<=db1050), as Resolved.
Wed, Jan 10, 3:48 PM · Patch-For-Review, Goal, Operations, DBA
Cmjohnson closed T175679: Decommission db1048 (was Move m3 slave to db1059) as Resolved.
Wed, Jan 10, 3:47 PM · hardware-requests, Operations, ops-eqiad, Phabricator, DBA
Cmjohnson closed T175679: Decommission db1048 (was Move m3 slave to db1059), a subtask of T134476: Decommission old coredb machines (<=db1050), as Resolved.
Wed, Jan 10, 3:47 PM · Patch-For-Review, Goal, Operations, DBA
Cmjohnson closed T181696: Decommission db1044 as Resolved.
Wed, Jan 10, 3:47 PM · hardware-requests, ops-eqiad, Patch-For-Review, Operations, DBA
Cmjohnson closed T181696: Decommission db1044, a subtask of T134476: Decommission old coredb machines (<=db1050), as Resolved.
Wed, Jan 10, 3:47 PM · Patch-For-Review, Goal, Operations, DBA
Cmjohnson moved T183771: dbstore1002 possibly MEMORY issues from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 10, 3:47 PM · ops-eqiad, Analytics-Kanban, Operations
Cmjohnson moved T183895: Decommission mw1180-1200 from Up next to Being worked on on the ops-eqiad board.
Wed, Jan 10, 3:47 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad

Tue, Jan 9

Cmjohnson added a comment to T183896: Degraded RAID on ms-be1033.

@fgiunchedi The disk has been replaced...please resolve this after confirmation

Tue, Jan 9, 6:28 PM · ops-eqiad, Operations
Cmjohnson reassigned T183935: rack/setup/install notebook[34] from Cmjohnson to RobH.
Tue, Jan 9, 4:13 PM · Patch-For-Review, ops-eqiad, Analytics, Operations
Cmjohnson added a comment to T183935: rack/setup/install notebook[34].

All the on-site work has been completed, production dns added and install server. @RobH can you look into the partman recipe and complete the installs please.

Tue, Jan 9, 4:13 PM · Patch-For-Review, ops-eqiad, Analytics, Operations
Cmjohnson updated the task description for T183935: rack/setup/install notebook[34].
Tue, Jan 9, 4:12 PM · Patch-For-Review, ops-eqiad, Analytics, Operations
Cmjohnson added a comment to T183937: rack/setup/install labvirt102[12].

These are hitting the install server but not receiving the image. Chasemp or robh can you take a look at this please. They were received with 10G Nics that I turned off and set the 1GB Nic to pxe. Please verify that everything looks okay.

Tue, Jan 9, 3:37 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson moved T184160: db1059 BBU issues from Backlog to Up next on the ops-eqiad board.
Tue, Jan 9, 3:20 PM · ops-eqiad, Operations, DBA
Cmjohnson moved T184262: Decommission db1039 from Backlog to Decommission on the ops-eqiad board.
Tue, Jan 9, 3:20 PM · hardware-requests, Operations, ops-eqiad, Patch-For-Review, DBA
Cmjohnson moved T184293: rack/setup/install lvs101[3-6] from Backlog to Up next on the ops-eqiad board.
Tue, Jan 9, 3:20 PM · Patch-For-Review, ops-eqiad, Operations, Traffic
Cmjohnson added a comment to T183896: Degraded RAID on ms-be1033.

Disk was shipped should be here today

Tue, Jan 9, 3:19 PM · ops-eqiad, Operations

Mon, Jan 8

Cmjohnson moved T165519: rack and setup mw1307-1348 from Blocked to Being worked on on the ops-eqiad board.
Mon, Jan 8, 8:40 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson updated the task description for T183937: rack/setup/install labvirt102[12].
Mon, Jan 8, 8:24 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson added a comment to T183771: dbstore1002 possibly MEMORY issues.

@elukey. Let’s schedule for 1500UTC tomorrow.

Mon, Jan 8, 1:48 PM · ops-eqiad, Analytics-Kanban, Operations
Cmjohnson added a comment to T177374: decom wtp1001-wtp1024.

This was me last week, these servers have not gone through the decom steps
yet and still have puppet running.

Mon, Jan 8, 1:47 PM · Parsoid, Patch-For-Review, ops-eqiad, DC-Ops, Operations
Cmjohnson added a comment to T183771: dbstore1002 possibly MEMORY issues.

@elukey yes, the server will need to be powered down for a minute to unlock
the Idrac. Can we do this right after meeting today or do you want to
schedule for tomorrow or Wednesday?

Mon, Jan 8, 1:45 PM · ops-eqiad, Analytics-Kanban, Operations

Fri, Jan 5

Cmjohnson closed T184196: cp1066's DRAC not responding to SSH as Resolved.

This did not need to be powered off. I was able to reset mgmt via the idrac using the racadmin racreset command. I verified using an ipmi command

Fri, Jan 5, 5:46 PM · Operations, ops-eqiad, DC-Ops

Wed, Jan 3

Cmjohnson updated the task description for T183937: rack/setup/install labvirt102[12].
Wed, Jan 3, 9:54 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson updated the task description for T183935: rack/setup/install notebook[34].
Wed, Jan 3, 9:52 PM · Patch-For-Review, ops-eqiad, Analytics, Operations
Cmjohnson added a comment to T183896: Degraded RAID on ms-be1033.

a case has been opened with HP Support.

Wed, Jan 3, 8:03 PM · ops-eqiad, Operations
Cmjohnson moved T183771: dbstore1002 possibly MEMORY issues from Up next to Being worked on on the ops-eqiad board.
Wed, Jan 3, 7:21 PM · ops-eqiad, Analytics-Kanban, Operations
Cmjohnson moved T183896: Degraded RAID on ms-be1033 from Up next to Being worked on on the ops-eqiad board.
Wed, Jan 3, 7:21 PM · ops-eqiad, Operations
Cmjohnson moved T184053: Degraded RAID on ms-be1013 from Up next to Being worked on on the ops-eqiad board.
Wed, Jan 3, 7:21 PM · ops-eqiad, Operations
Cmjohnson moved T175679: Decommission db1048 (was Move m3 slave to db1059) from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:21 PM · hardware-requests, Operations, ops-eqiad, Phabricator, DBA
Cmjohnson moved T175264: Decommission db1049 from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:21 PM · hardware-requests, ops-eqiad, Operations, DBA
Cmjohnson moved T174806: Decommission db1045 from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:21 PM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Cmjohnson moved T174763: Decommission db1026 from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:21 PM · hardware-requests, Patch-For-Review, ops-eqiad, Operations, DBA
Cmjohnson moved T181378: Decommission db1021 from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:20 PM · Patch-For-Review, hardware-requests, ops-eqiad, Operations, DBA
Cmjohnson moved T173570: Decommission db1015 from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:20 PM · Patch-For-Review, hardware-requests, ops-eqiad, DBA, Operations
Cmjohnson moved T181784: Decommission db104[67] from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:20 PM · Analytics, hardware-requests, Operations, ops-eqiad
Cmjohnson moved T181696: Decommission db1044 from Being worked on to Blocked on the ops-eqiad board.
Wed, Jan 3, 7:20 PM · hardware-requests, ops-eqiad, Patch-For-Review, Operations, DBA
Cmjohnson closed T178162: Decommission db1050 as Resolved.
Wed, Jan 3, 7:15 PM · hardware-requests, Operations, ops-eqiad, Patch-For-Review, DBA
Cmjohnson closed T178162: Decommission db1050, a subtask of T134476: Decommission old coredb machines (<=db1050), as Resolved.
Wed, Jan 3, 7:15 PM · Patch-For-Review, Goal, Operations, DBA
Cmjohnson updated the task description for T178162: Decommission db1050.
Wed, Jan 3, 7:15 PM · hardware-requests, Operations, ops-eqiad, Patch-For-Review, DBA
Cmjohnson added a comment to T184053: Degraded RAID on ms-be1013.

@fgiunchedi The disk has been replaced (replaced with a larger capacity disk 4TB, all I had on-site for spare) and added back. Resolve once confirmed good.

Wed, Jan 3, 7:14 PM · ops-eqiad, Operations
Cmjohnson moved T183935: rack/setup/install notebook[34] from Backlog to Being worked on on the ops-eqiad board.
Wed, Jan 3, 4:06 PM · Patch-For-Review, ops-eqiad, Analytics, Operations
Cmjohnson moved T183937: rack/setup/install labvirt102[12] from Backlog to Being worked on on the ops-eqiad board.
Wed, Jan 3, 4:06 PM · Patch-For-Review, ops-eqiad, Operations
Cmjohnson moved T184053: Degraded RAID on ms-be1013 from Backlog to Up next on the ops-eqiad board.
Wed, Jan 3, 4:06 PM · ops-eqiad, Operations

Tue, Jan 2

Cmjohnson closed T181952: Requesting access to EventLogging data for Vinitha as Resolved.

Your user exists on stat1006 now and expires on 31/3/2018

Tue, Jan 2, 7:55 PM · Patch-For-Review, AICaptcha, WMF-NDA-Requests, Ops-Access-Requests, Operations
Cmjohnson moved T181121: Hardware errors on ganeti1005- ganeti1008 from Being worked on to Blocked on the ops-eqiad board.
Tue, Jan 2, 5:12 PM · ops-eqiad, Operations
Cmjohnson moved T182896: Rack and setup db1113 and db1114 from Being worked on to Blocked on the ops-eqiad board.
Tue, Jan 2, 5:12 PM · Patch-For-Review, ops-eqiad, DBA, Operations
Cmjohnson moved T183708: Degraded RAID on db1001 from Up next to Being worked on on the ops-eqiad board.
Tue, Jan 2, 5:12 PM · DBA, ops-eqiad, Operations
Cmjohnson added a comment to T183708: Degraded RAID on db1001.

Disk Swapped

Tue, Jan 2, 5:11 PM · DBA, ops-eqiad, Operations
Cmjohnson moved T182556: Decommission db1034 from Backlog to Decommission on the ops-eqiad board.
Tue, Jan 2, 4:09 PM · hardware-requests, ops-eqiad, Patch-For-Review, Operations, DBA
Cmjohnson moved T182805: Complete decom process for server caesium from Backlog to Decommission on the ops-eqiad board.
Tue, Jan 2, 4:09 PM · DC-Ops, ops-eqiad, Operations
Cmjohnson moved T182955: Decommission kafka1018 from Backlog to Decommission on the ops-eqiad board.
Tue, Jan 2, 4:09 PM · Analytics, Operations, ops-eqiad
Cmjohnson moved T183585: Rack/cable/configure asw2-a/b/c-eqiad switch stack from Backlog to Up next on the ops-eqiad board.
Tue, Jan 2, 4:08 PM · Operations, ops-eqiad
Cmjohnson moved T183708: Degraded RAID on db1001 from Backlog to Up next on the ops-eqiad board.
Tue, Jan 2, 4:08 PM · DBA, ops-eqiad, Operations
Cmjohnson moved T183771: dbstore1002 possibly MEMORY issues from Backlog to Up next on the ops-eqiad board.
Tue, Jan 2, 4:08 PM · ops-eqiad, Analytics-Kanban, Operations
Cmjohnson moved T183895: Decommission mw1180-1200 from Backlog to Up next on the ops-eqiad board.
Tue, Jan 2, 4:08 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Cmjohnson moved T183896: Degraded RAID on ms-be1033 from Backlog to Up next on the ops-eqiad board.
Tue, Jan 2, 4:08 PM · ops-eqiad, Operations
Cmjohnson moved T183209: decom uranium from Backlog to Decommission on the ops-eqiad board.
Tue, Jan 2, 4:08 PM · Patch-For-Review, hardware-requests, ops-eqiad, monitoring, Technical-Debt, Operations
Cmjohnson moved T183390: unrack/decom pfw1-eqiad and pfw2-eqiad from Backlog to Up next on the ops-eqiad board.
Tue, Jan 2, 4:08 PM · hardware-requests, netops, ops-eqiad, Operations

Dec 20 2017

Cmjohnson added a comment to T179640: mw1191 ipmi-sel cpu errors.

@Joe this was a host identified for decommission and is well out of warranty. There is little I can do to fix. I had assumed that the replacements would have been installed by now.

Dec 20 2017, 3:02 PM · Operations, ops-eqiad
Cmjohnson added a comment to T182896: Rack and setup db1113 and db1114.

@Marostegui These are ready for installs.

Dec 20 2017, 12:20 AM · Patch-For-Review, ops-eqiad, DBA, Operations
Cmjohnson updated the task description for T182896: Rack and setup db1113 and db1114.
Dec 20 2017, 12:19 AM · Patch-For-Review, ops-eqiad, DBA, Operations

Dec 19 2017

Cmjohnson moved T182896: Rack and setup db1113 and db1114 from Backlog to Being worked on on the ops-eqiad board.
Dec 19 2017, 8:21 PM · Patch-For-Review, ops-eqiad, DBA, Operations
Cmjohnson updated the task description for T178162: Decommission db1050.
Dec 19 2017, 8:20 PM · hardware-requests, Operations, ops-eqiad, Patch-For-Review, DBA
Cmjohnson updated the task description for T175264: Decommission db1049.
Dec 19 2017, 8:19 PM · hardware-requests, ops-eqiad, DBA, Operations
Cmjohnson updated the task description for T174806: Decommission db1045.
Dec 19 2017, 8:19 PM · Patch-For-Review, hardware-requests, ops-eqiad, Operations, DBA
Cmjohnson updated the task description for T174763: Decommission db1026.
Dec 19 2017, 8:19 PM · hardware-requests, Patch-For-Review, ops-eqiad, DBA, Operations
Cmjohnson updated the task description for T181378: Decommission db1021.
Dec 19 2017, 8:18 PM · Patch-For-Review, hardware-requests, ops-eqiad, Operations, DBA
Cmjohnson updated the task description for T173570: Decommission db1015.
Dec 19 2017, 8:18 PM · Patch-For-Review, hardware-requests, ops-eqiad, Operations, DBA
Cmjohnson updated the task description for T181696: Decommission db1044.
Dec 19 2017, 8:18 PM · hardware-requests, ops-eqiad, Patch-For-Review, Operations, DBA
Cmjohnson updated the task description for T181784: Decommission db104[67].
Dec 19 2017, 8:18 PM · Analytics, hardware-requests, ops-eqiad, Operations
Cmjohnson added a comment to T181784: Decommission db104[67].

Disks are wiped

Dec 19 2017, 8:17 PM · Analytics, hardware-requests, ops-eqiad, Operations
Cmjohnson added a comment to T181696: Decommission db1044.

Disks are wiped

Dec 19 2017, 8:17 PM · hardware-requests, ops-eqiad, Patch-For-Review, Operations, DBA
Cmjohnson added a comment to T182853: Degraded RAID on db1059.

There were 2 failed disks. Replaced both and they're rebuilding

Dec 19 2017, 6:54 PM · DBA, ops-eqiad, Operations