Page MenuHomePhabricator

crusnov (Cas Rusnov)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 15 2018, 5:56 PM (27 w, 1 d)
Availability
Available
LDAP User
CRusnov
MediaWiki User
Unknown

Recent Activity

Yesterday

crusnov added a comment to T217074: Reduce Manual Steps in Provisioning by 4.

getting the netbox module in the cookbooks will save steps on decoms and probably reimages and installs (which share many procedures); the caveat is that in decoms it will have to prompt as to the state to transition into (decom or spare).

Tue, Apr 23, 10:49 PM · User-crusnov, Operations-Software-Development
crusnov added a comment to T217074: Reduce Manual Steps in Provisioning by 4.

After several conversations with robh, I think we can start looking at the low hanging fruit. For the record all of these processes are mediated by a dynamic, ever changing checklist.

Tue, Apr 23, 10:48 PM · User-crusnov, Operations-Software-Development
crusnov moved T217074: Reduce Manual Steps in Provisioning by 4 from Ready to In Progress on the User-crusnov board.
Tue, Apr 23, 10:21 PM · User-crusnov, Operations-Software-Development
crusnov moved T217072: Spicerack module for Netbox from In Progress to Pending on the User-crusnov board.
Tue, Apr 23, 10:21 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T220422: Netbox Reports: General Cleanup and Improvement from In Progress to Pending on the User-crusnov board.
Tue, Apr 23, 10:21 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov moved T218709: Add Spicerack module for Ganeti from In Progress to Pending on the User-crusnov board.
Tue, Apr 23, 10:21 PM · Operations-Software-Development, User-crusnov
crusnov added a comment to T220422: Netbox Reports: General Cleanup and Improvement.
Tue, Apr 23, 4:55 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov moved T215229: Keep Ganeti VMs synchronized in Netbox from In Code Review to Pending release/deployment on the Operations-Software-Development board.
Tue, Apr 23, 3:17 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T203963: Convert makevm to spicerack cookbook from In Code Review to Pending release/deployment on the Operations-Software-Development board.
Tue, Apr 23, 3:17 PM · Patch-For-Review, User-crusnov, serviceops, Operations-Software-Development, User-jijiki, User-Joe, Operations
crusnov moved T218709: Add Spicerack module for Ganeti from In Code Review to Pending release/deployment on the Operations-Software-Development board.
Tue, Apr 23, 3:17 PM · Operations-Software-Development, User-crusnov
crusnov moved T217072: Spicerack module for Netbox from In Code Review to Pending release/deployment on the Operations-Software-Development board.
Tue, Apr 23, 3:17 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov closed T215229: Keep Ganeti VMs synchronized in Netbox as Resolved.
Tue, Apr 23, 2:50 AM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov closed T215229: Keep Ganeti VMs synchronized in Netbox, a subtask of T213114: Q3 2018/19 Goal: TEC6: Build automated workflows for server provisioning (Tracking Task), as Resolved.
Tue, Apr 23, 2:50 AM · User-crusnov, Operations-Software-Development

Wed, Apr 17

crusnov added a comment to T220422: Netbox Reports: General Cleanup and Improvement.

exclude esams from console report

Wed, Apr 17, 5:27 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov added a comment to T220422: Netbox Reports: General Cleanup and Improvement.

robh requests that the status show up in test_netbox_in_puppetdb

Wed, Apr 17, 3:53 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov moved T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number from In Progress to Backlog on the User-crusnov board.
Wed, Apr 17, 3:50 PM · User-crusnov, Operations-Software-Development

Tue, Apr 16

crusnov created T221113: Netbox Reports: Create an icinga check for alerting on a set of Netbox reports.
Tue, Apr 16, 5:08 PM · User-crusnov, Operations-Software-Development

Fri, Apr 12

crusnov added a comment to T220422: Netbox Reports: General Cleanup and Improvement.

Thanks for refiguring the checklist :)

Fri, Apr 12, 10:19 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov added a comment to T220422: Netbox Reports: General Cleanup and Improvement.

I forgot another one, the opposite of this:

We needs a new method, to check for devices with Status: Offline, that have row/rack assigned. I'm sure there are plenty of those now.

i.e. alert on any devices with status not in (offline, planned) but with no row/rack assigned :)

Fri, Apr 12, 8:42 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development

Thu, Apr 11

crusnov added a comment to T220422: Netbox Reports: General Cleanup and Improvement.

OK, so, after the efforts in the past few days, we're in a much better shape! The PuppetDB report seems to be (almost?) entirely indicative of real issues and is actionable now - I will involve DC Ops to start fixing the cases that are known to be real errors, and we'll see if there are any false positives (I know of at least one, that is tough to handle!).

The Coherence report in its current state is not super useful. Issues to fix:

  • Blacklist sites esams and knams for now; these are known to be wildly inconsistent (or should I say incoherent? :) and they are unfortunately not actionable right now. We can re-enable once we do a big cleanup project there, hopefully in a few months.
  • Blacklist certain roles, which are known to miss data and we accept that's OK. I can think of at least: Cable management, Storage bin, Optical device. Possibly others.
Thu, Apr 11, 9:47 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development

Wed, Apr 10

crusnov moved T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number from Backlog to In Progress on the User-crusnov board.
Wed, Apr 10, 3:31 PM · User-crusnov, Operations-Software-Development
crusnov claimed T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.
Wed, Apr 10, 3:31 PM · User-crusnov, Operations-Software-Development
crusnov moved T220422: Netbox Reports: General Cleanup and Improvement from Backlog to In Progress on the User-crusnov board.
Wed, Apr 10, 3:29 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov added a project to T220422: Netbox Reports: General Cleanup and Improvement: User-crusnov.
Wed, Apr 10, 3:29 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov added a comment to T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.

Okay the only question that seems open in my mind is how does the service map serial to fqdn?

Wed, Apr 10, 3:28 PM · User-crusnov, Operations-Software-Development

Tue, Apr 9

crusnov added a comment to T205900: Cumin: add backend for Netbox.

It'd be neat if teh code written for spicerack for this purpose could be reused somehow.

Tue, Apr 9, 6:20 PM · Operations, Operations-Software-Development

Mon, Apr 8

crusnov added a comment to T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.

I suppose the conversation we need is:

  • Where will this live?

The install servers seems a good option IMHO

  • What information will be sent, exactly?

See iPXE documentation, the API will be called like https://ipxe.org/scripting#dynamic_scripts and it should return the boot options for the appropriate d-i image, see also https://ipxe.org/scripting

  • Where will the information be stored?
    • Netbox?
    • Puppet?

Which information are you referring to?
This API should need a storage, just a short-lived in memory cache and it should be writable by the cumin hosts (e.g. the reimage cookbook). It will connect to Netbox in RO mode and gather the FQDN of the host from the Serial Number and then resolve it's addresses via DNS.

Mon, Apr 8, 8:58 PM · User-crusnov, Operations-Software-Development
crusnov moved T215229: Keep Ganeti VMs synchronized in Netbox from Pending to Complete on the User-crusnov board.
Mon, Apr 8, 5:01 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov updated the task description for T220422: Netbox Reports: General Cleanup and Improvement.
Mon, Apr 8, 4:42 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development
crusnov closed T212526: Implement netbox reports which check against PuppetDB as Resolved.
Mon, Apr 8, 4:41 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov created T220422: Netbox Reports: General Cleanup and Improvement.
Mon, Apr 8, 4:37 PM · Patch-For-Review, User-crusnov, DC-Ops, Operations-Software-Development

Thu, Apr 4

crusnov committed rCUMIN7b06f44ce8ae: Make the puppetdb backend process primitive types for queries (authored by crusnov).
Make the puppetdb backend process primitive types for queries
Thu, Apr 4, 4:11 PM
crusnov committed rCUMINd97cdf22775b: Make the puppetdb backend process primitive types for queries (authored by crusnov).
Make the puppetdb backend process primitive types for queries
Thu, Apr 4, 3:02 PM
crusnov triaged T219933: parsoid-vd on scandium randomly died as Normal priority.
Thu, Apr 4, 4:23 AM · Patch-For-Review, Operations

Wed, Apr 3

crusnov updated subscribers of T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.
Wed, Apr 3, 10:59 PM · User-crusnov, Operations-Software-Development
crusnov added a comment to T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.

I suppose the conversation we need is:

Wed, Apr 3, 10:58 PM · User-crusnov, Operations-Software-Development

Tue, Apr 2

crusnov created T219933: parsoid-vd on scandium randomly died.
Tue, Apr 2, 8:41 PM · Patch-For-Review, Operations
crusnov triaged T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number as Normal priority.
Tue, Apr 2, 6:06 PM · User-crusnov, Operations-Software-Development
crusnov updated the task description for T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.
Tue, Apr 2, 6:03 PM · User-crusnov, Operations-Software-Development
crusnov renamed T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number from Build an API for generating boot options for iPXE from Netbox et al. based on asset tag to Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.
Tue, Apr 2, 6:02 PM · User-crusnov, Operations-Software-Development
crusnov updated the task description for T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.
Tue, Apr 2, 5:56 PM · User-crusnov, Operations-Software-Development
crusnov added a parent task for T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number: T217074: Reduce Manual Steps in Provisioning by 4.
Tue, Apr 2, 5:54 PM · User-crusnov, Operations-Software-Development
crusnov added a subtask for T217074: Reduce Manual Steps in Provisioning by 4: T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.
Tue, Apr 2, 5:54 PM · User-crusnov, Operations-Software-Development
crusnov created T219908: Build an API for generating boot options for iPXE from Netbox et al. based on Serial Number.
Tue, Apr 2, 5:51 PM · User-crusnov, Operations-Software-Development
crusnov added a comment to T219696: Alert "kubelet operational latencies".

Minor suggestion, perhaps we could increase the alert threshold if operation isn't actually affected at these levels. Quite often kubelet will sit on the alert threshold and flap alerts.

Tue, Apr 2, 2:40 PM · Prod-Kubernetes, Kubernetes, Operations

Thu, Mar 28

crusnov added a comment to T219454: Make Spicerack cookbook to resize ganeti VM.

Well that's convenient. In the wikitech page about resizing it recommends checking the grafana views for this information.

Thu, Mar 28, 4:37 PM · Operations-Software-Development
crusnov closed T218736: Discussions around having a Ganeti RAPI R/W User as Resolved.
Thu, Mar 28, 4:36 PM · Operations-Software-Development, User-crusnov
crusnov closed T218736: Discussions around having a Ganeti RAPI R/W User, a subtask of T218709: Add Spicerack module for Ganeti, as Resolved.
Thu, Mar 28, 4:36 PM · Operations-Software-Development, User-crusnov
crusnov added a comment to T218736: Discussions around having a Ganeti RAPI R/W User.

Incidentally this is the strategy we're pursuing anyway. For the time being write operations will take the form of remote execution in the cookbook while the ganeti module will provide information to said cookbooks.

Thu, Mar 28, 4:36 PM · Operations-Software-Development, User-crusnov
crusnov added a comment to T174916: electron/pdfrender hangs.

Additional follow-up: THere were numerous OOMs in the log, even though the box has around 20gb of free ram +/- buffers. I'm not sure if there's a service that spikes up that high or if its the slice that's causing the OOM, but an interesting data point.

Thu, Mar 28, 2:29 PM · Core Platform Team Backlog (Attic), Services (attic), Electron-PDFs, Operations
crusnov added a comment to T174916: electron/pdfrender hangs.

Just a note, the service was flapping for a while, and I have restarted it on scb1004.

Thu, Mar 28, 2:54 AM · Core Platform Team Backlog (Attic), Services (attic), Electron-PDFs, Operations

Wed, Mar 27

crusnov closed T219456: Add ram to Proton* as Resolved.
Wed, Mar 27, 10:00 PM · Operations
crusnov updated the task description for T219456: Add ram to Proton*.
Wed, Mar 27, 10:00 PM · Operations
crusnov created T219456: Add ram to Proton*.
Wed, Mar 27, 9:43 PM · Operations
crusnov created T219454: Make Spicerack cookbook to resize ganeti VM.
Wed, Mar 27, 9:37 PM · Operations-Software-Development
crusnov moved T217072: Spicerack module for Netbox from In Progress to In Code Review on the Operations-Software-Development board.
Wed, Mar 27, 6:24 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T218709: Add Spicerack module for Ganeti from In Progress to In Code Review on the Operations-Software-Development board.
Wed, Mar 27, 6:24 PM · Operations-Software-Development, User-crusnov

Mar 22 2019

crusnov added a comment to T207037: Cumin: allow to query for Puppet primitive types.

For the record, the latest patchset was just pending on me testing null values work in puppetdb queries, which I've done now so this should be coming soon.

Mar 22 2019, 2:59 PM · Patch-For-Review, Operations-Software-Development

Mar 21 2019

crusnov moved T217074: Reduce Manual Steps in Provisioning by 4 from In Progress to Ready on the User-crusnov board.
Mar 21 2019, 9:44 PM · User-crusnov, Operations-Software-Development
crusnov moved T217074: Reduce Manual Steps in Provisioning by 4 from Backlog to In Progress on the User-crusnov board.
Mar 21 2019, 9:44 PM · User-crusnov, Operations-Software-Development
crusnov moved T218736: Discussions around having a Ganeti RAPI R/W User from Backlog to In Progress on the User-crusnov board.
Mar 21 2019, 9:44 PM · Operations-Software-Development, User-crusnov
crusnov moved T218956: Should we deploy sshguard on external IP addresses? from Backlog to In Progress on the User-crusnov board.
Mar 21 2019, 9:44 PM · Patch-For-Review, User-crusnov, Security-Team
crusnov added a comment to T218956: Should we deploy sshguard on external IP addresses?.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/498231

Mar 21 2019, 9:41 PM · Patch-For-Review, User-crusnov, Security-Team
crusnov created T218956: Should we deploy sshguard on external IP addresses?.
Mar 21 2019, 9:39 PM · Patch-For-Review, User-crusnov, Security-Team
Mill <mill@mail.com> committed rCUMINbcbe70b0021d: k7aaaaaaaaaaaa (authored by crusnov).
k7aaaaaaaaaaaa
Mar 21 2019, 12:37 AM

Mar 19 2019

crusnov created T218736: Discussions around having a Ganeti RAPI R/W User.
Mar 19 2019, 8:12 PM · Operations-Software-Development, User-crusnov
crusnov closed T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency as Resolved.
Mar 19 2019, 6:15 PM · Patch-For-Review, Operations, Operations-Software-Development
crusnov closed T205899: Develop and deploy at least three Netbox reports to assist with data correctness and consistency, a subtask of T205868: Expand Netbox usage - Q2 2018-19 Goal, as Resolved.
Mar 19 2019, 6:15 PM · Operations, Operations-Software-Development, Goal
crusnov closed T215041: Netbox reports: add tox / integration tests for repository as Resolved.
Mar 19 2019, 6:14 PM · Operations-Software-Development
crusnov closed T212524: Upgrade Netbox to 2.5.x as Resolved.
Mar 19 2019, 6:14 PM · Patch-For-Review, Operations-Software-Development
crusnov closed T212524: Upgrade Netbox to 2.5.x, a subtask of T213114: Q3 2018/19 Goal: TEC6: Build automated workflows for server provisioning (Tracking Task), as Resolved.
Mar 19 2019, 6:14 PM · User-crusnov, Operations-Software-Development
crusnov moved T218709: Add Spicerack module for Ganeti from Backlog to In Progress on the Operations-Software-Development board.
Mar 19 2019, 6:12 PM · Operations-Software-Development, User-crusnov
crusnov moved T217072: Spicerack module for Netbox from Backlog to In Progress on the Operations-Software-Development board.
Mar 19 2019, 6:12 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T217074: Reduce Manual Steps in Provisioning by 4 from Backlog to In Progress on the Operations-Software-Development board.
Mar 19 2019, 6:12 PM · User-crusnov, Operations-Software-Development
crusnov moved T205900: Cumin: add backend for Netbox from In Progress to Up next on the Operations-Software-Development board.
Mar 19 2019, 6:11 PM · Operations, Operations-Software-Development
crusnov moved T203963: Convert makevm to spicerack cookbook from In Progress to In Code Review on the Operations-Software-Development board.
Mar 19 2019, 6:11 PM · Patch-For-Review, User-crusnov, serviceops, Operations-Software-Development, User-jijiki, User-Joe, Operations
crusnov added a comment to T218709: Add Spicerack module for Ganeti.

Upside of this is that python3-ganeti-rapi is already in stretch-backports.

Mar 19 2019, 5:55 PM · Operations-Software-Development, User-crusnov
crusnov added a comment to T218709: Add Spicerack module for Ganeti.

As part of the MakeVM port, it was requested that this be pursued as a stretch goal.

Mar 19 2019, 5:54 PM · Operations-Software-Development, User-crusnov
crusnov created T218709: Add Spicerack module for Ganeti.
Mar 19 2019, 5:46 PM · Operations-Software-Development, User-crusnov
crusnov moved T212526: Implement netbox reports which check against PuppetDB from In Progress to Pending on the User-crusnov board.
Mar 19 2019, 5:44 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T203963: Convert makevm to spicerack cookbook from In Progress to Pending on the User-crusnov board.
Mar 19 2019, 5:44 PM · Patch-For-Review, User-crusnov, serviceops, Operations-Software-Development, User-jijiki, User-Joe, Operations

Mar 18 2019

crusnov moved T212524: Upgrade Netbox to 2.5.x from In Code Review to Pending release/deployment on the Operations-Software-Development board.
Mar 18 2019, 11:06 PM · Patch-For-Review, Operations-Software-Development
crusnov moved T215041: Netbox reports: add tox / integration tests for repository from In Code Review to Pending release/deployment on the Operations-Software-Development board.
Mar 18 2019, 11:06 PM · Operations-Software-Development

Mar 16 2019

crusnov updated the task description for T212526: Implement netbox reports which check against PuppetDB.
Mar 16 2019, 12:26 AM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T212526: Implement netbox reports which check against PuppetDB from In Progress to In Code Review on the Operations-Software-Development board.
Mar 16 2019, 12:25 AM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T212526: Implement netbox reports which check against PuppetDB from Up next to In Progress on the Operations-Software-Development board.
Mar 16 2019, 12:25 AM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T203963: Convert makevm to spicerack cookbook from Up next to In Progress on the Operations-Software-Development board.
Mar 16 2019, 12:25 AM · Patch-For-Review, User-crusnov, serviceops, Operations-Software-Development, User-jijiki, User-Joe, Operations
crusnov moved T205900: Cumin: add backend for Netbox from Up next to In Progress on the Operations-Software-Development board.
Mar 16 2019, 12:25 AM · Operations, Operations-Software-Development
crusnov moved T215229: Keep Ganeti VMs synchronized in Netbox from In Progress to In Code Review on the Operations-Software-Development board.
Mar 16 2019, 12:25 AM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov added a comment to T215229: Keep Ganeti VMs synchronized in Netbox.

This is deployed and works in production, it needs only to have the timer deployed

Mar 16 2019, 12:24 AM · Patch-For-Review, User-crusnov, Operations-Software-Development

Mar 15 2019

crusnov added a comment to T218440: Cumin: allow running as non-root.

As an extension to this, it would perhaps be necessary for Cumin to look in standards directories, like ~/.config/ (or maybe ~/.config/cumin) for its configuration file, otherwise root is still required to edit things in /etc/cumin/...

Mar 15 2019, 9:28 PM · Patch-For-Review, Operations-Software-Development
crusnov added a comment to T218440: Cumin: allow running as non-root.

The current workaround is to set the environment variables that it checks like SUDO_USER=$USER USER=root cumin ... but I strongly agree with this feature request, as I've made a similar one :)

Mar 15 2019, 9:27 PM · Patch-For-Review, Operations-Software-Development
crusnov added a comment to T215229: Keep Ganeti VMs synchronized in Netbox.

I don't expect that changes all that often, but I agree that the script could take that into account (there is an API for tose devices, of course). Now that it's in place it should be straight forward to modify.

Mar 15 2019, 3:39 PM · Patch-For-Review, User-crusnov, Operations-Software-Development

Mar 14 2019

crusnov moved T217072: Spicerack module for Netbox from Backlog to In Progress on the User-crusnov board.
Mar 14 2019, 8:36 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T212526: Implement netbox reports which check against PuppetDB from Backlog to In Progress on the User-crusnov board.
Mar 14 2019, 8:36 PM · Patch-For-Review, User-crusnov, Operations-Software-Development
crusnov moved T203963: Convert makevm to spicerack cookbook from Pending to In Progress on the User-crusnov board.
Mar 14 2019, 8:36 PM · Patch-For-Review, User-crusnov, serviceops, Operations-Software-Development, User-jijiki, User-Joe, Operations
crusnov moved T203963: Convert makevm to spicerack cookbook from Backlog to Pending on the User-crusnov board.
Mar 14 2019, 8:36 PM · Patch-For-Review, User-crusnov, serviceops, Operations-Software-Development, User-jijiki, User-Joe, Operations
crusnov claimed T203963: Convert makevm to spicerack cookbook.
Mar 14 2019, 8:34 PM · Patch-For-Review, User-crusnov, serviceops, Operations-Software-Development, User-jijiki, User-Joe, Operations
crusnov moved T215229: Keep Ganeti VMs synchronized in Netbox from In Progress to Pending on the User-crusnov board.
Mar 14 2019, 8:33 PM · Patch-For-Review, User-crusnov, Operations-Software-Development

Mar 4 2019

crusnov updated the task description for T217072: Spicerack module for Netbox.
Mar 4 2019, 6:24 PM · Patch-For-Review, User-crusnov, Operations-Software-Development