Page MenuHomePhabricator
Create Task
Maniphest T228388

Configuration management for network operations
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

  • Productionize existing configuration management software jnt (potentially renaming it => Homer)
  • Integrate with Netbox for device selection
  • Integrate with Netbox for topology data gathering
  • Add safe push method for the configuration: interactive and sequential
  • [stretch] Evaluate Netbox to store network secrets => NO

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Homer: set SSH_AUTH_SOCKoperations/puppetproduction+1 -1
Homer: specify username to connect to devicesoperations/puppetproduction+1 -0
Make transport username configurableoperations/software/homermaster+14 -9
Homer: add Netbox configoperations/puppetproduction+42 -11
homer: add netbox credentials to the configurationoperations/puppetproduction+17 -8
Release v0.1.0operations/software/homer/deploymaster+18 -10
Updated changelog for first release.operations/software/homermaster+11 -0
netbox: split generic and device-specific dataoperations/software/homermaster+86 -28
Netbox: expose additional metadataoperations/software/homermaster+70 -6
devices: allow to expose arbitrary metadataoperations/software/homermaster+32 -113
netbox: allow to select the devices from Netboxoperations/software/homermaster+208 -17
Initial support for Netbox integrationoperations/software/homermaster+153 -18
Add commit action to the Homer classoperations/software/homermaster+114 -24
homer: cleanup absented resourceoperations/puppetproduction+0 -8
Homer: setup private repooperations/puppetproduction+71 -6
Homer, remove rsyncoperations/puppetproduction+1 -19
Homer: add bash wrapper to make it easy to run itoperations/puppetproduction+12 -0
Homer, add make package for scap deployoperations/puppetproduction+1 -1
homer->deploy-homer on deploy serveroperations/puppetproduction+1 -1
Homer, use deploy-homer user for deploy + fix files permsoperations/puppetproduction+10 -10
homer: add missing dependency virtualenvoperations/puppetproduction+3 -0
scap: add dsh group for homeroperations/puppetproduction+5 -0
Deploy homeroperations/puppetproduction+99 -0
Add fake SSH keypair for user homerlabs/privatemaster+4 -0
CLI: suppress ncclient noisy loggeroperations/software/homermaster+2 -0
config: inject role and site to the configurationoperations/software/homermaster+9 -2
transports: add JunOS transportoperations/software/homermaster+417 -10
actions: add generate actionoperations/software/homermaster+219 -42
templates: add rendering of templatesoperations/software/homermaster+137 -8
devices: add loggingoperations/software/homermaster+10 -3
cli: rename action compile to generateoperations/software/homermaster+3 -3
devices: add query capabilityoperations/software/homermaster+101 -3
Configuration: load and merge private configoperations/software/homermaster+105 -26
doc: add configuration example in documentationoperations/software/homermaster+26 -0
setup.py: add missing PyYAML dependencyoperations/software/homermaster+1 -0
doc: add link to Homer documentationintegration/docrootmaster+1 -0
Initial draft of devices configuration parsingoperations/software/homermaster+204 -1
Initial draft of the CLIoperations/software/homermaster+207 -4
Initial structure of the projectoperations/software/homermaster+1 K -0
Setup CI for operations/software/homerintegration/configmaster+19 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Aug 26 2019, 8:59 PM

Change 532225 merged by jenkins-bot:
[operations/software/homer@master] Configuration: load and merge private config

https://gerrit.wikimedia.org/r/532225

gerritbot added a comment.Aug 26 2019, 8:59 PM

Change 532226 merged by jenkins-bot:
[operations/software/homer@master] devices: add query capability

https://gerrit.wikimedia.org/r/532226

gerritbot added a comment.Aug 26 2019, 8:59 PM

Change 532227 merged by jenkins-bot:
[operations/software/homer@master] cli: rename action compile to generate

https://gerrit.wikimedia.org/r/532227

gerritbot added a comment.Aug 26 2019, 10:30 PM

Change 532452 merged by jenkins-bot:
[operations/software/homer@master] devices: add logging

https://gerrit.wikimedia.org/r/532452

gerritbot added a comment.Aug 27 2019, 12:17 PM

Change 532453 merged by jenkins-bot:
[operations/software/homer@master] templates: add rendering of templates

https://gerrit.wikimedia.org/r/532453

gerritbot added a comment.Aug 27 2019, 12:24 PM

Change 532454 merged by jenkins-bot:
[operations/software/homer@master] actions: add generate action

https://gerrit.wikimedia.org/r/532454

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2019, 1:10 PM
gerritbot added a comment.Aug 30 2019, 4:56 PM

Change 533558 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] transports: add JunOS transport

https://gerrit.wikimedia.org/r/533558

gerritbot added a project: Patch-For-Review.Aug 30 2019, 4:56 PM
gerritbot added a comment.Aug 30 2019, 6:18 PM

Change 533568 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] config: inject role and site to the configuration

https://gerrit.wikimedia.org/r/533568

gerritbot added a comment.Aug 30 2019, 6:32 PM

Change 533570 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] CLI: suppress ncclient noisy logger

https://gerrit.wikimedia.org/r/533570

gerritbot added a comment.Sep 9 2019, 3:40 PM

Change 533558 merged by jenkins-bot:
[operations/software/homer@master] transports: add JunOS transport

https://gerrit.wikimedia.org/r/533558

gerritbot added a comment.Sep 9 2019, 3:41 PM

Change 533568 merged by jenkins-bot:
[operations/software/homer@master] config: inject role and site to the configuration

https://gerrit.wikimedia.org/r/533568

gerritbot added a comment.Sep 9 2019, 3:41 PM

Change 533570 merged by jenkins-bot:
[operations/software/homer@master] CLI: suppress ncclient noisy logger

https://gerrit.wikimedia.org/r/533570

Maintenance_bot removed a project: Patch-For-Review.Sep 9 2019, 4:10 PM
gerritbot added a comment.Sep 25 2019, 5:33 PM

Change 534538 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Deploy homer

https://gerrit.wikimedia.org/r/534538

gerritbot added a project: Patch-For-Review.Sep 25 2019, 5:33 PM
gerritbot added a comment.Sep 25 2019, 5:53 PM

Change 539169 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[labs/private@master] Add fake SSH keypair for user homer

https://gerrit.wikimedia.org/r/539169

gerritbot added a comment.Sep 25 2019, 6:08 PM

Change 539169 merged by Ayounsi:
[labs/private@master] Add fake SSH keypair for user homer

https://gerrit.wikimedia.org/r/539169

ayounsi mentioned this in rLPRI69bdeb4dffd7: Add fake SSH keypair for user homer.Sep 25 2019, 6:12 PM
gerritbot added a comment.Sep 26 2019, 7:54 AM

Change 534538 merged by Volans:
[operations/puppet@production] Deploy homer

https://gerrit.wikimedia.org/r/534538

gerritbot added a comment.Sep 26 2019, 8:02 AM

Change 539271 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] scap: add dsh group for homer

https://gerrit.wikimedia.org/r/539271

gerritbot added a comment.Sep 26 2019, 8:07 AM

Change 539271 merged by Volans:
[operations/puppet@production] scap: add dsh group for homer

https://gerrit.wikimedia.org/r/539271

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 8:10 AM
gerritbot added a comment.Sep 26 2019, 9:26 AM

Change 539284 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] homer: add missing dependency virtualenv

https://gerrit.wikimedia.org/r/539284

gerritbot added a project: Patch-For-Review.Sep 26 2019, 9:26 AM
gerritbot added a comment.Sep 26 2019, 9:30 AM

Change 539284 merged by Volans:
[operations/puppet@production] homer: add missing dependency virtualenv

https://gerrit.wikimedia.org/r/539284

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 10:10 AM
gerritbot added a comment.Sep 26 2019, 4:45 PM

Change 539367 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Homer, use deploy-homer user for deploy + fix files perms

https://gerrit.wikimedia.org/r/539367

gerritbot added a project: Patch-For-Review.Sep 26 2019, 4:45 PM
gerritbot added a comment.Sep 26 2019, 5:02 PM

Change 539367 merged by Ayounsi:
[operations/puppet@production] Homer, use deploy-homer user for deploy + fix files perms

https://gerrit.wikimedia.org/r/539367

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 5:10 PM
gerritbot added a comment.Sep 26 2019, 5:50 PM

Change 539377 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] homer->deploy-homer on deploy server

https://gerrit.wikimedia.org/r/539377

gerritbot added a project: Patch-For-Review.Sep 26 2019, 5:50 PM

Change 539377 merged by Ayounsi:
[operations/puppet@production] homer->deploy-homer on deploy server

https://gerrit.wikimedia.org/r/539377

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 6:10 PM
gerritbot added a comment.Sep 26 2019, 6:39 PM

Change 539390 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Homer, add make package for scap deploy

https://gerrit.wikimedia.org/r/539390

gerritbot added a project: Patch-For-Review.Sep 26 2019, 6:39 PM
gerritbot added a comment.Sep 26 2019, 6:43 PM

Change 539390 merged by Ayounsi:
[operations/puppet@production] Homer, add make package for scap deploy

https://gerrit.wikimedia.org/r/539390

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 7:10 PM
gerritbot added a comment.Sep 26 2019, 7:36 PM

Change 539404 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Homer: add bash wrapper to make it easy to run it

https://gerrit.wikimedia.org/r/539404

gerritbot added a project: Patch-For-Review.Sep 26 2019, 7:36 PM
gerritbot added a comment.Sep 26 2019, 7:50 PM

Change 539404 merged by Volans:
[operations/puppet@production] Homer: add bash wrapper to make it easy to run it

https://gerrit.wikimedia.org/r/539404

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 8:10 PM
gerritbot added a comment.Sep 26 2019, 8:14 PM

Change 539411 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Homer, add secondary server for sync

https://gerrit.wikimedia.org/r/539411

gerritbot added a project: Patch-For-Review.Sep 26 2019, 8:14 PM
gerritbot added a comment.Sep 26 2019, 9:19 PM

Change 539411 merged by Ayounsi:
[operations/puppet@production] Homer, remove rsync

https://gerrit.wikimedia.org/r/539411

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 10:10 PM
gerritbot added a comment.Sep 27 2019, 6:10 AM

Change 539453 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Homer: setup private repo

https://gerrit.wikimedia.org/r/539453

gerritbot added a project: Patch-For-Review.Sep 27 2019, 6:10 AM
Volans updated the task description. (Show Details)Sep 27 2019, 6:22 AM
gerritbot added a comment.Sep 30 2019, 8:21 AM

Change 539453 merged by Volans:
[operations/puppet@production] Homer: setup private repo

https://gerrit.wikimedia.org/r/539453

gerritbot added a comment.Sep 30 2019, 8:32 AM

Change 539837 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] homer: cleanup absented resource

https://gerrit.wikimedia.org/r/539837

gerritbot added a comment.Sep 30 2019, 8:35 AM

Change 539837 merged by Volans:
[operations/puppet@production] homer: cleanup absented resource

https://gerrit.wikimedia.org/r/539837

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2019, 9:10 AM
gerritbot added a comment.Oct 3 2019, 6:06 PM

Change 539551 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] Add commit action to the Homer class

https://gerrit.wikimedia.org/r/539551

gerritbot added a project: Patch-For-Review.Oct 3 2019, 6:06 PM
elukey added a project: Wikimedia-Incident.Oct 7 2019, 9:57 AM
gerritbot added a comment.Oct 7 2019, 5:32 PM

Change 539551 merged by Ayounsi:
[operations/software/homer@master] Add commit action to the Homer class

https://gerrit.wikimedia.org/r/539551

ayounsi mentioned this in rOSHO4ffe0e1f0c99: Add commit action to the Homer class.Oct 7 2019, 5:41 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 7 2019, 6:11 PM
gerritbot added a comment.Oct 15 2019, 10:59 AM

Change 543114 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] Initial support for Netbox integration

https://gerrit.wikimedia.org/r/543114

gerritbot added a project: Patch-For-Review.Oct 15 2019, 10:59 AM
gerritbot added a comment.Oct 15 2019, 3:01 PM

Change 543114 merged by jenkins-bot:
[operations/software/homer@master] Initial support for Netbox integration

https://gerrit.wikimedia.org/r/543114

Maintenance_bot removed a project: Patch-For-Review.Oct 15 2019, 3:10 PM
gerritbot added a comment.Oct 17 2019, 4:06 PM

Change 543890 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] netbox: allow to select the devices from Netbox

https://gerrit.wikimedia.org/r/543890

gerritbot added a project: Patch-For-Review.Oct 17 2019, 4:06 PM
gerritbot added a comment.Oct 21 2019, 12:14 PM

Change 544881 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] homer: add netbox credentials to the configuration

https://gerrit.wikimedia.org/r/544881

gerritbot added a comment.Oct 31 2019, 11:52 AM

Change 543890 merged by jenkins-bot:
[operations/software/homer@master] netbox: allow to select the devices from Netbox

https://gerrit.wikimedia.org/r/543890

Volans updated the task description. (Show Details)Oct 31 2019, 2:34 PM
Volans added a comment.Oct 31 2019, 2:47 PM

Basic integration with Netbox has been developed and is now merged, pending the next release. Some improvements are already WIP and should be ready for CR later today.

gerritbot added a comment.Oct 31 2019, 7:40 PM

Change 547638 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] devices: allow to expose arbitrary metadata

https://gerrit.wikimedia.org/r/547638

gerritbot added a comment.Oct 31 2019, 7:40 PM

Change 547639 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] Netbox: expose additional metadata

https://gerrit.wikimedia.org/r/547639

gerritbot added a comment.Nov 1 2019, 8:07 PM

Change 547638 merged by jenkins-bot:
[operations/software/homer@master] devices: allow to expose arbitrary metadata

https://gerrit.wikimedia.org/r/547638

gerritbot added a comment.Nov 4 2019, 6:33 PM

Change 547639 merged by jenkins-bot:
[operations/software/homer@master] Netbox: expose additional metadata

https://gerrit.wikimedia.org/r/547639

gerritbot added a comment.Dec 12 2019, 3:35 PM

Change 556703 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] netbox: split generic and device-specific data

https://gerrit.wikimedia.org/r/556703

gerritbot added a comment.Dec 16 2019, 10:20 AM

Change 556703 merged by jenkins-bot:
[operations/software/homer@master] netbox: split generic and device-specific data

https://gerrit.wikimedia.org/r/556703

gerritbot added a comment.Dec 17 2019, 9:42 AM

Change 558435 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer@master] Updated changelog for first release.

https://gerrit.wikimedia.org/r/558435

gerritbot added a comment.Dec 17 2019, 9:50 AM

Change 558435 merged by jenkins-bot:
[operations/software/homer@master] Updated changelog for first release.

https://gerrit.wikimedia.org/r/558435

gerritbot added a comment.Dec 17 2019, 10:03 AM

Change 558442 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/homer/deploy@master] Release v0.1.0

https://gerrit.wikimedia.org/r/558442

gerritbot added a comment.Dec 17 2019, 10:07 AM

Change 558442 merged by Volans:
[operations/software/homer/deploy@master] Release v0.1.0

https://gerrit.wikimedia.org/r/558442

Stashbot added a comment.Dec 17 2019, 10:13 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-17T10:13:12Z] <volans@deploy1001> Started deploy [homer/deploy@996f7be]: Homer release v0.1.0 - T228388

Stashbot added a comment.Dec 17 2019, 10:13 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-17T10:13:45Z] <volans@deploy1001> Finished deploy [homer/deploy@996f7be]: Homer release v0.1.0 - T228388 (duration: 00m 32s)

Volans updated the task description. (Show Details)Dec 17 2019, 10:17 AM
gerritbot added a comment.Dec 17 2019, 10:38 AM

Change 558456 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Homer: add Netbox config

https://gerrit.wikimedia.org/r/558456

gerritbot added a comment.Dec 17 2019, 10:56 AM

Change 544881 abandoned by Volans:
homer: add netbox credentials to the configuration

Reason:
Superseeded by https://gerrit.wikimedia.org/r/c/operations/puppet/ /558456

https://gerrit.wikimedia.org/r/544881

gerritbot added a comment.Dec 17 2019, 10:58 AM

Change 558456 merged by Ayounsi:
[operations/puppet@production] Homer: add Netbox config

https://gerrit.wikimedia.org/r/558456

Volans mentioned this in rOSHP996f7be39285: Release v0.1.0.Dec 17 2019, 11:30 AM
gerritbot added a comment.Dec 17 2019, 11:57 AM

Change 558479 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/software/homer@master] Make transport username configurable

https://gerrit.wikimedia.org/r/558479

gerritbot added a comment.Dec 17 2019, 12:25 PM

Change 558479 merged by Ayounsi:
[operations/software/homer@master] Make transport username configurable

https://gerrit.wikimedia.org/r/558479

gerritbot added a comment.Dec 17 2019, 12:56 PM

Change 558497 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Homer: specify username to connect to devices

https://gerrit.wikimedia.org/r/558497

Stashbot added a comment.Dec 17 2019, 1:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-17T13:01:20Z] <ayounsi@deploy1001> Started deploy [homer/deploy@359de04]: Homer release v0.1.1 - T228388

Stashbot added a comment.Dec 17 2019, 1:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-17T13:01:44Z] <ayounsi@deploy1001> Finished deploy [homer/deploy@359de04]: Homer release v0.1.1 - T228388 (duration: 00m 30s)

gerritbot added a comment.Dec 17 2019, 1:05 PM

Change 558497 merged by Ayounsi:
[operations/puppet@production] Homer: specify username to connect to devices

https://gerrit.wikimedia.org/r/558497

gerritbot added a comment.Dec 17 2019, 1:14 PM

Change 558504 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Homer: set SSH_AUTH_SOCK

https://gerrit.wikimedia.org/r/558504

gerritbot added a comment.Dec 17 2019, 1:51 PM

Change 558504 merged by Ayounsi:
[operations/puppet@production] Homer: set SSH_AUTH_SOCK

https://gerrit.wikimedia.org/r/558504

Krinkle moved this task from Active investigation to Draft and review incident doc on the Wikimedia-Incident board.Dec 17 2019, 10:16 PM
ayounsi updated the task description. (Show Details)Feb 4 2020, 10:32 PM

Everything here is done.
Doc is there https://wikitech.wikimedia.org/wiki/Homer and has been tested by other SREs than Riccardo or me.

Future developments will need their own task.

ayounsi closed this task as Resolved.Feb 4 2020, 10:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits