Page MenuHomePhabricator

Audit puppet usage in cloud hosts
Open, In Progress, MediumPublic

Description

To get a better idea of what is being used in cloud services and provide input to T285539 we should audit the current usage to get an idea on how much puppet ius currently used.

Event Timeline

jbond triaged this task as Medium priority.Aug 25 2021, 8:42 AM
Volans renamed this task from Audid puppet usage in cloud hosts to Audit puppet usage in cloud hosts.Aug 25 2021, 9:52 AM

Since I had a random conversation about this in IRC today with @nskaggs, I thought I should record what I found here. I was curious about the numbers of cloud users who are actively doing something with puppet.

Of 168 projects active today, I found that 110 projects have actually applied puppet classes using horizon to something besides what hits the entire cloud. I got this by checking on instance-puppet git with: ls -l */*.roles |awk '($5 > 3 ) {print $9}'| awk -F '/' '{print $1}' | uniq | wc -l (for checking my method). Validating the size was bigger than "3" was to filter the empty files with just brackets in them.

I did not evaluate how many project owners are using custom puppet code vs. "cloud utility" roles like role::labs::mediawiki_vagrant, labs_lvm and cinderutils. I can confirm that nobody is using mariadb modules outside of staff-involved projects.

following on from brokes script i create a small python script to give us a list of used classes

#!/usr/bin/python3

import logging

from argparse import ArgumentParser
from collections import defaultdict
from pathlib import Path

import yaml


def get_args() -> None:
    """Parse arguments.

    Returns:
        `argparse.Namespace`: The parsed argparser Namespace
    """
    parser = ArgumentParser(description=__doc__)
    parser.add_argument('-v', '--verbose', action='count', default=0)
    parser.add_argument('--repo_dir', default=Path.cwd(), type=Path)
    return parser.parse_args()


def get_log_level(args_level: int) -> int:
    """Convert an integer to a logging log level.

    Arguments:
        args_level (int): The log level as an integer

    Returns:
        int: the logging loglevel
    """
    return {
        0: logging.ERROR,
        1: logging.WARN,
        2: logging.INFO,
        3: logging.DEBUG,
    }.get(args_level, logging.DEBUG)


def main() -> int:
    """Main entry point.

    Returns:
        int: an int representing the exit code
    """
    args = get_args()
    logging.basicConfig(level=get_log_level(args.verbose))
    project_classes = defaultdict(set)
    for project in args.repo_dir.iterdir():
        for role_file in project.glob('*.roles'):
            roles = yaml.load(role_file.read_text())
            if roles:
                project_classes[project.name].update(set(roles))
    # convert back to lists as they look better in yaml
    project_classes = [{k: list(v)} for k, v in project_classes.items()]
    print(yaml.safe_dump(project_classes))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
analytics:
- role::puppetmaster::standalone
- profile::airflow
- profile::kafka::broker
- role::aqs
- profile::zookeeper::server
- role::kafka::simple::broker
- profile::base::firewall
annotation:
- role::labs::mediawiki_vagrant
auditlogging:
- role::puppetmaster::standalone
- role::syslog::centralserver_cloud
bastion:
- role::labs::bastion
centralnotice-staging:
- role::labs::mediawiki_vagrant
cloud-analytics:
- role::cloud_analytics::config
- role::cloud_analytics::worker
clouddb-services:
- role::puppetmaster::standalone
- role::labs::lvm::srv
- profile::labs::lvm::srv
- role::wmcs::db::wikireplicas::querysampler
- role::wmcs::db::osmdb_secondary
- role::wmcs::db::wikilabels
- role::wmcs::db::osmdb_primary
- role::wmcs::db::toolsdb_secondary
- role::wmcs::db::wikireplicas::cloudproxy
- role::wmcs::db::toolsdb_primary
- role::wmcs::db::wikilabels_secondary
cloudinfra:
- role::puppetmaster::standalone
- role::mail::smarthost::wmcs
- role::wmcs::services::ntp
- profile::openstack::eqiad1::cumin::master
- role::wmcs::openstack::eqiad1::puppetmaster::frontend_vm
- role::acme_chief::cloud
- role::wmcs::openstack::eqiad1::puppetmaster::backend_vm
- role::mariadb::cloudinfra
cloudstore:
- role::puppetmaster::standalone
- role::wmcs::nfs::test
- httpd
codesearch:
- role::codesearch
commonsarchive:
- role::labs::mediawiki_vagrant
- profile::labs::lvm::srv
commtech:
- role::labs::mediawiki_vagrant
cvn:
- role::labs::lvm::srv
cyberbot:
- role::labs::lvm::srv
- role::cyberbot::db
- role::cyberbot::exec
dashiki:
- role::simplestatic
deployment-prep:
- profile::ores::worker
- role::url_downloader
- profile::envoy
- role::mediawiki::memcached
- role::cache::upload
- profile::standard
- role::wdqs::labs
- role::kafka::logging
- role::swift::storage
- profile::maps::apps
- role::mw_rc_irc
- role::logstash7
- profile::prometheus::postgres_exporter
- profile::maps::cassandra
- role::kafka::main
- profile::eventschemas::service
- profile::maps::osm_master
- profile::mariadb::misc::eventlogging::database
- role::puppetmaster::standalone
- profile::labs::lvm::srv
- role::parsoid
- role::puppetmaster::puppetdb
- role::webperf::xhgui
- role::elasticsearch::beta
- role::restbase::production
- profile::pki::client
- profile::ores::web
- role::mariadb::beta
- profile::redis::master
- role::cache::text
- profile::conftool::master
- profile::tlsproxy::envoy
- role::kafka::jumbo::broker
- profile::etcd::v3
- profile::ci::slave::labs::common
- profile::base::firewall
- profile::eventlogging::analytics::processor
- role::mail::mx
- role::webperf::processors_and_site
- role::swift::proxy
- role::beta::mediawiki
- role::mediawiki::appserver
- role::aqs
- role::beta::docker_services
- profile::openstack::eqiad1::cumin::master
- role::jobqueue_redis::master
- role::aptly::server
- role::thumbor::mediawiki
- profile::phabricator::main
- role::beta::cassandra
- role::acme_chief::cloud
- role::poolcounter::server
- role::phabricator_server
- profile::eventlogging::analytics::mysql
- profile::services_proxy::envoy
- profile::rsyslog::kafka_shipper
- role::beta::deploymentserver
- role::webperf::profiling_tools
- role::labs::lvm::srv
- role::beta::puppetmaster
- role::mediawiki::maintenance
- role::deployment_server
- profile::zookeeper::server
- role::beta::mediawiki::jobrunner
- role::dumps::generation::worker::beta_testbed
- role::prometheus::beta
- role::labs::ores::redis
- profile::rsyslog::udp_localhost_compat
- profile::swift::storage::labs
- role::logging::mediawiki::udp2log
devtools:
- role::puppetmaster::standalone
- role::phabricator
- role::deployment_server
- profile::doc
- role::doc
- role::gerrit
download:
- profile::labs::downloadserver
dumps:
- role::labs::lvm::srv
dwl:
- role::labs::lvm::srv
entity-detection:
- role::labs::lvm::srv
etytree:
- role::labs::lvm::srv
extdist:
- role::labs::lvm::srv
- role::extdist
git:
- role::labs::lvm::srv
- role::gerrit
gitlab-test:
- profile::labs::lvm::srv
- profile::gitlab_test::gitlab
glampipe:
- role::simplelamp2
google-api-proxy:
- role::labs::google_api_proxy
gratitude:
- role::labs::lvm::srv
hhvm:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
hound:
- role::puppetmaster::standalone
ign2commons:
- role::labs::lvm::srv
iiab:
- profile::labs::lvm::srv
incubator:
- role::labs::lvm::srv
integration:
- role::puppetmaster::standalone
- role::ci::slave::labs::docker
- role::ci::castor::server
- profile::openstack::eqiad1::cumin::target
- profile::openstack::eqiad1::cumin::master
- role::ci::slave::package_builder
library-upgrader:
- role::labs::lvm::srv
logging:
- profile::zookeeper::server
- role::puppetmaster::pontoon
lta-tracker:
- role::puppetmaster::standalone
- role::simplelamp2
- role::labs::mediawiki_vagrant
mailman:
- role::puppetmaster::standalone
- role::lists3
- role::lists
maps:
- role::puppetmaster::standalone
maps-experiments:
- role::puppetmaster::standalone
- profile::redis::master
- profile::maps::apps
- profile::prometheus::postgres_exporter
- profile::maps::cassandra
- role::beta::docker_services
- profile::maps::osm_master
mariadb104-test:
- role::puppetmaster::pontoon
math:
- role::labs::lvm::srv
mcr-dev:
- role::labs::mediawiki_vagrant
mediawiki-vagrant:
- role::puppetmaster::standalone
- role::labs::mediawiki_vagrant
meet:
- role::meet::accountmanager
metricsinfra:
- role::wmcs::metricsinfra::prometheus
- role::puppetmaster::standalone
- role::wmcs::metricsinfra::haproxy
- role::wmcs::metricsinfra::controller
- role::wmcs::metricsinfra::alertmanager
monitoring:
- role::puppetmaster::standalone
- role::swift::proxy
- role::rancid::server
- role::librenms
- role::swift::storage
- role::servermon::wmf
- role::wmcs::prometheus
- role::smokeping
- profile::swift::storage::labs
mwoffliner:
- role::labs::lvm::srv
- profile::labs::lvm::srv
mwv-apt:
- role::aptly::server
newsletter:
- role::labs::mediawiki_vagrant
ogvjs-integration:
- role::labs::mediawiki_vagrant
openocr:
- role::simplelamp2
openstack:
- role::wmcs::openstack::eqiad1::virt
- role::wmcs::openstack::eqiad1::control
- role::wmcs::openstack::eqiad1::net
- role::wmcs::openstack::codfw1dev::control
- role::labs::bootstrapvz
ores:
- role::labs::ores::redis
- role::labs::ores::web
- role::labs::ores::worker
- role::labs::ores::lb
ores-staging:
- role::labs::ores::staging
osmit:
- role::labs::lvm::srv
packaging:
- role::package::builder
- role::builder
- role::calico::builder
- profile::envoy::builder
paws:
- role::aptly::server
- role::puppetmaster::standalone
- role::wmcs::paws::trove::backup
- role::wmcs::paws::k8s::control
- profile::wmcs::paws::common
- role::wmcs::paws::k8s::worker
- role::acme_chief::cloud
- role::wmcs::paws::k8s::haproxy
pipelinelib-experimental:
- profile::labs::lvm::srv
- docker
pki:
- role::puppetmaster::standalone
- mariadb::config
- mariadb::service
- profile::mariadb::packages_wmf
- profile::pki::root_ca
- profile::pki::multirootca
- profile::pki::client
planet:
- role::planet
privpol-captcha:
- apt::unattendedupgrades
project-proxy:
- role::puppetmaster::standalone
- role::acme_chief::cloud
- role::wmcs::novaproxy
- profile::wmcs::proxy::static
puppet:
- role::puppetmaster::standalone
- profile::base::firewall
puppet-diffs:
- role::puppet_compiler
quarry:
- role::labs::lvm::srv
- role::labs::quarry::web
- role::labs::quarry::celeryrunner
- role::labs::quarry::redis
- role::labs::quarry::database
rcm:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
reading-web-staging:
- role::labs::mediawiki_vagrant
- role::simplelamp2
- role::labs::lvm::srv
recommendation-api:
- role::labs::lvm::srv
redirects:
- role::labs::redirector
sciencesource:
- role::labs::lvm::srv
search:
- role::puppetmaster::standalone
- role::labs::vagrant_lxc
- profile::labs::lvm::srv
- role::labs::mediawiki_vagrant
services:
- role::labs::mediawiki_vagrant
shiny-r:
- role::discovery::dashboards
signwriting:
- role::simplelamp2
snuggle:
- role::labs::lvm::srv
soweego:
- role::labs::lvm::srv
srwiki-dev:
- role::labs::mediawiki_vagrant
sso:
- role::puppetmaster::standalone
- profile::debmonitor::client
- role::debmonitor::server
- gitlab
striker:
- profile::labs::lvm::srv
- role::labs::mediawiki_vagrant
- profile::keyholder::server
- profile::mediawiki::deployment::server
- role::striker::web
- striker::build
structurednavigation:
- role::labs::mediawiki_vagrant
swift:
- role::swift::proxy
techblog:
- role::puppetmaster::standalone
testlabs:
- role::puppetmaster::standalone
- profile::ceph::testing
- role::labs::lvm::srv
- role::wmcs::ceph::mon
- 'testing:blah::'
- profile::ceph::common
- role::labs::lvm::mnt
- profile::openstack::base::galera::node
- role::wmcs::ceph::osd
- role::wmcs::toolforge::elastic7
- profile::alextest
tools:
- role::wmcs::toolserver_legacy
- role::wmcs::toolforge::k8s::control
- role::wmcs::toolforge::grid::web::generic
- role::labs::lvm::mnt
- role::wmcs::toolforge::k8s::etcd
- role::prometheus::node_exporter
- role::toollabs::etcd::flannel
- role::wmcs::toolforge::elastic7
- profile::toolforge::apt_pinning
- role::wmcs::toolforge::elastic
- role::puppetmaster::standalone
- role::toollabs::logging::centralserver
- role::wmcs::toolforge::package_builder
- profile::toolforge::grid::base
- role::wmcs::toolforge::mailrelay
- role::toollabs::k8s::worker
- role::wmcs::toolforge::static
- role::wmcs::toolforge::docker::builder
- role::wmcs::toolforge::legacy_redirector
- role::wmcs::toolforge::k8s::haproxy
- role::wmcs::toolforge::docker::image_builder
- role::wmcs::toolforge::prometheus
- role::wmcs::toolforge::grid::shadow
- role::wmcs::toolforge::bastion
- role::aptly::client
- role::wmcs::toolforge::services
- role::wmcs::toolforge::proxy
- role::wmcs::toolforge::docker::registry
- role::wmcs::toolforge::grid::cronrunner
- role::wmcs::toolforge::grid::web::lighttpd
- profile::toolforge::infrastructure
- role::acme_chief::cloud
- role::wmcs::toolforge::grid::master
- role::toollabs::k8s::master
- role::labs::lvm::srv
- role::wmcs::toolforge::redis
- role::wmcs::toolforge::checker
- role::wmcs::toolforge::k8s::worker
- role::toollabs::cronrunner
- role::wmcs::toolforge::grid::compute::general
- profile::toolforge::clush::target
- role::wmcs::toolforge::clush::master
toolsbeta:
- role::wmcs::toolforge::docker::image_builder
- role::wmcs::toolforge::harbor
- role::wmcs::toolforge::k8s::control
- role::wmcs::toolforge::grid::web::generic
- role::labs::lvm::mnt
- role::wmcs::toolforge::bastion
- role::wmcs::toolforge::docker::registry
- role::aptly::client
- profile::toolforge::clush::target
- role::wmcs::toolforge::proxy
- role::wmcs::toolforge::k8s::etcd
- role::toollabs::etcd::flannel
- role::wmcs::toolforge::grid::cronrunner
- role::wmcs::toolforge::elastic7
- profile::toolforge::infrastructure
- role::acme_chief::cloud
- role::wmcs::toolforge::grid::web::lighttpd
- role::wmcs::toolforge::redis_sentinel
- role::wmcs::toolforge::grid::master
- role::toollabs::k8s::master
- role::puppetmaster::standalone
- role::toollabs::master
- role::labs::lvm::srv
- role::puppetmaster::puppetdb
- role::wmcs::toolforge::mailrelay
- role::wmcs::toolforge::k8s::worker
- role::toollabs::k8s::worker
- role::wmcs::toolforge::legacy_redirector
- role::wmcs::toolforge::k8s::haproxy
- role::wmcs::toolforge::grid::compute::general
- role::wmcs::toolforge::grid::shadow
- role::wmcs::toolforge::clush::master
traffic:
- role::puppetmaster::standalone
- profile::cache::envoy
- role::cache::text
- role::rpkivalidator
- profile::diffscan
- role::dnsbox
- role::acme_chief::cloud
- role::ncredir
video:
- role::labs::lvm::srv
visualeditor:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wcdo:
- role::labs::lvm::srv
wikiapiary:
- role::labs::lvm::srv
wikibrain:
- role::postgres::common
- role::labs::lvm::srv
wikidata-dev:
- role::labs::lvm::srv
- role::wikibase
wikidata-history-query-service:
- role::labs::lvm::srv
wikidata-query:
- role::puppetmaster::standalone
- role::wcqs::cloud
wikidata-realtime-dumps:
- role::labs::lvm::srv
wikidumpparse:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wikifactmine:
- role::puppetmaster::standalone
wikilabels:
- role::wikilabels::staging
- role::wikilabels::server
wikimetrics:
- role::wikimetrics::production
wikipathways:
- role::simplelamp2
- role::labs::lvm::srv
wikisp:
- role::simplelamp2
wikispeech:
- role::labs::mediawiki_vagrant
wikispore:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wikistats:
- role::wikistats::instance
wikitextexp:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wildcat:
- role::simplelamp2
- role::labs::lvm::srv
wmcz-stats:
- profile::mariadb::packages_client
wmde-dashboards:
- role::labs::lvm::srv
wmde-templates-alpha:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv

following on from brokes script i create a small python script to give us a list of used classes

https://openstack-browser.toolforge.org/puppetclass/ is related. It groups by puppet manifest rather than project so it really doesn't show the spread across projects that @jbond's report helps to surface.

joanna_borun changed the task status from Open to In Progress.Tue, Sep 21, 1:36 PM