Page MenuHomePhabricator
Create Task
Maniphest T289658

Audit puppet usage in cloud hosts
Closed, ResolvedPublic
Actions

Description

To get a better idea of what is being used in cloud services and provide input to T285539 we should audit the current usage to get an idea on how much puppet ius currently used.

Related Objects
Search...

Event Timeline

jbond created this task.Aug 25 2021, 8:41 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptAug 25 2021, 8:42 AM
jbond triaged this task as Medium priority.Aug 25 2021, 8:42 AM
Volans renamed this task from Audid puppet usage in cloud hosts to Audit puppet usage in cloud hosts.Aug 25 2021, 9:52 AM
jbond added a comment.Aug 25 2021, 11:00 AM
In T285539#7306974, @Bstorm wrote:

Since I had a random conversation about this in IRC today with @nskaggs, I thought I should record what I found here. I was curious about the numbers of cloud users who are actively doing something with puppet.

Of 168 projects active today, I found that 110 projects have actually applied puppet classes using horizon to something besides what hits the entire cloud. I got this by checking on instance-puppet git with: ls -l */*.roles |awk '($5 > 3 ) {print $9}'| awk -F '/' '{print $1}' | uniq | wc -l (for checking my method). Validating the size was bigger than "3" was to filter the empty files with just brackets in them.

I did not evaluate how many project owners are using custom puppet code vs. "cloud utility" roles like role::labs::mediawiki_vagrant, labs_lvm and cinderutils. I can confirm that nobody is using mariadb modules outside of staff-involved projects.

jbond mentioned this in T285539: Easing pain points caused by divergence between cloudservices and production puppet usecases .Aug 25 2021, 11:01 AM
jbond added a comment.Aug 25 2021, 11:49 AM

following on from brokes script i create a small python script to give us a list of used classes

#!/usr/bin/python3

import logging

from argparse import ArgumentParser
from collections import defaultdict
from pathlib import Path

import yaml


def get_args() -> None:
    """Parse arguments.

    Returns:
        `argparse.Namespace`: The parsed argparser Namespace
    """
    parser = ArgumentParser(description=__doc__)
    parser.add_argument('-v', '--verbose', action='count', default=0)
    parser.add_argument('--repo_dir', default=Path.cwd(), type=Path)
    return parser.parse_args()


def get_log_level(args_level: int) -> int:
    """Convert an integer to a logging log level.

    Arguments:
        args_level (int): The log level as an integer

    Returns:
        int: the logging loglevel
    """
    return {
        0: logging.ERROR,
        1: logging.WARN,
        2: logging.INFO,
        3: logging.DEBUG,
    }.get(args_level, logging.DEBUG)


def main() -> int:
    """Main entry point.

    Returns:
        int: an int representing the exit code
    """
    args = get_args()
    logging.basicConfig(level=get_log_level(args.verbose))
    project_classes = defaultdict(set)
    for project in args.repo_dir.iterdir():
        for role_file in project.glob('*.roles'):
            roles = yaml.load(role_file.read_text())
            if roles:
                project_classes[project.name].update(set(roles))
    # convert back to lists as they look better in yaml
    project_classes = [{k: list(v)} for k, v in project_classes.items()]
    print(yaml.safe_dump(project_classes))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
analytics:
- role::puppetmaster::standalone
- profile::airflow
- profile::kafka::broker
- role::aqs
- profile::zookeeper::server
- role::kafka::simple::broker
- profile::base::firewall
annotation:
- role::labs::mediawiki_vagrant
auditlogging:
- role::puppetmaster::standalone
- role::syslog::centralserver_cloud
bastion:
- role::labs::bastion
centralnotice-staging:
- role::labs::mediawiki_vagrant
cloud-analytics:
- role::cloud_analytics::config
- role::cloud_analytics::worker
clouddb-services:
- role::puppetmaster::standalone
- role::labs::lvm::srv
- profile::labs::lvm::srv
- role::wmcs::db::wikireplicas::querysampler
- role::wmcs::db::osmdb_secondary
- role::wmcs::db::wikilabels
- role::wmcs::db::osmdb_primary
- role::wmcs::db::toolsdb_secondary
- role::wmcs::db::wikireplicas::cloudproxy
- role::wmcs::db::toolsdb_primary
- role::wmcs::db::wikilabels_secondary
cloudinfra:
- role::puppetmaster::standalone
- role::mail::smarthost::wmcs
- role::wmcs::services::ntp
- profile::openstack::eqiad1::cumin::master
- role::wmcs::openstack::eqiad1::puppetmaster::frontend_vm
- role::acme_chief::cloud
- role::wmcs::openstack::eqiad1::puppetmaster::backend_vm
- role::mariadb::cloudinfra
cloudstore:
- role::puppetmaster::standalone
- role::wmcs::nfs::test
- httpd
codesearch:
- role::codesearch
commonsarchive:
- role::labs::mediawiki_vagrant
- profile::labs::lvm::srv
commtech:
- role::labs::mediawiki_vagrant
cvn:
- role::labs::lvm::srv
cyberbot:
- role::labs::lvm::srv
- role::cyberbot::db
- role::cyberbot::exec
dashiki:
- role::simplestatic
deployment-prep:
- profile::ores::worker
- role::url_downloader
- profile::envoy
- role::mediawiki::memcached
- role::cache::upload
- profile::standard
- role::wdqs::labs
- role::kafka::logging
- role::swift::storage
- profile::maps::apps
- role::mw_rc_irc
- role::logstash7
- profile::prometheus::postgres_exporter
- profile::maps::cassandra
- role::kafka::main
- profile::eventschemas::service
- profile::maps::osm_master
- profile::mariadb::misc::eventlogging::database
- role::puppetmaster::standalone
- profile::labs::lvm::srv
- role::parsoid
- role::puppetmaster::puppetdb
- role::webperf::xhgui
- role::elasticsearch::beta
- role::restbase::production
- profile::pki::client
- profile::ores::web
- role::mariadb::beta
- profile::redis::master
- role::cache::text
- profile::conftool::master
- profile::tlsproxy::envoy
- role::kafka::jumbo::broker
- profile::etcd::v3
- profile::ci::slave::labs::common
- profile::base::firewall
- profile::eventlogging::analytics::processor
- role::mail::mx
- role::webperf::processors_and_site
- role::swift::proxy
- role::beta::mediawiki
- role::mediawiki::appserver
- role::aqs
- role::beta::docker_services
- profile::openstack::eqiad1::cumin::master
- role::jobqueue_redis::master
- role::aptly::server
- role::thumbor::mediawiki
- profile::phabricator::main
- role::beta::cassandra
- role::acme_chief::cloud
- role::poolcounter::server
- role::phabricator_server
- profile::eventlogging::analytics::mysql
- profile::services_proxy::envoy
- profile::rsyslog::kafka_shipper
- role::beta::deploymentserver
- role::webperf::profiling_tools
- role::labs::lvm::srv
- role::beta::puppetmaster
- role::mediawiki::maintenance
- role::deployment_server
- profile::zookeeper::server
- role::beta::mediawiki::jobrunner
- role::dumps::generation::worker::beta_testbed
- role::prometheus::beta
- role::labs::ores::redis
- profile::rsyslog::udp_localhost_compat
- profile::swift::storage::labs
- role::logging::mediawiki::udp2log
devtools:
- role::puppetmaster::standalone
- role::phabricator
- role::deployment_server
- profile::doc
- role::doc
- role::gerrit
download:
- profile::labs::downloadserver
dumps:
- role::labs::lvm::srv
dwl:
- role::labs::lvm::srv
entity-detection:
- role::labs::lvm::srv
etytree:
- role::labs::lvm::srv
extdist:
- role::labs::lvm::srv
- role::extdist
git:
- role::labs::lvm::srv
- role::gerrit
gitlab-test:
- profile::labs::lvm::srv
- profile::gitlab_test::gitlab
glampipe:
- role::simplelamp2
google-api-proxy:
- role::labs::google_api_proxy
gratitude:
- role::labs::lvm::srv
hhvm:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
hound:
- role::puppetmaster::standalone
ign2commons:
- role::labs::lvm::srv
iiab:
- profile::labs::lvm::srv
incubator:
- role::labs::lvm::srv
integration:
- role::puppetmaster::standalone
- role::ci::slave::labs::docker
- role::ci::castor::server
- profile::openstack::eqiad1::cumin::target
- profile::openstack::eqiad1::cumin::master
- role::ci::slave::package_builder
library-upgrader:
- role::labs::lvm::srv
logging:
- profile::zookeeper::server
- role::puppetmaster::pontoon
lta-tracker:
- role::puppetmaster::standalone
- role::simplelamp2
- role::labs::mediawiki_vagrant
mailman:
- role::puppetmaster::standalone
- role::lists3
- role::lists
maps:
- role::puppetmaster::standalone
maps-experiments:
- role::puppetmaster::standalone
- profile::redis::master
- profile::maps::apps
- profile::prometheus::postgres_exporter
- profile::maps::cassandra
- role::beta::docker_services
- profile::maps::osm_master
mariadb104-test:
- role::puppetmaster::pontoon
math:
- role::labs::lvm::srv
mcr-dev:
- role::labs::mediawiki_vagrant
mediawiki-vagrant:
- role::puppetmaster::standalone
- role::labs::mediawiki_vagrant
meet:
- role::meet::accountmanager
metricsinfra:
- role::wmcs::metricsinfra::prometheus
- role::puppetmaster::standalone
- role::wmcs::metricsinfra::haproxy
- role::wmcs::metricsinfra::controller
- role::wmcs::metricsinfra::alertmanager
monitoring:
- role::puppetmaster::standalone
- role::swift::proxy
- role::rancid::server
- role::librenms
- role::swift::storage
- role::servermon::wmf
- role::wmcs::prometheus
- role::smokeping
- profile::swift::storage::labs
mwoffliner:
- role::labs::lvm::srv
- profile::labs::lvm::srv
mwv-apt:
- role::aptly::server
newsletter:
- role::labs::mediawiki_vagrant
ogvjs-integration:
- role::labs::mediawiki_vagrant
openocr:
- role::simplelamp2
openstack:
- role::wmcs::openstack::eqiad1::virt
- role::wmcs::openstack::eqiad1::control
- role::wmcs::openstack::eqiad1::net
- role::wmcs::openstack::codfw1dev::control
- role::labs::bootstrapvz
ores:
- role::labs::ores::redis
- role::labs::ores::web
- role::labs::ores::worker
- role::labs::ores::lb
ores-staging:
- role::labs::ores::staging
osmit:
- role::labs::lvm::srv
packaging:
- role::package::builder
- role::builder
- role::calico::builder
- profile::envoy::builder
paws:
- role::aptly::server
- role::puppetmaster::standalone
- role::wmcs::paws::trove::backup
- role::wmcs::paws::k8s::control
- profile::wmcs::paws::common
- role::wmcs::paws::k8s::worker
- role::acme_chief::cloud
- role::wmcs::paws::k8s::haproxy
pipelinelib-experimental:
- profile::labs::lvm::srv
- docker
pki:
- role::puppetmaster::standalone
- mariadb::config
- mariadb::service
- profile::mariadb::packages_wmf
- profile::pki::root_ca
- profile::pki::multirootca
- profile::pki::client
planet:
- role::planet
privpol-captcha:
- apt::unattendedupgrades
project-proxy:
- role::puppetmaster::standalone
- role::acme_chief::cloud
- role::wmcs::novaproxy
- profile::wmcs::proxy::static
puppet:
- role::puppetmaster::standalone
- profile::base::firewall
puppet-diffs:
- role::puppet_compiler
quarry:
- role::labs::lvm::srv
- role::labs::quarry::web
- role::labs::quarry::celeryrunner
- role::labs::quarry::redis
- role::labs::quarry::database
rcm:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
reading-web-staging:
- role::labs::mediawiki_vagrant
- role::simplelamp2
- role::labs::lvm::srv
recommendation-api:
- role::labs::lvm::srv
redirects:
- role::labs::redirector
sciencesource:
- role::labs::lvm::srv
search:
- role::puppetmaster::standalone
- role::labs::vagrant_lxc
- profile::labs::lvm::srv
- role::labs::mediawiki_vagrant
services:
- role::labs::mediawiki_vagrant
shiny-r:
- role::discovery::dashboards
signwriting:
- role::simplelamp2
snuggle:
- role::labs::lvm::srv
soweego:
- role::labs::lvm::srv
srwiki-dev:
- role::labs::mediawiki_vagrant
sso:
- role::puppetmaster::standalone
- profile::debmonitor::client
- role::debmonitor::server
- gitlab
striker:
- profile::labs::lvm::srv
- role::labs::mediawiki_vagrant
- profile::keyholder::server
- profile::mediawiki::deployment::server
- role::striker::web
- striker::build
structurednavigation:
- role::labs::mediawiki_vagrant
swift:
- role::swift::proxy
techblog:
- role::puppetmaster::standalone
testlabs:
- role::puppetmaster::standalone
- profile::ceph::testing
- role::labs::lvm::srv
- role::wmcs::ceph::mon
- 'testing:blah::'
- profile::ceph::common
- role::labs::lvm::mnt
- profile::openstack::base::galera::node
- role::wmcs::ceph::osd
- role::wmcs::toolforge::elastic7
- profile::alextest
tools:
- role::wmcs::toolserver_legacy
- role::wmcs::toolforge::k8s::control
- role::wmcs::toolforge::grid::web::generic
- role::labs::lvm::mnt
- role::wmcs::toolforge::k8s::etcd
- role::prometheus::node_exporter
- role::toollabs::etcd::flannel
- role::wmcs::toolforge::elastic7
- profile::toolforge::apt_pinning
- role::wmcs::toolforge::elastic
- role::puppetmaster::standalone
- role::toollabs::logging::centralserver
- role::wmcs::toolforge::package_builder
- profile::toolforge::grid::base
- role::wmcs::toolforge::mailrelay
- role::toollabs::k8s::worker
- role::wmcs::toolforge::static
- role::wmcs::toolforge::docker::builder
- role::wmcs::toolforge::legacy_redirector
- role::wmcs::toolforge::k8s::haproxy
- role::wmcs::toolforge::docker::image_builder
- role::wmcs::toolforge::prometheus
- role::wmcs::toolforge::grid::shadow
- role::wmcs::toolforge::bastion
- role::aptly::client
- role::wmcs::toolforge::services
- role::wmcs::toolforge::proxy
- role::wmcs::toolforge::docker::registry
- role::wmcs::toolforge::grid::cronrunner
- role::wmcs::toolforge::grid::web::lighttpd
- profile::toolforge::infrastructure
- role::acme_chief::cloud
- role::wmcs::toolforge::grid::master
- role::toollabs::k8s::master
- role::labs::lvm::srv
- role::wmcs::toolforge::redis
- role::wmcs::toolforge::checker
- role::wmcs::toolforge::k8s::worker
- role::toollabs::cronrunner
- role::wmcs::toolforge::grid::compute::general
- profile::toolforge::clush::target
- role::wmcs::toolforge::clush::master
toolsbeta:
- role::wmcs::toolforge::docker::image_builder
- role::wmcs::toolforge::harbor
- role::wmcs::toolforge::k8s::control
- role::wmcs::toolforge::grid::web::generic
- role::labs::lvm::mnt
- role::wmcs::toolforge::bastion
- role::wmcs::toolforge::docker::registry
- role::aptly::client
- profile::toolforge::clush::target
- role::wmcs::toolforge::proxy
- role::wmcs::toolforge::k8s::etcd
- role::toollabs::etcd::flannel
- role::wmcs::toolforge::grid::cronrunner
- role::wmcs::toolforge::elastic7
- profile::toolforge::infrastructure
- role::acme_chief::cloud
- role::wmcs::toolforge::grid::web::lighttpd
- role::wmcs::toolforge::redis_sentinel
- role::wmcs::toolforge::grid::master
- role::toollabs::k8s::master
- role::puppetmaster::standalone
- role::toollabs::master
- role::labs::lvm::srv
- role::puppetmaster::puppetdb
- role::wmcs::toolforge::mailrelay
- role::wmcs::toolforge::k8s::worker
- role::toollabs::k8s::worker
- role::wmcs::toolforge::legacy_redirector
- role::wmcs::toolforge::k8s::haproxy
- role::wmcs::toolforge::grid::compute::general
- role::wmcs::toolforge::grid::shadow
- role::wmcs::toolforge::clush::master
traffic:
- role::puppetmaster::standalone
- profile::cache::envoy
- role::cache::text
- role::rpkivalidator
- profile::diffscan
- role::dnsbox
- role::acme_chief::cloud
- role::ncredir
video:
- role::labs::lvm::srv
visualeditor:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wcdo:
- role::labs::lvm::srv
wikiapiary:
- role::labs::lvm::srv
wikibrain:
- role::postgres::common
- role::labs::lvm::srv
wikidata-dev:
- role::labs::lvm::srv
- role::wikibase
wikidata-history-query-service:
- role::labs::lvm::srv
wikidata-query:
- role::puppetmaster::standalone
- role::wcqs::cloud
wikidata-realtime-dumps:
- role::labs::lvm::srv
wikidumpparse:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wikifactmine:
- role::puppetmaster::standalone
wikilabels:
- role::wikilabels::staging
- role::wikilabels::server
wikimetrics:
- role::wikimetrics::production
wikipathways:
- role::simplelamp2
- role::labs::lvm::srv
wikisp:
- role::simplelamp2
wikispeech:
- role::labs::mediawiki_vagrant
wikispore:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wikistats:
- role::wikistats::instance
wikitextexp:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
wildcat:
- role::simplelamp2
- role::labs::lvm::srv
wmcz-stats:
- profile::mariadb::packages_client
wmde-dashboards:
- role::labs::lvm::srv
wmde-templates-alpha:
- role::labs::mediawiki_vagrant
- role::labs::lvm::srv
bd808 added a comment.Aug 25 2021, 3:03 PM
In T289658#7308070, @jbond wrote:

following on from brokes script i create a small python script to give us a list of used classes

https://openstack-browser.toolforge.org/puppetclass/ is related. It groups by puppet manifest rather than project so it really doesn't show the spread across projects that @jbond's report helps to surface.

jbond moved this task from Backlog to In Progress on the Infrastructure-Foundations board.Aug 30 2021, 10:02 AM
taavi removed a project: Cloud Services Proposals.Sep 19 2021, 3:23 PM
joanna_borun changed the task status from Open to In Progress.Sep 21 2021, 1:36 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:46 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
jbond closed this task as Resolved.Jun 12 2023, 5:33 PM
jbond claimed this task.

going to resolve this i think the original question was answered

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL