Page MenuHomePhabricator
Create Task
Maniphest T248994

Add CI check to ensure defaults exist in cloud.yaml
Closed, ResolvedPublic
Actions

Description

When creating new puppet class paramter, defaults should be added to the production environment in hiera under hieradata/common/profile/ and the cloud.yaml file. The later step often gets missed and causes errors on cloud admin. We shold investigate if we can create a CI check to ensure defaults are added to both the cloud and production hiera files

Related Objects
Search...

Event Timeline

jbond created this task.Mar 31 2020, 3:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 31 2020, 3:05 PM
jbond triaged this task as Medium priority.Mar 31 2020, 3:05 PM
jbond added a subscriber: Andrew.EditedApr 1 2020, 1:59 PM

@Andrew the following keys live in the production hiera but don't exist in cloud.yaml. do you want entries for all of theses or are there some we can safely skip?

---
profile::idp::client::httpd::cookie_path: /var/cache/apache2/mod_auth_cas
profile::idp::client::httpd::certificate_path: /etc/ssl/certs
profile::idp::client::httpd::authn_header: CAS-User
profile::idp::client::httpd::attribute_prefix: X-CAS-
profile::idp::client::httpd::debug: false
profile::idp::client::httpd::validate_saml: false
profile::idp::client::httpd::apache_owner: www-data
profile::idp::client::httpd::apache_group: www-data
profile::idp::client::httpd::required_groups: []
profile::idp::client::httpd::priority: 99
profile::idp::client::httpd::enable_monitor: true
profile::idp::client::httpd::protected_uri: /
profile::ganeti::rapi::ro_user: ro_user
profile::tlsproxy::service::check_uri: /
profile::tlsproxy::service::contact_group: admin
profile::tlsproxy::envoy::global_cert_name: 
profile::dumps::generation::worker::common::php: /usr/bin/php7.2
profile::dumps::generation::worker::common::nfs_extra_mountopts: actimeo=0
profile::dumps::generation::worker::common::dumps_misc_cronrunner: false
profile::dumps::generation::worker::dumper::runtype: regular
profile::dumps::generation::worker::dumper::maxjobs: 28
profile::dumps::distribution::basedatadir: /srv/dumps/xmldatadumps
profile::dumps::distribution::miscdumpsdir: /srv/dumps/xmldatadumps/public/other
profile::dumps::distribution::xmldumpspublicdir: /srv/dumps/xmldatadumps/public
profile::dumps::distribution::xmldumpsprivatedir: /srv/dumps/xmldatadumps/private
profile::dumps::distribution::dumpstempdir: /srv/dumps/xmldatadumps/temp
profile::dumps::distribution::rsync_config: {"dumps_user"=>"dumpsgen", "dumps_group"=>"dumpsgen", "dumps_deploygroup"=>"wikidev", "dumps_mntpoint"=>"/srv/dumps"}
profile::dumps::distribution::nfs_clients: *****Snipped for readability *****
profile::dumps::distribution::datasets::cleanup::isreplica: true
profile::ci::firewall::zuul_merger_hosts: ["208.80.154.17", "208.80.153.15"]
profile::ores::web::statsd: localhost:9125
profile::netbox::active_server: netbox1001.wikimedia.org
profile::netbox::service_hostname: netbox.wikimedia.org
profile::netbox::netbox_api: https://netbox.wikimedia.org/
profile::netbox::frontends: %{alias('netbox_frontend')}
profile::netbox::ganeti_sync_interval: 15
profile::netbox::netbox_report_checks: *****Snipped for readability *****
profile::netbox::dump_interval: *-*-* *:32:00
profile::base::systemd::cpu_accounting: no
profile::base::systemd::blockio_accounting: no
profile::base::systemd::memory_accounting: no
profile::base::firewall::block_abuse_nets: false
profile::base::debdeploy::exclude_mounts: 
profile::base::debdeploy::filter_services: {"bacula-fd"=>["*"], "agetty"=>["*"], "nginx"=>["libtiff", "libpng16"], "systemd"=>["*"], "dbus-daemon"=>["*"], "(sd-pam"=>["*"], "(sd-pam)"=>["*"]}
profile::base::puppet::dns_alt_names: 
profile::rsyslog::kafka_shipper::kafka_brokers: ["logstash1010.eqiad.wmnet:9093", "logstash1011.eqiad.wmnet:9093", "logstash1012.eqiad.wmnet:9093"]
profile::trafficserver::backend::mapping_rules: *****Snipped for readability *****
profile::trafficserver::backend::default_lua_script: default
profile::trafficserver::backend::enable_compress: false
profile::trafficserver::backend::outbound_tls_settings:  *****Snipped for readability *****
profile::trafficserver::backend::storage_elements: [{"devname"=>"sda3", "id"=>"disk.0"}, {"devname"=>"sdb3", "id"=>"disk.1"}]
profile::trafficserver::backend::log_formats: *****Snipped for readability *****
profile::trafficserver::backend::log_filters:  *****Snipped for readability *****
profile::trafficserver::backend::network_settings: {"connections_throttle"=>0, "sock_option_flag_in"=>5, "default_inactivity_timeout"=>360}
profile::trafficserver::backend::http_settings: *****Snipped for readability *****
profile::trafficserver::tls::port: 443
profile::trafficserver::tls::network_settings: {"connections_throttle"=>0, "sock_option_flag_in"=>13, "default_inactivity_timeout"=>300}
profile::trafficserver::tls::mapping_rules:  *****Snipped for readability *****
profile::trafficserver::tls::inbound_tls_settings:  *****Snipped for readability *****
profile::trafficserver::tls::log_formats: *****Snipped for readability *****
profile::trafficserver::tls::logs: [{"filename"=>"tls", "format"=>"wmf-tls", "mode"=>"ascii_pipe"}, {"filename"=>"analytics", "format"=>"wmf-analytics", "mode"=>"ascii_pipe"}]
profile::trafficserver::tls::parent_rules: []
profile::trafficserver::tls::unified_certs: ["globalsign-2019-ecdsa-unified", "globalsign-2019-rsa-unified", "digicert-2019a-ecdsa-unified", "digicert-2019a-rsa-unified"]
profile::trafficserver::tls::unified_acme_chief: true
profile::trafficserver::tls::available_unified_certs:  *****Snipped for readability *****
profile::trafficserver::tls::http_settings:  *****Snipped for readability *****
profile::puppetdb::microservice::enabled: true
profile::puppetdb::microservice::port: 8090
profile::puppetdb::microservice::uwsgi_port: 8091
profile::puppetdb::microservice::allowed_hosts: %{alias('netbox_frontend')}
profile::puppetdb::monitoring_agentrun::warn: 1
profile::puppetdb::monitoring_agentrun::crit: 5
profile::cache::kafka::webrequest::monitoring_enabled: true
profile::cache::kafka::webrequest::kafka_cluster_name: jumbo-eqiad
profile::cache::kafka::webrequest::ssl_enabled: true
profile::racktables::racktables_host: racktables.wikimedia.org
profile::lvs::realserver::use_conftool: true
profile::idp::key_password: changeit
profile::idp::keystore_password: changeit
profile::idp::tgc_encryption_key: changeit
profile::idp::tgc_signing_key: changeit
profile::idp::webflow_encryption_key: changeit
profile::idp::webflow_signing_key: changeit
profile::idp::u2f_encryption_key: changeit
profile::idp::u2f_signing_key: changeit
profile::idp::gauth_encryption_key: changeit
profile::idp::gauth_signing_key: changeit
profile::idp::idp_primary: idp2001.wikimedia.org
profile::idp::idp_failover: idp1001.wikimedia.org
profile::dumps::stats_hosts: stat1006.eqiad.wmnet stat1007.eqiad.wmnet
profile::dumps::peer_hosts: dumpsdata1001.eqiad.wmnet dumpsdata1002.eqiad.wmnet dumpsdata1003.eqiad.wmnet labstore1006.wikimedia.org labstore1007.wikimedia.org
profile::dumps::phab_hosts: phab1001.eqiad.wmnet
profile::dumps::rsync_internal_clients: *****Snipped for readability *****
profile::dumps::generation_worker_cron_php: /usr/bin/php7.2
profile::dumps::single_backend: false
profile::dumps::public: ["labstore1006.wikimedia.org", "labstore1007.wikimedia.org"]
profile::dumps::internal: ["dumpsdata1003.eqiad.wmnet"]
profile::dumps::xmldumpsdir: /data/xmldatadumps/public
profile::dumps::miscdumpsdir: /data/otherdumps
profile::dumps::miscsubdirs: incr,categoriesrdf
profile::samplicator::port: 2000
profile::samplicator::targets: []
profile::icinga::status_file: /var/icinga-tmpfs/status.dat
profile::icinga::temp_file: /var/icinga-tmpfs/icinga.tmp
profile::icinga::temp_path: /var/icinga-tmpfs
profile::icinga::check_result_path: /var/icinga-tmpfs
profile::icinga::max_concurrent_checks: 0
profile::icinga::retention_file: /var/lib/icinga/retention.dat
profile::icinga::icinga_group: nagios
profile::icinga::icinga_user: nagios
profile::icinga::ensure_service: running
profile::atlasexporter::exporter_port: 9107
profile::dns::auth::update::netbox_dns_snippets_dir: /srv/git/netbox_dns_snippets
profile::dns::auth::update::netbox_exports_domain: %{alias('profile::netbox::automation::git_hostname')}
profile::poolcounter::exporter_port: 9106
profile::fastnetmon::thresholds_overrides: {"wmcs_eqiad_gw"=>{"hosts"=>["185.15.56.1/32"], "thresholds"=>{"pps"=>200000, "mbps"=>1000}}}
profile::services_proxy::envoy::listeners:*****Snipped for readability *****
profile::envoy::ensure: present
profile::spicerack::ganeti_rapi_timeout: 30
profile::puppetdb::jvm_opts: -Xmx4G
profile::puppetdb::elk_logging: true
profile::puppetdb::filter_job_id: false
profile::puppetdb::ca_path: /etc/ssl/certs/Puppet_Internal_CA.pem
profile::puppetdb::puppetboard_hosts: 
profile::puppetdb::ssldir: 
profile::prometheus::haproxy_exporter::listen_port: 9901
profile::prometheus::ops::ensure_rsync: absent
profile::prometheus::varnishkafka_exporter::stats_default: *****Snipped for readability *****
profile::prometheus::cadvisor_exporter::port: 4194
profile::librenms::dbuser: librenms
profile::librenms::dbhost: m1-master.eqiad.wmnet
profile::librenms::dbname: librenms
profile::openstack::eqiad1::region: eqiad1-r
profile::openstack::eqiad1::version: queens
profile::openstack::eqiad1::horizon_version: train
profile::openstack::eqiad1::nova_controller: cloudcontrol1003.wikimedia.org
profile::openstack::eqiad1::nova_controller_standby: cloudcontrol1004.wikimedia.org
profile::openstack::eqiad1::nova_api_host: cloudcontrol1003.wikimedia.org
profile::openstack::eqiad1::designate_host: cloudservices1003.wikimedia.org
profile::openstack::eqiad1::designate_host_standby: cloudservices1004.wikimedia.org
profile::openstack::eqiad1::second_region_designate_host: cloudservices1003.wikimedia.org
profile::openstack::eqiad1::second_region_designate_host_standby: cloudservices1004.wikimedia.org
profile::openstack::eqiad1::puppetmaster_hostname: puppetmaster.cloudinfra.wmflabs.org
profile::openstack::eqiad1::osm_host: wikitech.wikimedia.org
profile::openstack::eqiad1::rabbit_monitor_user: monitoring
profile::openstack::eqiad1::rabbit_file_handles: 8192
profile::openstack::eqiad1::ldap_hosts: ["ldap-labs.eqiad.wikimedia.org", "ldap-labs.codfw.wikimedia.org"]
profile::openstack::eqiad1::labweb_hosts: ["labweb1001.wikimedia.org", "labweb1002.wikimedia.org"]
profile::openstack::eqiad1::statsd_host: cloudmetrics1002.eqiad.wmnet
profile::openstack::eqiad1::statsd_host_standby: cloudmetrics1001.eqiad.wmnet
profile::openstack::eqiad1::observer_password: Fs6Dq2RtG8KwmM2Z
profile::openstack::eqiad1::labs_hosts_range: 10.64.20.0/24
profile::openstack::eqiad1::labs_hosts_range_v6: 2620:0:861:118::/64
profile::openstack::eqiad1::glance_host: cloudcontrol1003.wikimedia.org
profile::openstack::eqiad1::keystone_host: openstack.eqiad1.wikimediacloud.org
profile::openstack::eqiad1::all_regions: ["eqiad1-r"]
profile::openstack::base::designate::db_user: designate
profile::openstack::base::designate::db_name: designate
profile::openstack::base::designate::pdns_db_user: pdns
profile::openstack::base::designate::pdns_db_name: pdns
profile::openstack::base::designate::db_admin_user: pdns_admin
profile::openstack::base::keystone::db_name: keystone
profile::openstack::base::keystone::db_user: keystone
profile::openstack::base::keystone::db_max_pool_size: 4
profile::openstack::base::keystone::admin_workers: 8
profile::openstack::base::keystone::public_workers: 8
profile::openstack::base::keystone::auth_protocol: http
profile::openstack::base::keystone::auth_port: 35357
profile::openstack::base::keystone::public_port: 5000
profile::openstack::base::keystone::wiki_status_page_prefix: Nova_Resource:
profile::openstack::base::keystone::daemon_active: true
profile::openstack::base::keystone::wsgi_server: keystone
profile::openstack::base::keystone::admin_bind_port: 35357
profile::openstack::base::keystone::public_bind_port: 5000
profile::openstack::base::neutron::network_flat_tagged_base_interface: eth1
profile::openstack::base::neutron::db_user: neutron
profile::openstack::base::neutron::rabbit_user: neutron
profile::openstack::base::neutron::bridges: {}
profile::openstack::base::neutron::bridge_mappings: {}
profile::openstack::base::neutron::physical_interface_mappings: {}
profile::openstack::base::neutron::tld: wmflabs
profile::openstack::base::neutron::report_interval: 30
profile::openstack::base::neutron::agent_down_time: 64
profile::openstack::base::neutron::log_agent_heartbeats: True
profile::openstack::base::neutron::bind_port: 9696
profile::openstack::base::metrics::cpu_allocation_ratio: 1.0
profile::openstack::base::metrics::ram_allocation_ratio: 1.0
profile::openstack::base::metrics::disck_allocation_ratio: 1.0
profile::openstack::base::metrics::prometheus_listen_port: 12345
profile::openstack::base::metrics::cache_refresh_interval: 10
profile::openstack::base::metrics::cache_file: /var/cache/prometheus-openstack-exporter/base
profile::openstack::base::metrics::sched_ram_mbs: 4096
profile::openstack::base::metrics::sched_vcpu: 2
profile::openstack::base::metrics::sched_disk_gbs: 2
profile::openstack::base::haproxy::logging: true
profile::openstack::base::glance::api_bind_port: 9292
profile::openstack::base::glance::data_dir: /srv/glance
profile::openstack::base::glance::db_name: glance
profile::openstack::base::glance::db_user: glance
profile::openstack::base::glance::image_dir: /srv/glance/images
profile::openstack::base::glance::registry_bind_port: 9191
profile::openstack::base::nova::db_user: nova
profile::openstack::base::nova::db_name: nova
profile::openstack::base::nova::db_name_api: nova_api
profile::openstack::base::nova::compute_dev: /dev/md1
profile::openstack::base::nova::rabbit_user: nova
profile::openstack::base::nova::compute_workers: 3
profile::openstack::base::nova::metadata_listen_port: 8775
profile::openstack::base::nova::placement_api_port: 8778
profile::openstack::base::nova::metadata_workers: 3
profile::openstack::base::nova::osapi_compute_listen_port: 8774
profile::openstack::base::nova::scheduler_filters: ["ComputeFilter", "AvailabilityZoneFilter", "ServerGroupAffinityFilter", "ServerGroupAntiAffinityFilter", "ImagePropertiesFilter", "ComputeCapabilitiesFilter", "AggregateInstanceExtraSpecsFilter", "RamFilter", "CoreFilter", "DiskFilter"]
profile::openstack::base::pdns::db_host: localhost
profile::openstack::base::pdns::host_secondary: 
profile::openstack::eqiad1::metrics::cpu_allocation_ratio: 1.0
profile::openstack::eqiad1::metrics::ram_allocation_ratio: 1.0
profile::openstack::eqiad1::metrics::disck_allocation_ratio: 1.0
profile::openstack::eqiad1::metrics::prometheus_listen_port: 12345
profile::openstack::eqiad1::metrics::cache_refresh_interval: 10
profile::openstack::eqiad1::metrics::cache_file: /var/cache/prometheus-openstack-exporter/eqiad1
profile::openstack::eqiad1::metrics::sched_ram_mbs: 4096
profile::openstack::eqiad1::metrics::sched_vcpu: 2
profile::openstack::eqiad1::metrics::sched_disk_gbs: 2
profile::openstack::codfw1dev::version: queens
profile::openstack::codfw1dev::horizon_version: train
profile::openstack::codfw1dev::nova_controller: cloudcontrol2001-dev.wikimedia.org
profile::openstack::codfw1dev::nova_controller_standby: cloudcontrol2003-dev.wikimedia.org
profile::openstack::codfw1dev::glance_host: cloudcontrol2001-dev.wikimedia.org
profile::openstack::codfw1dev::keystone_host: openstack.codfw1dev.wikimediacloud.org
profile::openstack::codfw1dev::nova_api_host: cloudcontrol2001-dev.wikimedia.org
profile::openstack::codfw1dev::designate_host: cloudservices2002-dev.wikimedia.org
profile::openstack::codfw1dev::designate_host_standby: cloudservices2002-dev.wikimedia.org
profile::openstack::codfw1dev::second_region_designate_host: cloudservices2002-dev.wikimedia.org
profile::openstack::codfw1dev::second_region_designate_host_standby: cloudservices2002-dev.wikimedia.org
profile::openstack::codfw1dev::puppetmaster_hostname: labtest-puppetmaster.wikimedia.org
profile::openstack::codfw1dev::labs_hosts_range: 10.192.20.0/24
profile::openstack::codfw1dev::labs_hosts_range_v6: 2620:0:860:118::/64
profile::openstack::codfw1dev::spice_hostname: codfw1devspice.wikimedia.org
profile::openstack::codfw1dev::rabbit_monitor_user: monitoring
profile::openstack::codfw1dev::rabbit_file_handles: 8192
profile::openstack::codfw1dev::osm_host: labtestwikitech.wikimedia.org
profile::openstack::codfw1dev::ldap_hosts: ["cloudservices2002-dev.wikimedia.org"]
profile::openstack::codfw1dev::ldap_proxyuser_pass: Eche0ieng8UaNoo
profile::openstack::codfw1dev::region: codfw1dev-r
profile::openstack::codfw1dev::labweb_hosts: ["cloudweb2001-dev.wikimedia.org"]
profile::openstack::codfw1dev::all_regions: ["codfw", "codfw1dev-r"]
profile::openstack::codfw1dev::statsd_host: cloudmetrics1002.eqiad.wmnet
profile::openstack::base::ldap_proxyuser_pass: Eche0ieng8UaNoo
profile::openstack::base::region: eqiad1-r
profile::openstack::base::observer_user: novaobserver
profile::openstack::base::observer_project: observer
profile::openstack::base::ldap_base_dn: dc=wikimedia,dc=org
profile::openstack::base::ldap_user_id_attribute: uid
profile::openstack::base::ldap_user_name_attribute: cn
profile::openstack::base::ldap_user_dn: uid=novaadmin,ou=people,dc=wikimedia,dc=org
profile::openstack::base::version: queens
profile::netbox::db::primary: netboxdb1001.eqiad.wmnet
profile::netbox::db::secondaries: ["netboxdb2001.codfw.wmnet"]
profile::netbox::db::ipv6_ok: true
profile::netbox::automation::git_hostname: netbox-exports.wikimedia.org
profile::netbox::automation::dns_min_records: 2000
profile::ceph::admin_keyring: /etc/ceph/ceph.client.admin.keyring
profile::ceph::data_dir: /var/lib/ceph
profile::puppetmaster::common::storeconfigs: activerecord
jbond moved this task from Unsorted 💣 to Friday tasks on the User-jbond board.Apr 1 2020, 2:37 PM
Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM
jbond added a parent task: T285539: Easing pain points caused by divergence between cloudservices and production puppet usecases .Jun 25 2021, 11:41 AM
jbond closed this task as Resolved.Nov 3 2021, 11:52 AM
jbond claimed this task.

resolving this task, we do have a check however i think we should fix this by using via different means

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits