Page MenuHomePhabricator
Create Task
Maniphest T233203

Analyses octocatalog-diff output
Closed, ResolvedPublic
Actions

Description

analyse and fix any issues identified by otctocatalog-diff results in elnath:/root/1001_vs_1003/

octocatalog-diff.sh
#!/bin/bash

if [ ! -e "/root/wmf_prod_facts_json/${1}.yaml" ]
then
        printf "%s: does not exist\n" $1
        exit
fi
echo $1

octocatalog-diff -n ${1} \
        --from-puppet-master puppetmaster1001.eqiad.wmnet:8140 \
        --from-puppet-master-api-version 3 \
        --from-puppet-master-ssl-client-cert /var/lib/puppet/ssl/certs/elnath.codfw.wmnet.pem \
        --from-puppet-master-ssl-client-key /var/lib/puppet/ssl/private_keys/elnath.codfw.wmnet.pem \
        --from-puppet-master-ssl-ca /var/lib/puppet/ssl/certs/ca.pem \
        --to-puppet-master puppetmaster1003.eqiad.wmnet:8141 \
        --to-puppet-master-api-version 3 \
        --to-puppet-master-ssl-client-cert /var/lib/puppet/ssl/certs/elnath.codfw.wmnet.pem \
        --to-puppet-master-ssl-client-key /var/lib/puppet/ssl/private_keys/elnath.codfw.wmnet.pem \
        --to-puppet-master-ssl-ca /var/lib/puppet/ssl/certs/ca.pem \
        --puppetdb-api-version 4 \
        --from-fact-file /root/wmf_prod_facts_json/"${1}".yaml \
        --to-fact-file /root/wmf_prod_facts_json/"${1}".yaml \
        --from-save-catalog /root/cat/"${1}".from.json \
        --to-save-catalog /root/cat/"${1}".to.json \
        --ignore-attr 'require' \
        --ignore File[/etc/ssh/ssh_known_hosts] \
        --no-parallel \
        --debug 2>&1

Details

SubjectRepoBranchLines +/-
puppetmaster::frontend: add locale backend and promote puppetmaster1003operations/puppetproduction+19 -45
puppetmaster::frontend: add locale backendoperations/puppetproduction+18 -3
puppetmaster1003: promote puppetmaster1003 to a real puppetmaster backendoperations/puppetproduction+1 -46
prometheus - bastion: use per-resource default attributes not resource defaultsoperations/puppetproduction+25 -29
graphite: use per-resource default attributes not resource defaultsoperations/puppetproduction+64 -73
swift: use per-resource default attributes not resource defaultsoperations/puppetproduction+38 -60
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjbondT228657 Upgrade Puppet Masters and Puppet DB servers
 ResolvedjbondT233203 Analyses octocatalog-diff output

Event Timeline

jbond created this task.Sep 18 2019, 10:28 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptSep 18 2019, 10:28 AM
gerritbot added a comment.Sep 18 2019, 10:30 AM

Change 537618 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] prometheus - bastion: use per-resource defaiult attributes not resource defaults

https://gerrit.wikimedia.org/r/537618

gerritbot added a project: Patch-For-Review.Sep 18 2019, 10:30 AM
jbond added a comment.EditedSep 18 2019, 10:30 AM

there are a few nodes which have diffes like the following. This is caused when an attribute has a value of undef

diff production/cp1078.eqiad.wmnet production/cp1078.eqiad.wmnet
*******************************************
  Etcd::Client::Config[/etc/etcd/etcdrc] =>
   parameters =>
     settings =>
       ca_cert =>
        - undef
*******************************************
  Profile::Trafficserver::Logs[trafficserver_backend_logs] =>
   parameters =>
     paths =>
       base_path =>
        - undef
*******************************************
  Profile::Trafficserver::Monitoring[trafficserver_backend_monitoring] =>
   parameters =>
     paths =>
       base_path =>
        - undef
*******************************************
  Trafficserver::Instance[backend] =>
   parameters =>
     paths =>
       base_path =>
        - undef

When we look at the catalouge we see that the current puppet master includes the attribute with a value of "undef". however the new puppet master includes the attribute as null both will be interpretative as undef when the cataloge is applied

puppetmaster1001
{
         "line" : 35,
         "title" : "/etc/etcd/etcdrc",
         "file" : "/etc/puppet/modules/etcd/manifests/client/globalconfig.pp",
         "exported" : false,
         "type" : "Etcd::Client::Config",
         "parameters" : {
            "settings" : {
               "allow_reconnect" : true,
               "host" : "",
               "ca_cert" : "undef",
               "protocol" : "https",
               "srv_domain" : "conftool.eqiad.wmnet",
               "port" : ""
            },
            "group" : "root",
            "ensure" : "present",
            "owner" : "root",
            "world_readable" : true
         },
         "tags" : [
            "etcd::client::config",
            "etcd",
            "client",
            "config",
            "class",
            "etcd::client::globalconfig",
            "globalconfig",
            "profile::conftool::client",
            "profile",
            "conftool",
            "profile::cache::base",
            "cache",
            "base",
            "role::cache::upload",
            "role",
            "upload"
         ]
      }
puppetmaster1003
{
         "line" : 35,
         "tags" : [
            "etcd::client::config",
            "etcd",
            "client",
            "config",
            "class",
            "etcd::client::globalconfig",
            "globalconfig",
            "profile::conftool::client",
            "profile",
            "conftool",
            "profile::cache::base",
            "cache",
            "base",
            "role::cache::upload",
            "role",
            "upload"
         ],
         "parameters" : {
            "settings" : {
               "port" : "", 
               "srv_domain" : "conftool.eqiad.wmnet",
               "host" : "", 
               "protocol" : "https",
               "allow_reconnect" : true,
               "ca_cert" : null
            },
            "world_readable" : true,
            "ensure" : "present",
            "group" : "root",
            "owner" : "root"
         },
         "exported" : false,
         "title" : "/etc/etcd/etcdrc",
         "type" : "Etcd::Client::Config",
         "file" : "/etc/puppet/modules/etcd/manifests/client/globalconfig.pp"
      }
jbond updated the task description. (Show Details)Sep 18 2019, 10:54 AM
MoritzMuehlenhoff subscribed.Sep 18 2019, 11:00 AM
gerritbot added a comment.Sep 18 2019, 11:23 AM

Change 537629 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] swift: use per-resource defaiult attributes not resource defaults

https://gerrit.wikimedia.org/r/537629

jbond added a comment.EditedSep 18 2019, 12:22 PM

nodes with changes

HostChangesType of Changeremediation
graphite1004.eqiad.wmnet14Resource Defaultshttps://gerrit.wikimedia.org/r/c/operations/puppet/+/537641
bast3002.wikimedia.org4Resource Defaultshttps://gerrit.wikimedia.org/r/c/operations/puppet/+/537618
prometheus1003.eqiad.wmnetg4Resource Defaultshttps://gerrit.wikimedia.org/r/c/operations/puppet/+/537618
rdb1005.eqiad.wmnet5undef parametersNone
cp1078.eqiad.wmnet4undef parametersNone
ms-be1017.eqiad.wmnet2undef parametershttps://gerrit.wikimedia.org/r/c/operations/puppet/+/537629
maps1004.eqiad.wmnet2undef parametersNone
acrab.codfw.wmnet.log1undef parametersNone
acrux.codfw.wmnet.log1undef parametersNone
aqs1005.eqiad.wmnet.log1undef parametersNone
argon.eqiad.wmnet.log1undef parametersNone
bromine.eqiad.wmnet.log1undef parametersNone
chlorine.eqiad.wmnet.log1undef parametersNone
cloudweb2001-dev.wikimedia.org.log1undef parametersNone
conf2001.codfw.wmnet.log1undef parametersNone
contint1001.wikimedia.org.log1undef parametersNone
cp1079.eqiad.wmnet.log1undef parametersNone
cumin1001.eqiad.wmnet.log1undef parametersNone
deploy1001.eqiad.wmnet.log1undef parametersNone
druid1004.eqiad.wmnet.log1undef parametersNone
elastic1018.eqiad.wmnet.log1undef parametersNone
etcd1001.eqiad.wmnet.log1undef parametersNone
etherpad1001.eqiad.wmnet.log1undef parametersNone
grafana1001.eqiad.wmnet.log1undef parametersNone
icinga1001.wikimedia.org.log1undef parametersNone
kubestagetcd1001.eqiad.wmnet.log1undef parametersNone
kubetcd2001.codfw.wmnet.log1undef parametersNone
labweb1001.wikimedia.org.log1undef parametersNone
ldap-eqiad-replica01.wikimedia.org.log1undef parametersNone
maps1001.eqiad.wmnet.log1undef parametersNone
miscweb1001.eqiad.wmnet.log1undef parametersNone
ms-fe1006.eqiad.wmnet.log1undef parametersNone
mw1221.eqiad.wmnet.log1undef parametersNone
mwdebug1002.eqiad.wmnet.log1undef parametersNone
mwmaint1002.eqiad.wmnet.log1undef parametersNone
ores1002.eqiad.wmnet.log1undef parametersNone
people1001.eqiad.wmnet.log1undef parametersNone
phab1003.eqiad.wmnet.log1undef parametersNone
planet1001.eqiad.wmnet.log1undef parametersNone
proton1001.eqiad.wmnet.log1undef parametersNone
puppetmaster1001.eqiad.wmnet.log1undef parametersNone
pybal-test2001.codfw.wmnet.log1undef parametersNone
registry1001.eqiad.wmnet.log1undef parametersNone
rhodium.eqiad.wmnet.log1undef parametersNone
scb1002.eqiad.wmnet.log1undef parametersNone
schema1001.eqiad.wmnet.log1undef parametersNone
thumbor1002.eqiad.wmnet.log1undef parametersNone
vega.codfw.wmnet.log1undef parametersNone
wdqs1003.eqiad.wmnet.log1undef parametersNone
webperf1001.eqiad.wmnet.log1undef parametersNone
wtp1026.eqiad.wmnet.log1undef parametersNone

The vast majority of objects with one change relate to the following

*******************************************
  Etcd::Client::Config[/etc/etcd/etcdrc] =>
   parameters =>
     settings =>
       ca_cert =>
        - undef
*******************************************

The other hosts with on change relate to

*******************************************
  Envoyproxy::Tls_terminator[443] =>
   parameters =>
     upstreams =>
      - [{"server_names"=>["*"], "cert_path"=>"undef", "key_path"=>"undef", "upstream_port"=>80}]                                                                                   
      + [{"server_names"=>["*"], "cert_path"=>nil, "key_path"=>nil, "upstream_port"=>80}]
*******************************************
gerritbot added a comment.Sep 18 2019, 1:10 PM

Change 537641 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] graphite: use per-resource default attributes not resource defaults

https://gerrit.wikimedia.org/r/537641

gerritbot added a comment.Sep 18 2019, 3:32 PM

Change 537629 merged by Jbond:
[operations/puppet@production] swift: use per-resource default attributes not resource defaults

https://gerrit.wikimedia.org/r/537629

gerritbot added a comment.Sep 18 2019, 3:57 PM

Change 537641 merged by Jbond:
[operations/puppet@production] graphite: use per-resource default attributes not resource defaults

https://gerrit.wikimedia.org/r/537641

gerritbot added a comment.Sep 20 2019, 10:58 AM

Change 537618 merged by Jbond:
[operations/puppet@production] prometheus - bastion: use per-resource default attributes not resource defaults

https://gerrit.wikimedia.org/r/537618

gerritbot added a comment.Sep 23 2019, 10:26 AM

Change 538590 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: promote puppetmaster1003 to a real puppetmaster backend

https://gerrit.wikimedia.org/r/538590

jbond triaged this task as Medium priority.Sep 23 2019, 11:38 AM
gerritbot added a comment.Sep 23 2019, 1:23 PM

Change 538590 merged by Jbond:
[operations/puppet@production] puppetmaster1003: promote puppetmaster1003 to a real puppetmaster backend

https://gerrit.wikimedia.org/r/538590

gerritbot added a comment.Sep 23 2019, 2:48 PM

Change 538629 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster::frontend: add locale backend

https://gerrit.wikimedia.org/r/538629

gerritbot added a comment.Sep 23 2019, 2:51 PM

Change 538629 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster::frontend: add locale backend

https://gerrit.wikimedia.org/r/538629

gerritbot added a comment.Sep 23 2019, 3:24 PM

Change 538629 merged by Jbond:
[operations/puppet@production] puppetmaster::frontend: add locale backend

https://gerrit.wikimedia.org/r/538629

gerritbot added a comment.Sep 23 2019, 6:10 PM

Change 538686 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster::frontend: add locale backend and promote puppetmaster1003

https://gerrit.wikimedia.org/r/538686

gerritbot added a comment.Sep 24 2019, 1:49 PM

Change 538686 merged by Jbond:
[operations/puppet@production] puppetmaster::frontend: add locale backend and promote puppetmaster1003

https://gerrit.wikimedia.org/r/538686

jbond closed this task as Resolved.Sep 26 2019, 9:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL