Page MenuHomePhabricator
Create Task
Maniphest T399106

Puppet edit war in deployment-deploy04:/etc/ssh/ssh_known_hosts
Closed, DeclinedPublicBUG REPORT
Actions

Description

/etc/ssh/ssh_known_hosts is edited on every puppet run, even when two runs are done in immediate succession:

bd808@deployment-deploy04:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(6a2a8b7809) gitpuppet - beta: Add redirect for upload.wikimedia.beta.wmflabs.org'
Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:
--- /etc/ssh/ssh_known_hosts    2025-07-09 16:14:24.498717158 +0000
+++ /tmp/puppet-file20250709-2299892-1o14kv3    2025-07-09 16:15:08.463884484 +0000
@@ -1,6 +1,3 @@
-# HEADER: This file was autogenerated at 2025-07-09 16:14:24 +0000
-# HEADER: by puppet.  While it can still be managed manually, it
-# HEADER: is definitely not recommended.
 # This file is managed by puppet
 deleteme.deployment-prep.eqiad1.wikimedia.cloud,deleteme,172.16.18.93,2a02:ec80:a000:1::2a1 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXZOSVNVM8KIpA17ef/PtRw51G+OaIEMnSky4t4Hsh1rb2XL0iy/DckeNSCGiJXk3gLKFatQPhEsPU96PFKmGI=
 deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud,deployment-acme-chief05,172.16.4.3,fe80::f816:3eff:fe8e:e4dc ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNKxFW8xm8BypAq0/myF9ofyBgQ27G0WpM5Kc2bTm/nsiDgyJKc+QeN39JCMUCpiw6ggvV6EedLNLi6SjLxSCdE=
@@ -63,4 +60,3 @@
 deployment-wikifeeds02.deployment-prep.eqiad1.wikimedia.cloud,deployment-wikifeeds02,172.16.4.35,fe80::f816:3eff:fe6a:3c52 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRov6bw0g/1fTFuFysHF1jEshVWHHbAU6ZJAPKRxE+sKuIBQn3UqGzUgay1DYed+TPErViVwuDasJwv5/S2OJE=
 deployment-zookeeper-3.deployment-prep.eqiad1.wikimedia.cloud,deployment-zookeeper-3,172.16.6.11,fe80::f816:3eff:feed:a89d ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAn5JXibRrHVMLH63TyxRHP9ZuJ8cRoTRoCoOfdUZ/zwDMirB1JZBNA5b04FpCdbs6y5X/QDTMS3kOEaBSsAYVo=
 traindev01.deployment-prep.eqiad1.wikimedia.cloud,traindev01,172.16.3.141,fe80::f816:3eff:fe15:ff05 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLWaOfpDdGdHFgqDYixzVKaNvWGaw+A8pGhM2l1f9toFXsDxzW7JCpIHVIVMgthgB5GWqUPJdhk6uwX+7q9o0Q=
-gerrit.wikimedia.org,208.80.154.151,2620:0:861:2:208:80:154:151 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ==

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content: content changed '{sha256}4a665620372128490b7f8060e5f807369959d827fc563369d427f9726d974f4e' to '{sha256}a9cb64a3b6420570a9246d100f66128b502bf5bcfaf8b1fb46709c8468e02f61' (corrective)
Notice: /Stage[main]/Profile::Ci::Slave::Labs::Common/Profile::Gerrit::Sshkey[gerrit]/Sshkey[gerrit.wikimedia.org@ssh-rsa]/ensure: created (corrective)
Info: Computing checksum on file /etc/ssh/ssh_known_hosts
Notice: /Stage[main]/Beta::Autoupdater/Git::Clone[beta-mediawiki-skins]/Exec[git_set_origin_beta-mediawiki-skins]/returns: executed successfully (corrective)
Notice: Applied catalog in 16.44 seconds
bd808@deployment-deploy04:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(6a2a8b7809) gitpuppet - beta: Add redirect for upload.wikimedia.beta.wmflabs.org'
Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:
--- /etc/ssh/ssh_known_hosts    2025-07-09 16:15:15.872081232 +0000
+++ /tmp/puppet-file20250709-2305652-1pq4h7q    2025-07-09 16:15:45.832877081 +0000
@@ -1,6 +1,3 @@
-# HEADER: This file was autogenerated at 2025-07-09 16:15:15 +0000
-# HEADER: by puppet.  While it can still be managed manually, it
-# HEADER: is definitely not recommended.
 # This file is managed by puppet
 deleteme.deployment-prep.eqiad1.wikimedia.cloud,deleteme,172.16.18.93,2a02:ec80:a000:1::2a1 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXZOSVNVM8KIpA17ef/PtRw51G+OaIEMnSky4t4Hsh1rb2XL0iy/DckeNSCGiJXk3gLKFatQPhEsPU96PFKmGI=
 deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud,deployment-acme-chief05,172.16.4.3,fe80::f816:3eff:fe8e:e4dc ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNKxFW8xm8BypAq0/myF9ofyBgQ27G0WpM5Kc2bTm/nsiDgyJKc+QeN39JCMUCpiw6ggvV6EedLNLi6SjLxSCdE=
@@ -63,4 +60,3 @@
 deployment-wikifeeds02.deployment-prep.eqiad1.wikimedia.cloud,deployment-wikifeeds02,172.16.4.35,fe80::f816:3eff:fe6a:3c52 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRov6bw0g/1fTFuFysHF1jEshVWHHbAU6ZJAPKRxE+sKuIBQn3UqGzUgay1DYed+TPErViVwuDasJwv5/S2OJE=
 deployment-zookeeper-3.deployment-prep.eqiad1.wikimedia.cloud,deployment-zookeeper-3,172.16.6.11,fe80::f816:3eff:feed:a89d ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAn5JXibRrHVMLH63TyxRHP9ZuJ8cRoTRoCoOfdUZ/zwDMirB1JZBNA5b04FpCdbs6y5X/QDTMS3kOEaBSsAYVo=
 traindev01.deployment-prep.eqiad1.wikimedia.cloud,traindev01,172.16.3.141,fe80::f816:3eff:fe15:ff05 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLWaOfpDdGdHFgqDYixzVKaNvWGaw+A8pGhM2l1f9toFXsDxzW7JCpIHVIVMgthgB5GWqUPJdhk6uwX+7q9o0Q=
-gerrit.wikimedia.org,208.80.154.151,2620:0:861:2:208:80:154:151 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ==

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content: content changed '{sha256}808f73391890c009911def5e67d1dc639f1fdf516bbe28404191641f00eee8be' to '{sha256}a9cb64a3b6420570a9246d100f66128b502bf5bcfaf8b1fb46709c8468e02f61' (corrective)
Notice: /Stage[main]/Profile::Ci::Slave::Labs::Common/Profile::Gerrit::Sshkey[gerrit]/Sshkey[gerrit.wikimedia.org@ssh-rsa]/ensure: created (corrective)
Info: Computing checksum on file /etc/ssh/ssh_known_hosts
Notice: Applied catalog in 17.58 seconds

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
deployment-prep: mark gerrit key as being exportedHasharoperations/puppetproduction+2 -0
Allow setting profile::gerrit::sshkey::exportedHasharoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

• bd808 created this task.Jul 9 2025, 4:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2025, 4:17 PM
• bd808 triaged this task as Medium priority.Jul 9 2025, 4:18 PM
• bd808 added a comment.Jul 9 2025, 4:29 PM

The # This file is managed by puppet version is output from ::ssh::client:

modules/ssh/manifests/client.pp
if $manage_ssh_keys and wmflib::have_puppetdb() {
    file { '/etc/ssh/ssh_known_hosts':
        ensure  => file,
        content => template('ssh/known_hosts.erb'),
        backup  => false,
        owner   => 'root',
        group   => 'root',
        mode    => '0644',
    }
}
• bd808 added a comment.Jul 9 2025, 4:47 PM

The host key for gerrit that is removed during each run is from ::profile::gerrit::sshkey:

modules/profile/manifests/gerrit/sshkey.pp
# SPDX-License-Identifier: Apache-2.0
# @summary small wrapper to manage the gerrit key
# @param ensure ensurable param
# @param exported wether to export the resource
# @param override the default target
define profile::gerrit::sshkey (
    Wmflib::Ensure             $ensure   = 'present',
    Boolean                    $exported = false,
    Optional[Stdlib::Unixpath] $target   = undef,
) {
    $params = {
        'ensure' => $ensure,
        'name'   => 'gerrit.wikimedia.org',
        'host_aliases' => [
            ipresolve('gerrit.wikimedia.org', 4),
            ipresolve('gerrit.wikimedia.org', 6),
        ],
        'key'    => 'AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ==',
        'type'   => 'ssh-rsa',
        'target' => $target,
    }

    if $exported {
        @@sshkey { $title:
            * => $params,
        }
    } else {
        sshkey { $title:
            * => $params,
        }
    }
}

That profile adds the gerrit.wikimedia.org host key using the sshkey type and provider which is a puppet/provider/parsedfile subclass. The profile is applied as part of profile::ci::slave::labs::common which is applied to deployment-deploy04 because that instance is a Jenkins runner node so that Jenkins can run scap and wmf-beta-update-databases.py from scheduled jobs.

• bd808 closed this task as Declined.Jul 9 2025, 4:49 PM

This is noisy, but not harmful. The Jenkins integration will be going away at some point because of the Continuous-Integration-Infrastructure (Zuul upgrade) project.

hashar reopened this task as Open.Jul 9 2025, 5:14 PM
hashar added subscribers: jbond, hashar.

@jbond did the magic with https://gerrit.wikimedia.org/r/c/operations/puppet/+/965122

The trick is:

if $exported {
    @@sshkey { $title:
        * => $params,
    }
} else {
    sshkey { $title:
        * => $params,
    }
}

In Production it is exported since there is a PuppetDB. In WMCS that depends on whether a Puppet DB server has been set up and my guess is that is the case for the deployment server. So in that specific case, the key should be marked as exported as well but we got:

modules/profile/manifests/ci/slave/labs/common.pp
profile::gerrit::sshkey { 'gerrit': }

It is not marked as being exported.

And the manifest default to exported = false:

modules/profile/manifests/gerrit/sshkey.pp
define profile::gerrit::sshkey (
    Wmflib::Ensure             $ensure   = 'present',
    Boolean                    $exported = false,

So my guess is that on deployment-prep we can set profile::gerrit::sshkey::exported: true?

hashar added a comment.Jul 9 2025, 5:17 PM

Or maybe nowadays there is always a Puppet export :)

gerritbot added a comment.Jul 9 2025, 5:22 PM

Change #1167682 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] Allow setting profile::gerrit::sshkey::exported

https://gerrit.wikimedia.org/r/1167682

gerritbot added a project: Patch-For-Review.Jul 9 2025, 5:22 PM

Change #1167683 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] deployment-prep: mark gerrit key as being exported

https://gerrit.wikimedia.org/r/1167683

gerritbot added a comment.Jul 9 2025, 5:28 PM

Change #1167682 abandoned by Hashar:

[operations/puppet@production] Allow setting profile::gerrit::sshkey::exported

Reason:

Per Bryan: Puppet supports it out of the box

https://gerrit.wikimedia.org/r/1167682

gerritbot added a comment.Jul 9 2025, 5:29 PM

Change #1167683 abandoned by Hashar:

[operations/puppet@production] deployment-prep: mark gerrit key as being exported

Reason:

This was set on the affect host :)

https://gerrit.wikimedia.org/r/1167683

• bd808 added a comment.Jul 9 2025, 5:29 PM

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/be41e35bc911d272a15a58f8108367661e0861d9%5E%21/#F0

diff --git a/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml b/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml
index 1c2ae4a..dd99973 100644
--- a/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml
+++ b/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml

@@ -1,5 +1,5 @@
-aptly::manage_nginx: false
 profile::base::labs::cleanup_puppet_client_bucket: true
 profile::ci::slave::labs::common::manage_srv: false
+profile::gerrit::sshkey::exported: true
 profile::kubernetes::client::version: '1.23'
 profile::mediawiki::scap_client::is_master: true

The aptly::manage_nginx: false removal is T399076: Cleanup Aptly server from deployment-prep stuff that I spotted.

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 5:31 PM
• bd808 added a comment.Jul 9 2025, 5:40 PM
In T399106#10989259, @hashar wrote:

@jbond did the magic with https://gerrit.wikimedia.org/r/c/operations/puppet/+/965122

The trick is:

if $exported {
    @@sshkey { $title:
        * => $params,
    }
} else {
    sshkey { $title:
        * => $params,
    }
}

In Production it is exported since there is a PuppetDB. In WMCS that depends on whether a Puppet DB server has been set up and my guess is that is the case for the deployment server. So in that specific case, the key should be marked as exported as well but we got:

modules/profile/manifests/ci/slave/labs/common.pp
profile::gerrit::sshkey { 'gerrit': }

It is not marked as being exported.

And the manifest default to exported = false:

modules/profile/manifests/gerrit/sshkey.pp
define profile::gerrit::sshkey (
    Wmflib::Ensure             $ensure   = 'present',
    Boolean                    $exported = false,

So my guess is that on deployment-prep we can set profile::gerrit::sshkey::exported: true?

Something else is needed in modules/ssh/manifests/client.pp and/or modules/ssh/templates/known_hosts.erb to make use of the export of @@sshkey. The current logic somehow gets a known_hosts hash from PuppetDB. Something needs to collect the @@sshkey export with Sshkey <<| |>> and then use the collected fragments. This doesn't seem quite right however because the @@sshkey is already editing the hosts file.

• bd808 closed this task as Declined.Jul 10 2025, 11:52 PM

I don't think this is worth chasing.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits