Page MenuHomePhabricator
Paste P5608

Update production known hosts
ArchivedPublic

Authored by Volans on Jun 21 2017, 2:11 PM.
Tags
None
Referenced Files
F31011041: raw.txt
Nov 5 2019, 4:26 PM
F31011011: raw.txt
Nov 5 2019, 4:21 PM
F29275161: raw.txt
May 30 2019, 8:50 AM
F15953242: Update production known hosts
Mar 21 2018, 11:05 PM
F13738525: Update production known hosts
Feb 14 2018, 12:09 PM
F8544006: Update production known hosts
Jun 28 2017, 11:24 AM
F8538302: Update production known hosts
Jun 27 2017, 4:44 PM
F8505455: Update production known hosts
Jun 22 2017, 11:28 AM
Tokens
"Love" token, awarded by hashar."Love" token, awarded by dcausse.
#!/bin/bash
##############################################################################
# WMF Update production known hosts
#
# DESCRIPTION:
# - Populate a known_hosts file with all the production hosts and services
# in the Wikimedia Foundation production infrastructure for easy
# autocompletion while keeping StrictHostKeyChecking active:
# - sync all the known hosts from a bastion
# - clean the hostname without FQDN in it
# - optionally generate known hosts for services defined as CNAMEs in the
# DNS repository, see PARAMS below. This allows for the autocompletion of
# active/passive services like icinga.wikimedia.org.
# - Silently ignore all CNAMEs to the main DYNA record (dyna.wikimedia.org.)
# - Keeps a backup file with the previous known hosts
# - Show a diff between the new known hosts and the current ones
#
# It saves the known hosts into KNOWN_HOST_FILE, adjust this and/or the
# UserKnownHostsFile parameter in your ~/.ssh/config in order for them to
# match. A warning will be shown if they don't match.
#
# By default only the hosts from the choosen BASTION_HOST known_hosts file
# will be imported, cleaning the hostname (not the FQDN) to ease the auto-
# completion when ssh-ing.
#
# PARAMS:
# It accept one positional argument that, if specified, must be the path to
# a local clone of the Operations DNS repository, (either from Gerrit or from
# GitHub):
# https://gerrit.wikimedia.org/r/operations/dns
# In this case also the services defined as CNAMEs in the wikimedia.org and
# wmnet zone files will be added with the identity of the target host, if
# that is found in the known_hosts file, skipping the missing ones.
#
# USAGE:
# wmf-update-prod-known-hosts [PATH_TO_DNS_REPOSITORY]
#
# Author: Riccardo Coccioli <rcoccioli@wikimedia.org>
# Date: 2017-06-21
# Last update: 2019-11-05
# Dependencies: colordiff
# Version: 1.2
# License: GPLv3+
##############################################################################
set -e
DNS_REPO_PATH="${1}"
KNOWN_HOSTS_PATH="${HOME}/.ssh/known_hosts.d"
KNOWN_HOST_FILE="${KNOWN_HOSTS_PATH}/wmf-prod"
BASTION_HOST="bast2002.wikimedia.org"
MAIN_DYNA_RECORD="dyna.wikimedia.org."
if [[ ! -d "${KNOWN_HOSTS_PATH}" ]]; then
echo "ERROR: KNOWN_HOSTS_PATH '${KNOWN_HOSTS_PATH}' is not a directory, you might want to adjust the constant in the script or create it"
exit 1
fi
if [[ -n "${DNS_REPO_PATH}" ]]; then
if [[ ! -d "${DNS_REPO_PATH}" ]]; then
echo "ERROR: DNS_REPO_PATH '${DNS_REPO_PATH}' is not a directory"
exit 2
fi
if ! git -C "${DNS_REPO_PATH}" remote -v | egrep '(gerrit.wikimedia.org|github.com\/wikimedia)' | grep -cq 'operations[/-]dns'; then
echo "ERROR: DNS_REPO_PATH '${DNS_REPO_PATH}' doesn't seems to be a checkout of the operations/dns repository"
exit 3
fi
fi
function parse_line() {
local line="${1}"
local domain="${2}"
local name
local target
local found
name="$(echo "${line}" | cut -d' ' -f1)"
target="$(echo "${line}" | cut -d' ' -f2)"
if [[ "${target}" == "${MAIN_DYNA_RECORD}" ]]; then
# Silently ignore CNAMEs to the MAIN_DYNA_RECORD
return
fi
sep="\."
if [[ "${target: -1}" == '.' ]]; then
target="${target%?}"
sep=","
fi
set +e
found=$(grep -c "^${target}${sep}" "${KNOWN_HOST_FILE}.new")
set -e
if [[ "${found}" -eq "0" || "${found}" -gt "1" ]]; then
>&2 echo "Skipping '${target}' CNAME target, found ${found}/1 matches"
return
fi
grep "^${target}${sep}" "${KNOWN_HOST_FILE}.new" | awk -v name="${name}" -v domain="${domain}" '{ printf name"."domain; for (i = 2; i <= NF; i++) printf FS$i; print NL }'
}
function extract_cnames_from_zone() {
local zone_file
local origin
local boundaries
local start
local end
local domain
zone_file="${1}"
if [[ ! -f "${zone_file}" ]]; then
>&2 echo "Unable to find zone file ${zone_file}, skipping..."
return
fi
origin="${2}"
if [[ -n "${origin}" ]]; then
boundaries="$(grep -n "\$ORIGIN" "${zone_file}" | grep -A 1 "\$ORIGIN ${origin}\.$")"
start=$(echo "${boundaries}" | head -n1 | cut -d':' -f1)
end=$(echo "${boundaries}" | tail -n1 | cut -d':' -f1)
domain="${origin}"
head -n "${end}" "${zone_file}" | tail -n "$((end - start))" | awk '/ CNAME / { print $1, $5 }' | while read -r line; do
parse_line "${line}" "${domain}" >> "${KNOWN_HOST_FILE}.new"
done
else
domain="$(basename "${zone_file}")"
awk '/ CNAME / { print $1, $5 }' "${zone_file}" | while read -r line; do
parse_line "${line}" "${domain}" >> "${KNOWN_HOST_FILE}.new"
done
fi
}
# Get new known hosts
echo "===> SSHing to ${BASTION_HOST} (if a smartcard input is needed, check it now)"
ssh "${BASTION_HOST}" 'cat /etc/ssh/ssh_known_hosts' > "${KNOWN_HOST_FILE}.new"
# Remove the non-FQDN hostnames to avoid multiple autocompletions
awk -F ',' '{ printf $1; for (i = 3; i <= NF; i++) printf FS$i; print NL }' "${KNOWN_HOST_FILE}.new" > "${KNOWN_HOST_FILE}.new.clean"
mv -f "${KNOWN_HOST_FILE}.new.clean" "${KNOWN_HOST_FILE}.new"
if [[ -n "${DNS_REPO_PATH}" ]]; then
extract_cnames_from_zone "${DNS_REPO_PATH}/templates/wikimedia.org"
extract_cnames_from_zone "${DNS_REPO_PATH}/templates/wmnet" "eqiad.wmnet"
fi
PREV_COUNT=0
PREV_FILE=/dev/null
if [[ -f "${KNOWN_HOST_FILE}" ]]; then
PREV_COUNT="$(wc -l "${KNOWN_HOST_FILE}")"
PREV_FILE="${KNOWN_HOST_FILE}"
fi
echo "==== DIFFERENCES ===="
colordiff --fakeexitcode "${PREV_FILE}" "${KNOWN_HOST_FILE}.new"
echo "====================="
echo "Going from ${PREV_COUNT} to $(wc -l "${KNOWN_HOST_FILE}.new") known hosts and services"
if [[ -f "${KNOWN_HOST_FILE}" ]]; then
mv -vf "${KNOWN_HOST_FILE}" "${KNOWN_HOST_FILE}.old"
echo "Backup file is ${KNOWN_HOST_FILE}.old"
fi
mv -v "${KNOWN_HOST_FILE}.new" "${KNOWN_HOST_FILE}"
echo "New file generated at ${KNOWN_HOST_FILE}"
if ! egrep -cq "UserKnownHostsFile .*/wmf-prod( |$)" "${HOME}/.ssh/config"; then
echo "WARNING: You may need to add/update 'UserKnownHostsFile ${KNOWN_HOST_FILE}' to your ~/.ssh/config"
fi
if [[ "${SHELL}" == '/usr/bin/zsh' ]]; then
echo 'Add this line to your .zshrc to tab-complete remote hosts:'
echo "zstyle ':completion:*:hosts' known-hosts-files ${HOME}/.ssh/known_hosts ${KNOWN_HOST_FILE}"
fi
exit 0

Event Timeline

Volans created this object with visibility "SRE (Project)".
Volans changed the visibility from "SRE (Project)" to "Subscribers".Jun 21 2017, 3:28 PM
Volans removed a subscriber: SRE.
Volans changed the visibility from "Subscribers" to "Public (No Login Required)".Oct 16 2018, 9:04 PM

@Volans should this really be world-editable? 🤔

Volans changed the edit policy from "All Users" to "acl*sre-team (Project)".Dec 7 2018, 10:06 PM
In P5608#46552, @Neil_P._Quinn_WMF wrote:

@Volans should this really be world-editable? 🤔

Of course not, I've fixed it. Thanks for spotting this, I thought I've made it just readable by anyone.

In P5608#46554, @Volans wrote:
In P5608#46552, @Neil_P._Quinn_WMF wrote:

@Volans should this really be world-editable? 🤔

Of course not, I've fixed it. Thanks for spotting this, I thought I've made it just readable by anyone.

Glad to help! I was just wondering if I was missing something.

hashar subscribed.

I have been referred to this paste. Maybe it could be useful to have it in operations/puppet.git under ./utils ? :)

@hashar it's linked in https://wikitech.wikimedia.org/wiki/Production_shell_access#Known_host_files I was actually hoping we could create a repository for tool that should live and run from within our local environments and are generic and not tied to any specific other repo. So in the meanwhile I've put it here :) Open for suggestions.

In P5608#46704, @Volans wrote:

I was actually hoping we could create a repository for tool that should live and run from within our local environments and are generic and not tied to any specific other repo.

That sounds like an excellent idea. It's very much in line with Developer Productivity (cc @jeena) and it's something I've wanted to set up for a very long time. I've recently settled for writing Scap plugins in various repositories or just adding something to ~/bin. Neither is a satisfactory collaborative and reusable method for advancing our collective local dev environments.

...this comment got long so I'm breaking it out into a task: T212016

Paste diff is not that smart, I just added a check for the main DYNA record to silently skip its CNAMEs without spamming stderr.

The script has been moved to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/wmf-sre-laptop/+/refs/heads/master/scripts/wmf-update-known-hosts-production

If you're an SRE and run a Linux OS you're encouraged to follow https://wikitech.wikimedia.org/wiki/Wmf-sre-laptop, for all other use cases just use the version of the repo as the source of truth for the latest version of the script.
I've also updated the related wikitech documentation.

This paste is now archived.