confd fails with "no such host" in SRV lookup from _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Feb 13 2026, 10:05 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

Dzahn

Krinkle

Description

Since at least 19 January (oldest logs at time of writing), this has been spamming syslog on the codesearch9 vm

2026-01-19T00:00:51.145614+00:00 codesearch9 confd[4140034]: 2026-01-19T00:00:51Z codesearch9 /usr/bin/confd[4140034]: INFO SRV record set to _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud
2026-01-19T00:00:51.147887+00:00 codesearch9 confd[4140034]: 2026-01-19T00:00:51Z codesearch9 /usr/bin/confd[4140034]: FATAL Cannot get nodes from SRV records lookup _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud on 172.20.255.1:53: no such host
--
2026-01-19T00:01:01.370863+00:00 codesearch9 confd[4140040]: 2026-01-19T00:01:01Z codesearch9 /usr/bin/confd[4140040]: INFO SRV record set to _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud
2026-01-19T00:01:01.380739+00:00 codesearch9 confd[4140040]: 2026-01-19T00:01:01Z codesearch9 /usr/bin/confd[4140040]: FATAL Cannot get nodes from SRV records lookup _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud on 172.20.255.1:53: no such host
--
2026-01-19T00:01:11.629049+00:00 codesearch9 confd[4140048]: 2026-01-19T00:01:11Z codesearch9 /usr/bin/confd[4140048]: INFO SRV record set to _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud
2026-01-19T00:01:11.631151+00:00 codesearch9 confd[4140048]: 2026-01-19T00:01:11Z codesearch9 /usr/bin/confd[4140048]: FATAL Cannot get nodes from SRV records lookup _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud on 172.20.255.1:53: no such host

It seems to happen every 10 seconds, which checks out given we have ~8000 entries per day:

krinkle@codesearch9:~$ sudo zgrep 'Cannot get nodes from SRV records lookup _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud on 172.20.255.1:53: no such host' /var/log/syslog.19.gz | wc -l
8434
krinkle@codesearch9:~$ sudo zgrep 'Cannot get nodes from SRV records lookup _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud on 172.20.255.1:53: no such host' /var/log/syslog.18.gz | wc -l
8274

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	codesearch: stop confd spamming syslog	Dzahn	operations/puppet	production	+5 -0

Customize query in gerrit

Related Objects

Mentioned In: T395945: deployment-webperf21 logspam from /usr/bin/confd: "WARNING Found no templates"
T356296: confd setup left without configuration doesn't stop confd
T417397: Codesearch is regularly partially or fully unavailable
Mentioned Here: T356296: confd setup left without configuration doesn't stop confd
T116224: etcd/confd is not started on beta cluster Varnish caches
T196596: confd broken on deployment-redis hosts

Event Timeline

Krinkle created this task.Feb 13 2026, 10:05 PM

Herald added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2026, 10:05 PM

Krinkle mentioned this in T417397: Codesearch is regularly partially or fully unavailable.Feb 13 2026, 10:06 PM

Krinkle updated the task description. (Show Details)Feb 13 2026, 11:14 PM

(since it affects other confd's in other projects) is this related to work on T356296 ?

is confd even supposed to be on this instance?

Dzahn mentioned this in T356296: confd setup left without configuration doesn't stop confd.Feb 13 2026, 11:26 PM

modules/confd/templates/initscripts/confd.systemd.erb:Environment="CONFD_DISCOVERY=-srv-record _etcd-client-ssl._tcp.<%= @srv_dns %> -scheme https"

This confd.systemd.erb has not changed since 2023.

And yes, this is actually not in DNS:

Host _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud not found: 3(NXDOMAIN)

It seems like something must have changed in cloud VPS DNS.

Krinkle added a project: Collaboration-Services.Feb 14 2026, 1:43 AM

Dzahn moved this task from Incoming to Backlog on the Collaboration-Services board.Mar 3 2026, 7:41 PM

Mentioned in SAL (#wikimedia-cloud) [2026-03-03T23:49:39Z] <mutante> codesearch9 systemctl mask confd (T417458)

masked the confd service.

this removes the "FATALs spam" (tail -f /var/log/syslog | grep FATAL)

Mentioned in SAL (#wikimedia-cloud) [2026-03-03T23:56:03Z] <mutante> nevermind - puppet restarts it (debug) (T417458)

in the template for the systemd unit:

<%- if @scheme == "https" -%>
Environment="CONFD_DISCOVERY=-srv-record _etcd-client-ssl._tcp.<%= @srv_dns %> -scheme https"
<%- else -%>
Environment="CONFD_DISCOVERY=-srv-record _etcd-client._tcp.<%= @srv_dns %>"

so srv_dns is codesearch.eqiad1.wikimedia.cloud.

from profile::confd:

class profile::confd (
    String             $prefix    = lookup('conftool_prefix'),
    Hash[String, Hash] $instances = lookup('profile::confd::instances'),
) {
    # inject default values and ensure we always have a main section
    $defaults = {
        'main'    => {
            'prefix'  => $prefix,
            'srv_dns' => "${::site}.wmnet",
        },
    }
    class { 'confd':
        instances => deep_merge($defaults, $instances),

A couple other roles in cloud set profile::confd::instances defaults or {}, which I tried in Horizon Hiera at host level but had no effect.

I found I still had an open tab with a comment that I never submitted and was from a while back. Saving it now.

This gets pulled in from profile::firewall and is related to changes around $defs_from_etcd which can pull in network ranges for firewall rules but is not used here.

I set confd::ensure: absent in Horizon Hiera on project puppet level.

Mentioned in SAL (#wikimedia-cloud) [2026-05-29T17:32:29Z] <mutante> - adding "confd::ensure: absent" to Horizon Hiera at Project Puppet level - T417458

Change #1295492 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] codesearch: stop confd spamming syslog

https://gerrit.wikimedia.org/r/1295492

Change #1295492 merged by Dzahn:

[operations/puppet@production] codesearch: stop confd spamming syslog

https://gerrit.wikimedia.org/r/1295492

Figured out the fix is adding this to Hiera instead:

profile::confd::instances:
  main:
    interval: 300

This finally stopped the syslog spam.

I tested in Horizon and added it in the repo now.

Dzahn closed this task as Resolved.May 29 2026, 6:21 PM

Dzahn claimed this task.

Maintenance_bot removed a project: Patch-For-Review.May 29 2026, 6:30 PM

Dzahn mentioned this in T395945: deployment-webperf21 logspam from /usr/bin/confd: "WARNING Found no templates".Jun 30 2026, 11:49 PM

confd fails with "no such host" in SRV lookup from _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloudClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

confd fails with "no such host" in SRV lookup from _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud
Closed, ResolvedPublic
Actions