Page MenuHomePhabricator
Create Task
Maniphest T93691

dhclient overwrites /etc/resolv.conf
Closed, ResolvedPublic
Actions

Description

I noticed repeated Puppet updates to /etc/resolv.conf, but /var/log/syslog showed only changes in one "direction" (here for tools-dev):

Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) 
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) --- /etc/resolv.conf#0112015-03-23 08:27:57.160205067 +0000
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +++ /tmp/puppet-file20150323-20208-ri9403-0#0112015-03-23 08:33:05.240215730 +0000
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) @@ -1,3 +1,9 @@
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +## THIS FILE IS MANAGED BY PUPPET
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +##
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +## source: modules/base/resolv.conf.labs.erb
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +## from:   base::resolving
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content)  domain eqiad.wmflabs
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) -search eqiad.wmflabs
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +search eqiad.wmflabs labs.eqiad.wmnet
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +options timeout:5 ndots:2
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content)  nameserver 10.68.16.1
Mar 23 08:33:05 tools-dev puppet-agent[20208]: FileBucket got a duplicate file {md5}62f5c3a6299680b7d0be8120fb03fa84
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]) Filebucketed /etc/resolv.conf to puppet with sum 62f5c3a6299680b7d0be8120fb03fa84
Mar 23 08:33:05 tools-dev puppet-agent[20208]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) content changed '{md5}62f5c3a6299680b7d0be8120fb03fa84' to '{md5}aab4d07473a21395f5bee957079fae9b'
[…]
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) 
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) --- /etc/resolv.conf#0112015-03-23 17:45:16.764937164 +0000
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +++ /tmp/puppet-file20150323-14358-s8oskv-0#0112015-03-23 17:53:19.856940248 +0000
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) @@ -1,3 +1,9 @@
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +## THIS FILE IS MANAGED BY PUPPET
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +##
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +## source: modules/base/resolv.conf.labs.erb
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +## from:   base::resolving
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content)  domain eqiad.wmflabs
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) -search eqiad.wmflabs
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +search eqiad.wmflabs labs.eqiad.wmnet
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) +options timeout:5 ndots:2
Mar 23 17:53:19 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content)  nameserver 10.68.16.1
Mar 23 17:53:20 tools-dev puppet-agent[14358]: FileBucket got a duplicate file {md5}62f5c3a6299680b7d0be8120fb03fa84
Mar 23 17:53:20 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]) Filebucketed /etc/resolv.conf to puppet with sum 62f5c3a6299680b7d0be8120fb03fa84
Mar 23 17:53:20 tools-dev puppet-agent[14358]: (/Stage[main]/Base::Resolving/File[/etc/resolv.conf]/content) content changed '{md5}62f5c3a6299680b7d0be8120fb03fa84' to '{md5}aab4d07473a21395f5bee957079fae9b'

Looking at the timestamps of the files showed a correlation with dhclient:

Mar 23 08:27:56 tools-dev dhclient: DHCPREQUEST of 10.68.16.8 on eth0 to 10.68.16.1 port 67
Mar 23 08:27:57 tools-dev dhclient: DHCPACK of 10.68.16.8 from 10.68.16.1
Mar 23 08:27:57 tools-dev dhclient: bound to 10.68.16.8 -- renewal in 33439 seconds.
[…]
Mar 23 17:45:16 tools-dev dhclient: DHCPREQUEST of 10.68.16.8 on eth0 to 10.68.16.1 port 67
Mar 23 17:45:16 tools-dev dhclient: DHCPACK of 10.68.16.8 from 10.68.16.1
Mar 23 17:45:16 tools-dev dhclient: bound to 10.68.16.8 -- renewal in 39803 seconds.

So it appears that dhclient in saving the nameserver information received during DHCP to /etc/resolv.conf does not only update the nameserver directive, but replaces the whole file instead.

Googling suggests that this is part of a greater Ubuntu scheme called "resolvconf" (cf. resolvconf(8)) where options like ndots & Co. need to go in /etc/resolvconf/resolv.conf.d/tail and search domains are a bit more complicated (all unconfirmed).

I'll play around with it for a bit to see what sticks.

Details

SubjectRepoBranchLines +/-
WIP: Snapshotoperations/puppetproduction+46 -16
Make labs resolv.conf play nice with resolvconfoperations/puppetproduction+37 -7
Disable automatic updating of resolv.confoperations/puppetproduction+47 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

scfc created this task.Mar 23 2015, 11:35 PM
scfc claimed this task.
scfc raised the priority of this task from to Low.
scfc updated the task description. (Show Details)
scfc added a project: Cloud-Services.
scfc subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 23 2015, 11:35 PM
scfc added a parent task: T63897: Move LabsDB aliases to DNS.Mar 23 2015, 11:36 PM
coren raised the priority of this task from Low to High.Mar 25 2015, 2:27 PM
coren subscribed.

Ew. Forgot that labs actually use dhclient unlike prod. This will need to be fixed before we can switch to designate.

coren moved this task from Triage to In Progress on the Cloud-Services board.Mar 25 2015, 2:42 PM
yuvipanda added a project: ToolLabs-Goals-Q4.Mar 28 2015, 8:51 PM
yuvipanda subscribed.
Andrew subscribed.Mar 30 2015, 3:20 PM

We may be able to change the default dns server on labnet1001's dhcp server. That will fix part of this.

The remainder can (I think) be fixed by adding appropriate settings to head/tail/whatever.

gerritbot subscribed.Mar 30 2015, 7:24 PM

Change 200648 had a related patch set uploaded (by Tim Landscheidt):
WIP: Snapshot

https://gerrit.wikimedia.org/r/200648

gerritbot added a project: Patch-For-Review.Mar 30 2015, 7:24 PM
scfc added a comment.Mar 30 2015, 7:30 PM

The problem with the whole setup (resolvconf and dhclient, that is) is that it does a lot of DWIM, and so I find it hard to get a consistent state to migrate to. resolvconf doesn't work if /etc/resolv.conf isn't a symlink to its own copy, however it has not created its own copy on a number of Tools instances, probably because the order of installing resolvconf, Puppet overwriting /etc/resolv.conf, etc., etc., etc. has been rather random. /sbin/ifdown eth0; /sbin/ifup eth0 will reliably trigger dhclient-script to run, but whether resolvconf is executed depends on this and that. There is an --enable-updates option that is barely documented and has this and that effect, etc.

TL&DR: It's complicated.

scfc added a comment.Mar 30 2015, 7:38 PM

And forgot: As /run where resolvconf stores its copy is on a temporary file system, the state of the instances depends on whether the instance was rebooted after the Puppet change that set /etc/resolv.conf was merged.

gerritbot added a comment.Mar 30 2015, 8:54 PM

Change 200595 had a related patch set uploaded (by Andrew Bogott):
Make labs resolv.conf play nice with resolvconf

https://gerrit.wikimedia.org/r/200595

Andrew added a comment.Mar 31 2015, 1:23 PM

I would love to just puppetize the file, but is murdering resolvconf enough? It seems like lots of other services can trigger random updates to resolv.conf

Andrew added a comment.Mar 31 2015, 1:24 PM

Also, how bitter am I that resolvconf has a --disable-updates argument but that it can't be disabled in /etc/defaults?

Dzahn subscribed.Mar 31 2015, 7:48 PM
modules/labs_vmbuilder/files/postinst.sh:  rm -f /etc/resolv.conf
modules/base/manifests/resolving.pp:        file { '/etc/resolv.conf':
modules/base/manifests/resolving.pp:                'labs'  => template('base/resolv.conf.labs.erb'),
modules/base/manifests/resolving.pp:                default => template('base/resolv.conf.erb'),
modules/labs_bootstrapvz/files/labs-jessie.manifest.yaml:      - ['chroot', '{root}', 'rm', '-f', '/etc/resolv.conf']
gerritbot added a comment.Mar 31 2015, 8:12 PM

Change 200999 had a related patch set uploaded (by Andrew Bogott):
Disable automatic updating of resolv.conf

https://gerrit.wikimedia.org/r/200999

Andrew added a comment.Mar 31 2015, 8:16 PM

I have some faith in that last patch -- hopefully no one will find it too offensive.

scfc added a comment.Mar 31 2015, 8:28 PM

We could disable it like you did in that patch with nodnsupdate, but I don't like plastering yet another layer over dhclient, resolvconf, and what else Ubuntu and Debian gods have come up with :-).

(I assume Labs is stable enough again to spin up test instances?)

Andrew added a comment.Mar 31 2015, 9:56 PM

I'm open to any suggestions -- I've spent all day trying to do it the 'right way' cooperating with dhclient and I'm done.

gerritbot added a comment.Apr 1 2015, 2:20 PM

Change 200999 merged by Andrew Bogott:
Disable automatic updating of resolv.conf

https://gerrit.wikimedia.org/r/200999

Andrew mentioned this in rOPUP50895a74ed7e: Disable automatic updating of resolv.conf.Apr 1 2015, 2:20 PM
gerritbot added a comment.Apr 1 2015, 5:15 PM

Change 200595 abandoned by Andrew Bogott:
Make labs resolv.conf play nice with resolvconf

Reason:
Abandoned in favor of murdering resolvconf entirely. https://gerrit.wikimedia.org/r/#/c/200999/

https://gerrit.wikimedia.org/r/200595

gerritbot added a comment.Apr 1 2015, 6:15 PM

Change 200648 abandoned by Tim Landscheidt:
WIP: Snapshot

Reason:
I26b3eb07c98cb09cbe75b80a8c22736a35ac4232 overwrites /sbin/resolvconf, so any further iteration of this change would need to force a reinstall of the resolvconf package. Probably somehow possible with Puppet, but no fun.

https://gerrit.wikimedia.org/r/200648

scfc closed this task as Resolved.Apr 1 2015, 6:23 PM
scfc reassigned this task from scfc to Andrew.
scfc set Security to None.

Verified on tools-exec-13 (Precise), tools-exec-21 (Trusty) and toolsbeta-jessie-test2 (Jessie) that ifdown eth0; ifup eth0 does not overwrite /etc/resolv.conf.

yuvipanda moved this task from Backlog to Tech Debt / Simplification on the ToolLabs-Goals-Q4 board.Apr 15 2015, 8:28 AM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits