Page MenuHomePhabricator

Update DNS/NTP servers on the esams PDUs/SCS
Closed, ResolvedPublic

Description

Per @BBlack, during the recent esams rebuild:

  • The esams NTP server IPs did change in the process of rebuilding the site: they were 91.198.174.106 and 91.198.174.122, now they're 91.198.174.61 and 91.198.174.62.
  • All DNS client config should use the single nameserver IP "10.3.0.1" everywhere globally.

For server infrastructure, this was deployed with Puppet. For network gear, this will deployed with Homer.

That leaves esams' a) PDUs b) SCS that need to be updated manually.

Details

Related Gerrit Patches:
operations/homer/public : masterMake NTP servers a variable and update esams IPs

Event Timeline

faidon triaged this task as Normal priority.Thu, Oct 31, 1:45 PM
faidon created this task.
Restricted Application added a project: Operations. · View Herald TranscriptThu, Oct 31, 1:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
faidon renamed this task from Update DNS/NTP for all non-network/server gear to Update DNS/NTP servers on the esams PDUs/SCS.Thu, Oct 31, 1:51 PM
RobH closed this task as Resolved.Thu, Oct 31, 2:26 PM

All esams pdu's already had that nameserver set. I've updated the NTP to the new IP addresses; same for scs-oe16-esams, had the right dns but wrong nameserver info, corrected.

I had already manually updated cr[23]-esams, mr1-esams, and asw2-esams, as appropriate for NTP (DNS should've been already-correct on those), I believe.

Change 547589 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Make NTP servers a variable and update esams IPs

https://gerrit.wikimedia.org/r/547589

Change 547589 merged by Ayounsi:
[operations/homer/public@master] Make NTP servers a variable and update esams IPs

https://gerrit.wikimedia.org/r/547589

Mentioned in SAL (#wikimedia-operations) [2019-10-31T20:07:04Z] <XioNoX> Homer push to all asw* - new NTP servers - T237011

Mentioned in SAL (#wikimedia-operations) [2019-10-31T20:12:53Z] <XioNoX> Homer push to all msw* - new NTP servers - T237011

ayounsi reopened this task as Open.Thu, Oct 31, 11:43 PM
ayounsi added a subscriber: ayounsi.

All network devices updated.

For NTP the more servers configured the better, but for devices that only accept one or two servers (like PDUs) I was wondering if it wouldn't be better to use an anycast IP as well. For example 10.3.0.2.
As they usually all use ntp.eqiad/codfw.wikimedia.org, POPs end up with NTP servers far away from them.
We also found last time that some PDUs don't seem to be good at resolving the NTP FQDN.
@BBlack thoughts?

RobH added a comment.EditedFri, Nov 1, 1:38 AM

So far, none of the SCS or PDU devices seem to like the FQDN for ntp servers, but instead like plain old ipv4 addresses.

faidon closed this task as Resolved.Fri, Nov 1, 2:51 PM

Anycasting NTP sounds a good idea in general, but a) should be kept in a separate task b) it doesn't sound like a priority IMHO at this time. Things work OK, and that sounds like a time investment that won't pay off right now.

Given the comments above, I'll resolve this - hope that makes sense!