Page MenuHomePhabricator
Create Task
Maniphest T244153

Import IP addresses, interfaces and DNS names into Netbox for Primary Interfaces
Closed, ResolvedPublic
Actions

Description

Since Netbox needs records for DNS names and IP addresses, and we may as well correctly link them with the devices and interfaces, we shall import all of this data in a repeatable manner from PuppetDB.

Requirements:

  • Import data about interfaces, IP addresses, MTU, netmask and DNS names from PuppetDB into Netbox
  • Must be repeatable and safe
  • Complete this import.

Details

SubjectRepoBranchLines +/-
swift: remove old unused service recordsoperations/dnsmaster+0 -3
urldownloader: convert A record to CNAMEoperations/dnsmaster+1 -1
urldownloader: convert A record to CNAMEoperations/dnsmaster+3 -3
dns: do not try to generate PTR for external IPsoperations/software/netbox-extrasmaster+8 -3
dns: correctly sort IPv6 PTR recordsoperations/software/netbox-extrasmaster+1 -1
dns: split public zones per DCoperations/software/netbox-extrasmaster+32 -13
dns: make logging less noisyoperations/software/netbox-extrasmaster+3 -3
sre.dns.decommission: improve the DNS automationoperations/cookbooksmaster+37 -21
wmcs: remove leftover records for old hostsoperations/dnsmaster+0 -5
misc: remove leftover records for old hostsoperations/dnsmaster+0 -15
databases: remove leftover records from old hostsoperations/dnsmaster+0 -11
k8s: remove leftover records for old hostsoperations/dnsmaster+0 -11
lvs: convert IPv6 PTR ORIGINs to /64operations/dnsmaster+46 -71
puppetdb: remove leftover records for old hostsoperations/dnsmaster+0 -4
ores: remove leftover records for old hostsoperations/dnsmaster+0 -8
wmf-auto-reimage: move Netbox update later onoperations/puppetproduction+2 -1
interface_automation: Fix the messages when setting IP as primaryoperations/software/netbox-extrasmaster+7 -4
wmf-auto-reimage: fix Netbox update (take 2)operations/puppetproduction+2 -2
wmf-auto-reimage: fix Netbox updateoperations/puppetproduction+3 -2
wmf-auto-reimage: update Netbox interfacesoperations/puppetproduction+27 -0
interface_automation: Blacklist all interfaces that start with 'lo'operations/software/netbox-extrasmaster+1 -1
sre.ganeti.makevm: fix IP allocationoperations/cookbooksmaster+7 -3
scripts: enable primary IPs optionsoperations/software/netbox-extrasmaster+47 -50
customscripts/interface_automation.py: Add Interface and IP Importeroperations/software/netbox-extrasmaster+560 -1
dns: generate records for all VMsoperations/software/netbox-extrasmaster+1 -2
Cleanup leftover record hhvm-apioperations/dnsmaster+0 -2
Cleanup leftover record cloudceph.svc.eqiad.wmnetoperations/dnsmaster+1 -1
Cleanup leftover record druid-public-overlordoperations/dnsmaster+0 -3
puppetdb microservice: add some filteringoperations/puppetproduction+17 -11
puppetdb uservice: Add individual host queries, expand for interface automationoperations/puppetproduction+40 -5
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• crusnov added a comment.Jul 28 2020, 6:27 PM
In T244153#6327459, @crusnov wrote:
In T244153#6321695, @Volans wrote:

The output and logs (starting on line 2823) of the last test run can be found in P11972. The data was imported into netbox-next.wikimedia.org after its data was reset using the last production DB dump.

After checking the above paste and spot-checking devices in netbox-next, here my comments/questions/suggestions/issues found:

  • Devices with Netbox status of Planned and Inventory are reported as they were missing facts in PuppetDB, let's skip them entirely to reduce noise
  • Devices with a tenant must be managed in a different way as they are not managed by our production Puppet hence not in PuppetDB.
    • Do not lookup devices with a tenant (not null) in PuppetDB
    • Manage Frack devices
    • Manage OIT host (just one for now) or skip if we decide to manage them manually or not manage them
    • Manage RIPE Atlas anchors or skip if we decide to manage them manually or not manage them
  • d-i-test is a special case VM as will never be in PuppetDB and is most of the time poweroff in Ganeti
  • install4001 has some issues, seems unrelated to this script, see T254157#6241107
  • people2001 is a VM marked as offline in Netbox, to be double checked
  • LVS hosts have all the cross-VLAN interfaces, for which we don't currently manage their DNS records, maybe we should, TBD. See for example https://netbox-next.wikimedia.org/dcim/devices/121/
  • Loopback interfaces need some more thoughts on what to do with them. As they are virtual and mostly managed by Puppet we might even decide to skip them entirely, although we have some DNS records related to them, in that case we might want to manage them but we'll need to manage their DNS records too. Notable cases:
    • Every host behind LVS has a lo:LVS with the VIP IP address to make LVS-DR work. This end up being set on multiple devices, creating duplicate IP addresses in Netbox (see https://netbox-next.wikimedia.org/ipam/ip-addresses/2775/ for example). I think we should probably skip those.
    • Some category of hosts have a long dynamic list of loopback interfaces that most likely has no sense to have in Netbox, but we should double check with their owners probably. In particular:
      • Kubernetes hosts with cali* interfaces
      • Ganeti hosts with tap* interfaces
      • Openstack hosts (cloudvirt*) with tap* interfaces

Alrighty, dumped and reloaded and did a fresh import.

This fixes VM status (only online are considered for import) and loopback devices no longer set MTU.

Please review!

Import log:
https://phabricator.wikimedia.org/P12091

• crusnov added a comment.Jul 29 2020, 9:50 PM

I have dumped and reloaded, the results are similar but now all VMs have their interfaces set to the virtual type since the API explodes if you don't.

• crusnov added a comment.Jul 29 2020, 10:53 PM

Oh also note recent update fixes devices which use previously reserved ip addresses now set the state to active and clear the comment if the comment is 'reserved for infra'

Volans added a comment.Jul 30 2020, 8:32 AM

@crusnov was a new dump used? From a quick run I didn't get any diff from the last run of yesterday.

From the diff for now I can report that additional IPs on the same interface are not imported, see for example the restbase hosts that have:

restbase2013-a  1H  IN A    10.192.16.82 ; cassandra instance
restbase2013-b  1H  IN A    10.192.16.83 ; cassandra instance
restbase2013-c  1H  IN A    10.192.16.84 ; cassandra instance

Although they are reported by facter:

restbase2013  0 ~$ sudo facter -p networking
{
  domain => "codfw.wmnet",
  fqdn => "restbase2013.codfw.wmnet",
  hostname => "restbase2013",
  interfaces => {
    eno1 => {
      bindings => [
        {
          address => "10.192.16.80",
          netmask => "255.255.252.0",
          network => "10.192.16.0"
        },
        {
          address => "10.192.16.82",
          netmask => "255.255.255.255",
          network => "10.192.16.82"
        },
        {
          address => "10.192.16.83",
          netmask => "255.255.255.255",
          network => "10.192.16.83"
        },
        {
          address => "10.192.16.84",
          netmask => "255.255.255.255",
          network => "10.192.16.84"
        }
      ],
      bindings6 => [
        {
          address => "2620:0:860:102:10:192:16:80",
          netmask => "ffff:ffff:ffff:ffff::",
          network => "2620:0:860:102::"
        },
        {
          address => "fe80::4ed9:8fff:fe35:6e78",
          netmask => "ffff:ffff:ffff:ffff::",
          network => "fe80::"
        }
      ],
      ip => "10.192.16.80",
      ip6 => "2620:0:860:102:10:192:16:80",
      mac => "4c:d9:8f:35:6e:78",
      mtu => 1500,
      netmask => "255.255.252.0",
      netmask6 => "ffff:ffff:ffff:ffff::",
      network => "10.192.16.0",
      network6 => "2620:0:860:102::"
    },
[...SNIP...]
}
• crusnov added a comment.Jul 30 2020, 3:43 PM
In T244153#6347830, @Volans wrote:

@crusnov was a new dump used? From a quick run I didn't get any diff from the last run of yesterday.

No still the old dump (I was just updating the ticket for our 2nd dump/reload during debugging).

• crusnov added a comment.Jul 30 2020, 3:46 PM

In terms of the extra IP addresses, the import only looks at the ip/ip6 fields and not the binding fields. It would be "relatively" easy to make it look at interface bindings instead but it would mean restructuring some things. A big problem is that the names aren't stored in puppetdb at all, and would have to be manually added to those IP addresses anyway, which makes me think that extra things like these should be handled in a manual fashion or at least separately.

• crusnov added a comment.Jul 31 2020, 9:56 PM

I have dumped and reloaded from a more recent dump (2020-07-31) and rerun import with preserving all IP addresses in the 'bindings' list instead of just primary IPv4 and IPv6.

• ayounsi added a comment.EditedAug 3 2020, 6:21 AM

restbase2013 LGTM.
Edit: actually 2620:0:860:102:10:192:16:80/64 should probably be set as primary as well.

Volans added a comment.Aug 11 2020, 2:05 PM

We've decided on a plan for service IPs (intended as any /32 IPv4 and /128 IPv6) assigned to interfaces:

  • import them in Netbox only once (no duplicates)
  • do not attach them to any interface
  • mark them as role VIP
  • set their DNS name

The generation script has been already modified to support addresses not attached to any device interface.
The import script needs to be refactored a bit to support this workflow.

Volans added a comment.Aug 13 2020, 9:26 AM

Last import is not really usable as a test bed, because:

Volans added a comment.Aug 18 2020, 4:10 PM

@crusnov this is the report of the last import diff, if you could have a look in the next days that I'm off and fix in either side (dns repo or netbox) those that are obvious and investigate the others by end of week that would be great. Let me know if anything is not clear or you need more info or the deadline is too close.

TYPOs?
Found origin 0.20.10
Missing line: 1   1H  IN PTR  vrrp-gw-103.esams.wikimedia.org. vs vrrp-gw-103.esams.wmnet in https://netbox.wikimedia.org/ipam/ip-addresses/2691/
IN DNS but not yet installed:
Found origin 0.64.10
Missing line: 10  1H IN PTR   pki1001.eqiad.wmnet.
Missing line: pki1001         1H  IN A        10.64.0.10
Found origin 5.64.10
Missing line: 37  1H IN PTR   relforge1003.eqiad.wmnet.
Missing line: relforge1003    1H  IN A        10.64.5.37
Missing line: 38  1H IN PTR   an-test-worker1001.eqiad.wmnet.
Missing line: an-test-worker1001 1H IN A           10.64.5.38
Found origin 21.64.10
Missing line: 126 1H IN PTR   relforge1004.eqiad.wmnet.
Missing line: relforge1004    1H  IN A        10.64.21.126
Found origin 36.64.10
Missing line: 111     1H IN PTR       an-test-worker1002.eqiad.wmnet.
Missing line: an-test-worker1002 1H IN A           10.64.36.111
Found origin 53.64.10
Missing line: 21      1H IN PTR       an-test-worker1003.eqiad.wmnet.
Missing line: an-test-worker1003 1H IN A           10.64.53.21
Found origin 0.192.10
Missing line: 195 1H IN PTR   kubestage2001.codfw.wmnet.
Missing line: 137 1H IN PTR   kubestage2002.codfw.wmnet.
Missing line: kubestage2001   1h  IN A    10.192.0.195
Missing line: kubestage2002   1h  IN A    10.192.16.137
Found origin 48.192.10
Missing line: 53  1H IN PTR   restbase2009.codfw.wmnet.
Missing line: 54  1H IN PTR   restbase2009-a.codfw.wmnet.
Missing line: 55  1H IN PTR   restbase2009-b.codfw.wmnet.
Missing line: 56  1H IN PTR   restbase2009-c.codfw.wmnet.
Missing line: restbase2009    1H  IN A    10.192.48.53
Missing line: restbase2009-a  1H  IN A    10.192.48.54 ; cassandra instance
Missing line: restbase2009-b  1H  IN A    10.192.48.55 ; cassandra instance
Missing line: restbase2009-c  1H  IN A    10.192.48.56 ; cassandra instance
Found origin 1.0.0.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 2.1.0.0.6.2.0.0.5.3.0.0.8.9.1.0 1H IN PTR   install4001.wikimedia.org. ; VM on ganeti01.svc.ulsfo.wmnet
Found origin 1.0.0.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 3.6.0.0.4.7.1.0.8.9.1.0.1.9.0.0 1H IN PTR   install3001.wikimedia.org. ; VM on ganeti01.svc.esams.wmnet
Found origin 1.0.0.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 3.1.0.0.6.6.1.0.2.0.1.0.3.0.1.0 1H IN PTR   install5001.wikimedia.org. ; VM on ganeti01.svc.eqsin.wmnet
LEFTOVER decommissioned:
Found origin 0.64.10
Missing line: 87  1H IN PTR   orespoolcounter1001.eqiad.wmnet. ; VM on ganeti01.svc.eqiad.wmnet
Missing line: orespoolcounter1001 1H IN A     10.64.0.87 ; VM on ganeti01.svc.eqiad.wmnet
Found origin 16.64.10
Missing line: 156 1H IN PTR   pc1001.eqiad.wmnet.
Missing line: 157 1H IN PTR   pc1002.eqiad.wmnet.
Missing line: 158 1H IN PTR   pc1003.eqiad.wmnet.
Missing line: pc1001          1H  IN A        10.64.16.156
Missing line: pc1002          1H  IN A        10.64.16.157
Missing line: pc1003          1H  IN A        10.64.16.158
Found origin 20.64.10
Missing line: 14  1H IN PTR   virt1009.eqiad.wmnet.
Missing line: 26  1H IN PTR   labvirt1010.eqiad.wmnet.
Missing line: 27  1H IN PTR   labvirt1011.eqiad.wmnet.
Missing line: labvirt1010     1H  IN A        10.64.20.26
Missing line: labvirt1011     1H  IN A        10.64.20.27
Found origin 32.64.10
Missing line: 13      1H IN PTR       orespoolcounter1002.eqiad.wmnet. ; VM on ganeti01.svc.eqiad.wmnet
Missing line: orespoolcounter1002 1H IN A     10.64.32.13 ; VM on ganeti01.svc.eqiad.wmnet
Missing line: 188     1H IN PTR       technetium.eqiad.wmnet. ; VM on ganeti01.svc.eqiad.wmnet
Missing line: technetium      1H  IN A        10.64.32.188 ; VM on ganeti01.svc.eqiad.wmnet
Found origin 0.192.10
Missing line: 13  1H IN PTR   db2010.codfw.wmnet.
Missing line: 16  1H IN PTR   db2013.codfw.wmnet.
Missing line: db2013          1H  IN A    10.192.0.16
Missing line: 56  1H IN PTR   orespoolcounter2001.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: orespoolcounter2001 1H IN A 10.192.0.56 ; VM on ganeti01.svc.codfw.wmnet
Missing line: 94  1H IN PTR   kubetcd2001.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: 95  1H IN PTR   kubetcd2002.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: kubetcd2001     1H  IN A    10.192.0.94 ; VM on ganeti01.svc.codfw.wmnet
Missing line: kubetcd2002     1H  IN A    10.192.0.95 ; VM on ganeti01.svc.codfw.wmnet
Missing line: 191 1H IN PTR   people2001.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Found origin 1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.9.1.0.0.0.0.0.2.9.1.0.0.1.0.0 1H IN PTR   people2001.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: people2001      1H  IN A    10.192.0.191 ; VM on ganeti01.svc.codfw.wmnet
Missing line: people2001      1H  IN AAAA 2620:0:860:101:10:192:0:191 ; VM on ganeti01.svc.codfw.wmnet
Missing line: 214     1H IN PTR       puppetdb1001.eqiad.wmnet. ; VM on ganeti01.svc.eqiad.wmnet
Missing line: puppetdb1001    1H  IN A        10.64.32.214 ; VM on ganeti01.svc.eqiad.wmnet
Found origin 16.192.10
Missing line: 20  1H IN PTR   db2032.codfw.wmnet.
Missing line: db2032          1H  IN A    10.192.16.20
Missing line: 30  1H IN PTR   kubetcd2003.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: kubetcd2003     1H  IN A    10.192.16.30 ; VM on ganeti01.svc.codfw.wmnet
Missing line: 76  1H IN PTR   orespoolcounter2002.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: orespoolcounter2002 1H IN A 10.192.16.76 ; VM on ganeti01.svc.codfw.wmnet
Missing line: 168 1H IN PTR   cygnus.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: cygnus          1H  IN A    10.192.16.168 ; VM on ganeti01.svc.codfw.wmnet
Missing line: 183 1H IN PTR   puppetdb2001.codfw.wmnet. ; VM on ganeti01.svc.codfw.wmnet
Missing line: puppetdb2001     1H IN A    10.192.16.183 ; VM on ganeti01.svc.codfw.wmnet
Found origin 2.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 7.4.1.0.4.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR   californium.wikimedia.org.
Found origin 3.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 0.9.0.0.4.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR   polonium.wikimedia.org.
Missing line: 3.9.0.0.4.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR   radon.wikimedia.org.
FAILED not in puppetdb:
Found origin 21.64.10
Missing line: 125 1H IN PTR   an-presto1004.eqiad.wmnet.
Found origin 5.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 5.2.1.0.1.2.0.0.4.6.0.0.0.1.0.0 1H IN PTR   an-presto1004.eqiad.wmnet.
Missing line: an-presto1004    1H IN A             10.64.21.125
Missing line: an-presto1004    1H IN AAAA          2620:0:861:105:10:64:21:125
Found origin 16.192.10
Missing line: 47  1H IN PTR   wtp2005.codfw.wmnet.
Missing line: wtp2005         1H  IN A    10.192.16.47
DNS vs host mismatch:
Host ganeti5001 has inet6 2001:df2:e500:101:d294:66ff:fe81:9090/64 while DNS has  1H  IN AAAA 2001:df2:e500:101:10:132:0:21
similar for ganeti5002 and ganeti5003
LVS name not in SVC:
Found origin eqiad.wmnet.
Missing line: ms-fe           1H  IN A        10.2.2.27 ; LVS address for ms-fe100*  # Why is this needed?
TO BE INVESTIGATED
1)
Found origin 16.192.10
Missing line: 142 1H IN PTR   pybal-testsvc.codfw.wmnet.  ; service IP -> I6b163fc264e3eadd3d46834dbd794ced03991d8a
Missing line: pybal-testsvc   1H  IN A    10.192.16.142 ; service IP

2)
Missing zonefile 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa for 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.0.6.8.0.0.0.0.0.0.2.6.2
Missing line: a.f.0.0 1H IN PTR  git-ssh.codfw.wikimedia.org.  # .codfw.wikimedia.org?

3)
Missing zonefile 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa for 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.4.2.0                         1H IN PTR   cloudelastic.wikimedia.org.
Missing line: 6.1.0.0                         1H IN PTR   git-ssh.eqiad.wikimedia.org.

4)
Missing line: 4.2.0.0.4.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR   ns-recursor1.openstack.eqiad1.wikimediacloud.org.
Found origin 2.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 3.4.1.0.4.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR   ns-recursor0.openstack.eqiad1.wikimediacloud.org.
Found origin 4.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 9.1.1.0.5.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR   nfs-maps.wikimedia.org.
NETWORK DEVICES (check with Arzhel):
Found origin 1.0.0.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-510.cr3-eqsin.wikimedia.org.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-510.cr2-eqsin.wikimedia.org.
Found origin 1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-520.cr3-eqsin.eqsin.wmnet.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-520.cr2-eqsin.eqsin.wmnet.
Found origin 1.0.2.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-530.cr3-eqsin.wikimedia.org.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-530.cr2-eqsin.wikimedia.org.
Found origin 7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.e.f 1H IN PTR   ae4-1020.cr2-eqiad.wikimedia.org.
Missing zonefile 1.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa for 1.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   gr-4-3-0-1.cr1-eqiad.wikimedia.org.
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   gr-0-0-0-1.cr2-eqord.wikimedia.org.
Processing zone 2.6.8.0.0.0.0.0.0.2.6.2 in zonefile 2.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
Found origin 1.0.0.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-100.esams.wikimedia.org.
Found origin 2.0.1.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-103.esams.wikimedia.org.
Found origin 1.0.2.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-102.wikimedia.org.
Processing zone 3.6.8.0.0.0.0.0.0.2.6.2 in zonefile 3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
Found origin 1.0.0.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1201.wikimedia.org.
Found origin 1.0.1.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1211.wikimedia.org.
Found origin 1.0.2.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1221.wikimedia.org.
• ayounsi added a comment.Aug 19 2020, 10:30 AM

Missing line: 1 1H IN PTR vrrp-gw-103.esams.wikimedia.org. vs vrrp-gw-103.esams.wmnet in https://netbox.wikimedia.org/ipam/ip-addresses/2691/
Fixed.

Found origin 1.0.0.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-510.cr3-eqsin.wikimedia.org.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-510.cr2-eqsin.wikimedia.org.
Found origin 1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-520.cr3-eqsin.eqsin.wmnet.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-520.cr2-eqsin.eqsin.wmnet.
Found origin 1.0.2.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-530.cr3-eqsin.wikimedia.org.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-530.cr2-eqsin.wikimedia.org.
Found origin 7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.e.f 1H IN PTR   ae4-1020.cr2-eqiad.wikimedia.org.

Invalid (don't match a real IP).

Missing zonefile 1.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa for 1.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   gr-4-3-0-1.cr1-eqiad.wikimedia.org.
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   gr-0-0-0-1.cr2-eqord.wikimedia.org.

Decommissioned some time ago.

Processing zone 2.6.8.0.0.0.0.0.0.2.6.2 in zonefile 2.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
Found origin 1.0.0.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-100.esams.wikimedia.org.
Found origin 2.0.1.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-103.esams.wikimedia.org.
Found origin 1.0.2.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-102.wikimedia.org.
Processing zone 3.6.8.0.0.0.0.0.0.2.6.2 in zonefile 3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
Found origin 1.0.0.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1201.wikimedia.org.
Found origin 1.0.1.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1211.wikimedia.org.
Found origin 1.0.2.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1221.wikimedia.org.

Now correct in Netbox.

• crusnov added a comment.Aug 20 2020, 3:57 PM
In T244153#6395972, @ayounsi wrote:

Missing line: 1 1H IN PTR vrrp-gw-103.esams.wikimedia.org. vs vrrp-gw-103.esams.wmnet in https://netbox.wikimedia.org/ipam/ip-addresses/2691/
Fixed.

Found origin 1.0.0.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-510.cr3-eqsin.wikimedia.org.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-510.cr2-eqsin.wikimedia.org.
Found origin 1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-520.cr3-eqsin.eqsin.wmnet.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-520.cr2-eqsin.eqsin.wmnet.
Found origin 1.0.2.0.0.0.5.e.2.f.d.0.1.0.0.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-530.cr3-eqsin.wikimedia.org.
Missing line: 3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   ae1-530.cr2-eqsin.wikimedia.org.
Found origin 7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.e.f 1H IN PTR   ae4-1020.cr2-eqiad.wikimedia.org.

Invalid (don't match a real IP).

Missing zonefile 1.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa for 1.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   gr-4-3-0-1.cr1-eqiad.wikimedia.org.
Missing line: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   gr-0-0-0-1.cr2-eqord.wikimedia.org.

Decommissioned some time ago.

Processing zone 2.6.8.0.0.0.0.0.0.2.6.2 in zonefile 2.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
Found origin 1.0.0.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-100.esams.wikimedia.org.
Found origin 2.0.1.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-103.esams.wikimedia.org.
Found origin 1.0.2.0.2.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-102.wikimedia.org.
Processing zone 3.6.8.0.0.0.0.0.0.2.6.2 in zonefile 3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa
Found origin 1.0.0.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1201.wikimedia.org.
Found origin 1.0.1.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1211.wikimedia.org.
Found origin 1.0.2.0.3.6.8.0.0.0.0.0.0.2.6.2
Missing line: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR   vrrp-gw-1221.wikimedia.org.

Now correct in Netbox.

To be clear, all of the above noted are fixed?

• crusnov renamed this task from Import PuppetDB interfaces into Netbox to Import IP addresses, interfaces and DNS names into Netbox for Primary Interfaces.Aug 21 2020, 12:00 AM
• ayounsi added a comment.Aug 21 2020, 6:20 AM

To be clear, all of the above noted are fixed?

Correct.

• crusnov added a comment.Aug 21 2020, 9:28 PM
FAILED not in puppetdb:
Found origin 21.64.10
Missing line: 125 1H IN PTR   an-presto1004.eqiad.wmnet.
Found origin 5.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 5.2.1.0.1.2.0.0.4.6.0.0.0.1.0.0 1H IN PTR   an-presto1004.eqiad.wmnet.
Missing line: an-presto1004    1H IN A             10.64.21.125
Missing line: an-presto1004    1H IN AAAA          2620:0:861:105:10:64:21:125
Found origin 16.192.10
Missing line: 47  1H IN PTR   wtp2005.codfw.wmnet.
Missing line: wtp2005         1H  IN A    10.192.16.47

We have decided to add these manually in massimport.

TO BE INVESTIGATED
1)
Found origin 16.192.10
Missing line: 142 1H IN PTR   pybal-testsvc.codfw.wmnet.  ; service IP -> I6b163fc264e3eadd3d46834dbd794ced03991d8a
Missing line: pybal-testsvc   1H  IN A    10.192.16.142 ; service IP

This looks like a going thing. There are numerous refs in puppet for it. I think this should be treated like any svc address (although at present there doesn't seem to be any device with this address assigned to an interface so we will have to manually add it).

Missing zonefile 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa for 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.0.6.8.0.0.0.0.0.0.2.6.2
Missing line: a.f.0.0 1H IN PTR git-ssh.codfw.wikimedia.org. # .codfw.wikimedia.org?

Missing zonefile 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa for 3.0.0.0.0.0.0.0.0.0.0.0.a.1.d.e.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 6.1.0.0 1H IN PTR git-ssh.eqiad.wikimedia.org.

These two are svc addresses. They are assigned to lo:LVS so unblacklisting that on import with the new svc address handling should fix these.

Missing line: 4.2.0.0.4.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR ns-recursor1.openstack.eqiad1.wikimediacloud.org.
Found origin 2.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 3.4.1.0.4.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR ns-recursor0.openstack.eqiad1.wikimediacloud.org.

These are also on lo:LVS so as above.

Found origin 4.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 9.1.1.0.5.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR nfs-maps.wikimedia.org.

This appears to be an extra name on an interface-bound ip address. I think there might be a bug in the import or something but I don't quite understand what the above means in terms of what's wrong with the data.

• crusnov added a subscriber: herron.Aug 21 2020, 9:33 PM
DNS vs host mismatch:
Host ganeti5001 has inet6 2001:df2:e500:101:d294:66ff:fe81:9090/64 while DNS has  1H  IN AAAA 2001:df2:e500:101:10:132:0:21
similar for ganeti5002 and ganeti5003

@herron I was told you might know something about this.

Volans added a comment.Aug 25 2020, 8:54 AM
In T244153#6403406, @crusnov wrote:

Found origin 4.0.0.0.1.6.8.0.0.0.0.0.0.2.6.2
Missing line: 9.1.1.0.5.5.1.0.0.8.0.0.8.0.2.0 1H IN PTR nfs-maps.wikimedia.org.

This appears to be an extra name on an interface-bound ip address. I think there might be a bug in the import or something but I don't quite understand what the above means in terms of what's wrong with the data.

In the DNS repo we have both v4 and v6 address for this, while in the imported data we have only the v6 one.

• crusnov added a comment.Sep 1 2020, 8:43 PM

Some notes about the current progress:

  • Imports seem 99% good.
  • Massimport miniscript deals with the manual cases
  • there is a hack in the interface automation script to deal with some mis-netmasking on some extra ip addresses on aqs*,restbase*,sessionstore*

Some things to keep in mind:

  • We need to document when to run the interface automation script manually (e.g. on hardware changes/updates)
  • We should probably have a way to clean up interfaces not present in puppet for resyncing changes
  • We should make a report to detect interfaces getting out of sync
Volans added a comment.Sep 2 2020, 9:34 AM

Records with TTL different from the standard 1H in the wikimedia.org zone:

gerrit        600 IN A        208.80.154.137
gerrit        600 IN AAAA     2620:0:861:2:208:80:154:137
gerrit-replica    600 IN A        208.80.153.107
gerrit-replica    600 IN AAAA     2620:0:860:4:208:80:153:107
lists           5M  IN A    208.80.154.21
lists           5M  IN AAAA 2620:0:861:1:208:80:154:21
ns0         1D  IN A    208.80.154.238
ns1         1D  IN A    208.80.153.231
ns2         1D  IN A    91.198.174.239

I guess that we might prefer keeping the ns ones hardcoded manually and we could blacklist them in the generation phase marking them somehow in Netbox (and we need to think about potential future expansions too).
As for the others, my understanding is that they are required for failovers and can't use CNAMEs. One option could be to add a custom field in netbox's IP addresses for the TTL, the other is to have a tag or something in the description to mark them as "SHORT_TTL" that will apply a 5M TTL. Of course we could do the same for a LONG_TTL that will use 1D if we want to include the NS records too.

gerritbot added a comment.Sep 2 2020, 9:56 AM

Change 623764 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Cleanup leftover record druid-public-overlord

https://gerrit.wikimedia.org/r/623764

gerritbot added a comment.Sep 2 2020, 10:01 AM

Change 623765 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Cleanup leftover record hhvm-api

https://gerrit.wikimedia.org/r/623765

gerritbot added a comment.Sep 2 2020, 1:21 PM

Change 623764 abandoned by Volans:
[operations/dns@master] Cleanup leftover record druid-public-overlord

Reason:
Missed https://gerrit.wikimedia.org/r/c/operations/dns/ /622563 already existed

https://gerrit.wikimedia.org/r/623764

gerritbot added a comment.Sep 2 2020, 6:04 PM

Change 623843 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Cleanup leftover record cloudceph.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/623843

gerritbot added a comment.Sep 2 2020, 6:13 PM

Change 623843 merged by Volans:
[operations/dns@master] Cleanup leftover record cloudceph.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/623843

gerritbot added a comment.Sep 3 2020, 12:40 PM

Change 623765 merged by Volans:
[operations/dns@master] Cleanup leftover record hhvm-api

https://gerrit.wikimedia.org/r/623765

gerritbot added a comment.Sep 3 2020, 5:26 PM

Change 624154 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: generate records for all VMs

https://gerrit.wikimedia.org/r/624154

gerritbot added a comment.Sep 10 2020, 8:55 AM

Change 624154 merged by Volans:
[operations/software/netbox-extras@master] dns: generate records for all VMs

https://gerrit.wikimedia.org/r/624154

gerritbot added a comment.Sep 11 2020, 4:01 PM

Change 626712 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] scripts: enable primary IPs options

https://gerrit.wikimedia.org/r/626712

gerritbot added a comment.Sep 11 2020, 9:03 PM

Change 588036 merged by Volans:
[operations/software/netbox-extras@master] customscripts/interface_automation.py: Add Interface and IP Importer

https://gerrit.wikimedia.org/r/588036

Volans mentioned this in rOSNE4fad8a901300: customscripts/interface_automation.py: Add Interface and IP Importer.Sep 11 2020, 9:04 PM
• crusnov added a subscriber: Dzahn.Sep 11 2020, 11:40 PM

Update on offline VMs
https://netbox.wikimedia.org/virtualization/virtual-machines/?q=&status=offline&mac_address=

  • d-i-test we already deal with
  • fermium is actually in the 'decommissioning' state, and is already removed from dns
  • install[345]001 are in planned state, and are in DNS. I have added primary interfaces (PRIMARY) and ip addresses including DNS names
  • people2001 is semi-broken, as a precaution in case it is offline during import i've added a PRIMARY interface as above.

thanks @Dzahn 😃

Stashbot added a comment.Sep 14 2020, 11:00 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-14T11:00:23Z] <volans> Mass importing IPs from PuppetDB into Netbox T244153

gerritbot added a comment.Sep 14 2020, 11:15 AM

Change 626712 merged by Volans:
[operations/software/netbox-extras@master] scripts: enable primary IPs options

https://gerrit.wikimedia.org/r/626712

Volans added a comment.Sep 14 2020, 11:22 AM
  • Backup pre import is /srv/postgres-backup/volans-pre-primary-import.psql-all-dbs-20200914.sql.gz.bak
  • Mass import run, stdout available at P12577 (and in /root/netbox-massimport-20200914.out on netbox1001), import.log file available on netbox1001 at /root/netbox-massimport-20200914.log.

Running the post-import steps

Volans updated the task description. (Show Details)Sep 14 2020, 11:23 AM
Volans mentioned this in rOSNE92be7ec25f3f: scripts: enable primary IPs options.Sep 14 2020, 11:28 AM
gerritbot added a comment.Sep 14 2020, 12:10 PM

Change 627263 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.ganeti.makevm: fix IP allocation

https://gerrit.wikimedia.org/r/627263

gerritbot added a comment.Sep 14 2020, 12:14 PM

Change 627263 merged by jenkins-bot:
[operations/cookbooks@master] sre.ganeti.makevm: fix IP allocation

https://gerrit.wikimedia.org/r/627263

gerritbot added a comment.Sep 14 2020, 4:59 PM

Change 627334 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/netbox-extras@master] interface_automation: Blacklist all interfaces that start with 'lo'

https://gerrit.wikimedia.org/r/627334

gerritbot added a comment.Sep 14 2020, 5:07 PM

Change 627334 merged by CRusnov:
[operations/software/netbox-extras@master] interface_automation: Blacklist all interfaces that start with 'lo'

https://gerrit.wikimedia.org/r/627334

gerritbot added a comment.Sep 14 2020, 5:51 PM

Change 627336 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/netbox-extras@master] interface_automation: Fix the messages when setting IP as primary

https://gerrit.wikimedia.org/r/627336

gerritbot added a comment.Sep 14 2020, 6:04 PM

Change 627337 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] wmf-auto-reimage: update Netbox interfaces

https://gerrit.wikimedia.org/r/627337

gerritbot added a comment.Sep 14 2020, 6:30 PM

Change 627337 merged by Volans:
[operations/puppet@production] wmf-auto-reimage: update Netbox interfaces

https://gerrit.wikimedia.org/r/627337

ops-monitoring-bot added a comment.Sep 14 2020, 6:35 PM

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009141834_volans_6008_sretest1001_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 14 2020, 7:05 PM

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Sep 14 2020, 7:38 PM

Change 627352 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] wmf-auto-reimage: fix Netbox update

https://gerrit.wikimedia.org/r/627352

gerritbot added a comment.Sep 14 2020, 8:21 PM

Change 627352 merged by Volans:
[operations/puppet@production] wmf-auto-reimage: fix Netbox update

https://gerrit.wikimedia.org/r/627352

ops-monitoring-bot added a comment.Sep 14 2020, 8:24 PM

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009142024_volans_29184_sretest1001_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 14 2020, 8:39 PM

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

Of which those FAILED:

['sretest1001.eqiad.wmnet']
gerritbot added a comment.Sep 14 2020, 8:44 PM

Change 627367 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] wmf-auto-reimage: fix Netbox update (take 2)

https://gerrit.wikimedia.org/r/627367

gerritbot added a comment.Sep 14 2020, 8:48 PM

Change 627367 merged by Volans:
[operations/puppet@production] wmf-auto-reimage: fix Netbox update (take 2)

https://gerrit.wikimedia.org/r/627367

ops-monitoring-bot added a comment.Sep 14 2020, 8:51 PM

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009142051_volans_3184_sretest1001_eqiad_wmnet.log.

gerritbot added a comment.Sep 14 2020, 9:10 PM

Change 627336 merged by CRusnov:
[operations/software/netbox-extras@master] interface_automation: Fix the messages when setting IP as primary

https://gerrit.wikimedia.org/r/627336

ops-monitoring-bot added a comment.Sep 14 2020, 9:13 PM

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

Of which those FAILED:

['sretest1001.eqiad.wmnet']
• crusnov mentioned this in rOSNE527e7b870e91: interface_automation: Fix the messages when setting IP as primary.Sep 14 2020, 9:16 PM
gerritbot added a comment.Sep 14 2020, 9:17 PM

Change 627374 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] wmf-auto-reimage: move Netbox update later on

https://gerrit.wikimedia.org/r/627374

gerritbot added a comment.Sep 14 2020, 9:18 PM

Change 627374 merged by Volans:
[operations/puppet@production] wmf-auto-reimage: move Netbox update later on

https://gerrit.wikimedia.org/r/627374

ops-monitoring-bot added a comment.Sep 14 2020, 9:20 PM

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009142120_volans_11195_sretest1001_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 14 2020, 9:48 PM

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Sep 15 2020, 2:57 PM

Change 627518 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] databases: remove leftover records from old hosts

https://gerrit.wikimedia.org/r/627518

gerritbot added a comment.Sep 15 2020, 2:57 PM

Change 627519 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] wmcs: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627519

gerritbot added a comment.Sep 15 2020, 2:57 PM

Change 627520 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] puppetdb: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627520

gerritbot added a comment.Sep 15 2020, 2:57 PM

Change 627521 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] ores: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627521

gerritbot added a comment.Sep 15 2020, 2:57 PM

Change 627522 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] k8s: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627522

gerritbot added a comment.Sep 15 2020, 2:58 PM

Change 627523 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] misc: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627523

gerritbot added a comment.Sep 15 2020, 2:58 PM

Change 627524 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] lvs: convert IPv6 PTR ORIGINs to /64

https://gerrit.wikimedia.org/r/627524

gerritbot added a comment.Sep 15 2020, 3:32 PM

Change 627521 merged by Volans:
[operations/dns@master] ores: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627521

gerritbot added a comment.Sep 15 2020, 3:34 PM

Change 627520 merged by Volans:
[operations/dns@master] puppetdb: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627520

gerritbot added a comment.Sep 15 2020, 3:43 PM

Change 627524 merged by Volans:
[operations/dns@master] lvs: convert IPv6 PTR ORIGINs to /64

https://gerrit.wikimedia.org/r/627524

gerritbot added a comment.Sep 15 2020, 4:10 PM

Change 627522 merged by Volans:
[operations/dns@master] k8s: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627522

gerritbot added a comment.Sep 16 2020, 6:12 AM

Change 627518 merged by Volans:
[operations/dns@master] databases: remove leftover records from old hosts

https://gerrit.wikimedia.org/r/627518

gerritbot added a comment.Sep 16 2020, 6:13 AM

Change 627523 merged by Volans:
[operations/dns@master] misc: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627523

gerritbot added a comment.Sep 16 2020, 10:43 AM

Change 627519 merged by Volans:
[operations/dns@master] wmcs: remove leftover records for old hosts

https://gerrit.wikimedia.org/r/627519

gerritbot added a comment.Sep 16 2020, 1:26 PM

Change 627834 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: improve the DNS automation

https://gerrit.wikimedia.org/r/627834

gerritbot added a comment.Sep 16 2020, 6:49 PM

Change 627898 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: make logging less noisy

https://gerrit.wikimedia.org/r/627898

gerritbot added a comment.Sep 16 2020, 6:49 PM

Change 627899 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: split public zones per DC

https://gerrit.wikimedia.org/r/627899

gerritbot added a comment.Sep 16 2020, 7:18 PM

Change 627834 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.decommission: improve the DNS automation

https://gerrit.wikimedia.org/r/627834

gerritbot added a comment.Sep 16 2020, 8:38 PM

Change 627909 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: correctly sort IPv6 PTR records

https://gerrit.wikimedia.org/r/627909

gerritbot added a comment.Sep 16 2020, 8:38 PM

Change 627898 merged by Volans:
[operations/software/netbox-extras@master] dns: make logging less noisy

https://gerrit.wikimedia.org/r/627898

gerritbot added a comment.Sep 17 2020, 10:18 AM

Change 628061 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: do not try to generate PTR for external IPs

https://gerrit.wikimedia.org/r/628061

gerritbot added a comment.Sep 17 2020, 1:37 PM

Change 628086 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] swift: remove old unused service records

https://gerrit.wikimedia.org/r/628086

gerritbot added a comment.Sep 17 2020, 2:40 PM

Change 628102 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] urldownloader: convert A record to CNAME

https://gerrit.wikimedia.org/r/628102

gerritbot added a comment.Sep 17 2020, 3:12 PM

Change 627899 merged by Volans:
[operations/software/netbox-extras@master] dns: split public zones per DC

https://gerrit.wikimedia.org/r/627899

gerritbot added a comment.Sep 17 2020, 3:13 PM

Change 627909 merged by Volans:
[operations/software/netbox-extras@master] dns: correctly sort IPv6 PTR records

https://gerrit.wikimedia.org/r/627909

gerritbot added a comment.Sep 17 2020, 3:16 PM

Change 628061 merged by Volans:
[operations/software/netbox-extras@master] dns: do not try to generate PTR for external IPs

https://gerrit.wikimedia.org/r/628061

gerritbot added a comment.Sep 21 2020, 9:48 AM

Change 628102 merged by Muehlenhoff:
[operations/dns@master] urldownloader: convert A record to CNAME

https://gerrit.wikimedia.org/r/628102

gerritbot added a comment.Sep 21 2020, 9:56 AM

Change 628763 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] urldownloader: convert A record to CNAME

https://gerrit.wikimedia.org/r/628763

gerritbot added a comment.Sep 22 2020, 7:13 AM

Change 628763 merged by Muehlenhoff:
[operations/dns@master] urldownloader: convert A record to CNAME

https://gerrit.wikimedia.org/r/628763

• crusnov closed this task as Resolved.Sep 25 2020, 5:03 PM

I believe this is complete, any further patches should point at the parent task T258729

gerritbot added a comment.Oct 6 2020, 8:01 AM

Change 628086 merged by Volans:
[operations/dns@master] swift: remove old unused service records

https://gerrit.wikimedia.org/r/628086

• ayounsi mentioned this in T265904: Remove SLAAC IPs from Ganeti hosts.Oct 19 2020, 1:36 PM
Volans mentioned this in rCCKB44a1a7745d09: sre.ganeti.makevm: fix IP allocation.Dec 14 2022, 3:26 PM
Volans mentioned this in rCCKB3ff02ea6493a: sre.dns.decommission: improve the DNS automation.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits