This is fixed in master in 6d16ec2 (list of hosts to stdout, rest to stderr), but not yet deployed.

Resolving as it's a too old audit now, we could re-run it again if needed, but we've also alerts that check most of those scenarios.

We were discussing this offline with John and there are still various open questions, we plan to discuss them in the next I/F meeting next Wed.
I'll try to summarize them here:

I fear that we are going down another path like the dns generation in which the Netbox API don't really suits our needs in terms of performance and efficiency (multiple API calls per device). I'm wondering if we should convert John's patch into a Netbox script instead and take advantage of the speed and power of Django/Netbox internal APIs instead.

@jbond yeah I agree we can implement this assuming that all v6 addresses are mapped, with a sane fallback into the networking[ip6] one.
@crusnov could you implement that logic into the current import script? Basically pick the binding6 that endswith primary ipv4.replace('.', ':'), fallbacking into the current behaviour if no match.

@jbond thanks for looking into this, unfortunately the data used in the Netbox import comes from the networking fact because it needs all of them and parses that one, so not sure if the above patch would be useful in practice.

Do you think we could trick facter into reporting the non-SLAAC address as primary?

To partially fix Netbox we can just run the PuppetDB import script for the Ganeti hosts again, then it will just be a matter of deleting all those SLAAC addresses

We could get some inspiration from existing dashboards that were published at https://grafana.com/grafana/dashboards?search=django
Also relevant: https://github.com/korfuri/django-prometheus/issues/44

The IPs were allocated manually outside of Netbox and as such they could be allocated to a different host by Netbox in any upcoming provisioning causing conflicts.

In T265595#6547582, @jbond wrote:

im not sure what the hosting for wikitech-static is

Also the script doesn't seem to apply the same filters we do when importing from PuppetDB, I see VMs with lo interfaces. Those should not be created.

Resolving as this has been surpassed by later development, we're syncing the ifaces from puppetdb to attach the IPs to them now.

I'm being bold and resolve this task as there are only 5 files left and they are all in one personal home (./modules/admin/files/home/ori/.binned)

The topic has not been discussed any further since then nor I've heard any more request of a feature like this one for cumin.
Since then we've introduced Spicerack and Cookbooks that allow to add more logic and orchestration to the operations to perform, probably covering some of the use cases listed above, in conjunction with a wider usage of cumin aliases for canary hosts.
At this point this is more of a wish list item than anything else, I'm resolving it and abandoning the change.

The two remaining ones were done long time ago, but forgot to update this task. None of the frozen patterns exist in the Puppet repo as of now, resolving.

I think this could be converted into a method in the spicerack's netbox module that checks if there is a device or VM with the shortname and returns the DNS name of any of the primary IPs, if set and None (or raises) otherwise.

Once the script is fixed we should re-run the puppetdb import for any device that has been installed after the mass-import on Oct. 14th.

The agreed changes have been deployed, /30 and /31 prefixes have been consolidated into their parent prefix zone files.

Both scenarios are now covered by the cookbook and described in wikitech:

The previous approach was not working well because I found corner cases where we were consolidating into the same file different subnets for which one is managed by Netbox and one not and will probably never be (like frack vs frack mgmt).

@jbond FYI as a side note once this is deployed we can probably revisit a bit the firewall rules of the failoid hosts, that were designed already with reject-with icmp-port-unreachable to make the failure as quick as possible.

Yeah I was chatting on IRC with Arzhel and he was suggesting to force them to /24 anyway, even if we don't have the prefix in Netbox.

After a chat with @ayounsi I tried the approach that if the IP prefixlen is > 24 instead of picking the smallest prefix we try to get the first prefix above with status container and use that instead for deciding the zonefile.

@elukey if I try to ssh with the install console key I get a BusyBox... I guess that's the reason.
Basically the reimage script is polling the host waiting for the reboot after d-i and expects to ssh with the install console key. Apparently it got rebooted into busybox that has /proc/uptime and the reimage thought it was rebooted into the new OS and tried to run puppet inside the BusyBox.

Any news on this by any chance? On this random host I was checking today it taking up 1/3rd of syslog.

Nothing to do here, that is not a failure, the cookbook is just polling to get the craeted VM.
The [1/20, retrying in 3.00s] is the first call that failed, the second one succeeded and the cookbook continued its job as expected.

If instead we keep the current per-prefix approach we can easily add some additional checks in the sre.dns.netbox cookbook such as:

As spoken on IRC, if there is a clean cut to identify when we can use /24 in terms of Netbox data of the prefixes and such, and if it's ok to loose some of the flexibility we have right now to migrate gradually some things it's not a problem for me to adapt the generation script.

The PRs I had there got eventually merged, at that time puppetboard was barely maintained so there was no release, but I guess they are included in the latest releases.
Then there is https://github.com/voxpupuli/puppetboard/issues/461 but I have totally forgotten what I might have done in our setup for that, but I'm sure I can reconstruct it.

@MoritzMuehlenhoff ping me when this work will start as we might want to upgrade puppetboard too. At the time I had to apply a couple of internal patches because not yet merged upstream. We should re-check the status and evaluate.

from HW logs

@jbond I think we can just try the tmpfs first as you said and check the impact.

In T263578#6492417, @jbond wrote:

however i wonder if its worth mounting a tmpfs dir here? the risk is that we may loose a submission but as it likely receives a lot of IO is will likely give quite a nice boost

It looks it's marked as inactive on conftool:

$ confctl select 'name=mw1360.eqiad.wmnet' get
{"mw1360.eqiad.wmnet": {"weight": 30, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1360.eqiad.wmnet": {"weight": 30, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}

I've allocated them on Netbox, see:
https://netbox.wikimedia.org/search/?q=restbase1028
https://netbox.wikimedia.org/search/?q=restbase1029
https://netbox.wikimedia.org/search/?q=restbase1030

Ah so way before the IP allocation stuff was moved to Netbox. Let me add them to Netbox before merging.

When where those hosts provisioned? Basically I'm asking when was the IP allocation script on Netbox run, because it explicitely supports this use case.

Version 0.0.2 has been released to APT wikimedia for buster (python3-wmflib) and on PyPI (wmflib).

Totally agree that postgres issues should show on both eqiad and codfw so most likely a red herring, nevertheless the sizes seemed a bit too large for the data. But I agree to not go down that path if not needed.

I think so didn't get any report of issues.

I've merged https://gerrit.wikimedia.org/r/c/operations/dns/+/628995 and now authdns-update runs without errors and the DNS is unblocked.

The erros seems to be caused by the lack of the entry related to releases in the discovery-states file in gdnsd configuration. This in turn seems to be related to the fact that in hieradata/common/service.yaml the releases entry has state: monitoring_setup and in modules/profile/manifests/dns/auth/discovery.pp the entries are filtered by state: production:

wmflib::service::fetch().filter |$n, $svc| { 'discovery' in $svc  and $svc['state'] == 'production' }

I think that Debmonitor should only track packages for active-running-images, like it does for production hosts. Once a new image version is rolled out and the old one is not anymore running anywhere, it should just be deleted from Debmonitor.
Because of this missing GC we've what I've described in T261207#6433271, but that was meant as a temporary solution until a proper GC is implemented on the client side that is the only one knowing if an image version is running anywhere.

All done, resolving, thanks for the support!

Make sense.
I've added it manually to Netbox so that it's allocated and the DNS will be generated. I've marked it as VIP for now because it's not assigned to any host ( https://netbox.wikimedia.org/search/?q=nfs-maps.wikimedia.org ).

The device is still active in Netbox, shouldn't be marked as failed?

Patch sent, I have also a follow up question: on cloudstore1009 the IPv6 of nfs-maps.wikimedia.org is not assigned, but in the DNS repo is defined, as the mapped version of the v4 one. Is that intended/expected?

@Bstorm thanks a lot for the double check. I'll send a patch for the last 3 and make sure the other are retained.

• Backup pre import is /srv/postgres-backup/volans-pre-primary-import.psql-all-dbs-20200914.sql.gz.bak
• Mass import run, stdout available at P12577 (and in /root/netbox-massimport-20200914.out on netbox1001), import.log file available on netbox1001 at /root/netbox-massimport-20200914.log.