I've reverted the above patch as it was reporting most servers as false positive for the new name coherence report. The regex was wrong and not able to match our current hostnames.
Also is not totally clear to me the goal of this check, as it was opened for the asset tag names specifically but the check was expanded to all hosts.
For asset tag hostnames for example we should check that the hostname matches the asset tag of the same device, and if those are already lowercase, that would be already enough.

The debmonitor test didn't test much as the debmonitor client sends the puppet client cert (not the CA) and it's the server that validates it with the CA.

@jbond just to be on the safe side and to verify the theory, if possible make a quick test that the new cert in the CR is able to verify exiting puppet node certs and cergen certs.

A simplified version could be to use a cookbook to couple stuff:

In T233183#5665281, @BBlack wrote:

Seems sane! The only thing I'm a little iffy about iis from the "SHA1 written to etcd" onwards. I'm not sure it's a bad approach, but I'm not sure I've thought through all the implications, either. I think the key things to think about in that part of the flow that might be missing are, is emergency updates to the netbox-defined data. e.g. if all the things are borked and we need to manually edit a DNS entry in the netbox-derived zonefile fragments.... is there a way we can do that from the authdns servers with authdns-update? (e.g. a local commit and an override of the SHA1 argument?).

In T224946#5659653, @jijiki wrote:

I am not sure this is related, but we get many alerts of

• PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL
• PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed

If those are actionable from dc-ops or if we are getting more false positives than we should, we must fix it.

As we discussed a while ago about this, the easiest solution is to pick another port for the public TLS server on the debmonitor servers as the 443 is already taken for the internal clients to report the package list to it and it's used to perform authz/n with the client certificate.

@Krenair the default ones HTTPSConnectionPool(host='localhost', port=443)
@Dzahn that would be T179816, but it seems that there haven't been much interest in that backend.

I don't recall the details, too much time has passed, but indeed, it seem it's still supported by the puppet compiler code, so I'll resolve it.

I'd rather not do (3), seems a step back (not respecting awake hours and such).

I've some concerns to proceed with this. In our experience the BMCs are not that stable and an excessive interaction with them seems to aggravate the situation, statistically causing more BMCs to become unresponsive and requiring a reset.
For this reason we've kept to a minimum our checks of BMCs and I'd rather not add something that query the BMC so often.

No it's ok to keep it open and look at it at some point, no priority though.

We've migrated to the new puppetdb hosts with the newer version. The queue size is under control for now. Resolving.

It's great to see some movement in this space! I've tried it and indeed is much better.

@BBlack the current proposal is:

@CDanis the problem is that all of those identify clients, while for the CA validation we're mostly interested in the server side. So while that surely would help, it's a 1:1 mapping. Also there might be places that have hardcoded the path to the CA cert for validation, either in the puppet repo or, potentially, in other repos too (as a default for example, dunno).
I don't know if this CA is also used in the k8s world for example.

I've not had a chance to work on this in a while, I hope to get back to it soon. Leaving it open in the meanwhile. Most of the reimage single host script has been migrated, but some bits here and there were missing.

All the wmf-* scripts but the reimage ones were migrated to cookbooks. Most of the modules and functionalities in the library have been added to spicerack. I've not had a chance to work on this in a while, I hope to get back to it soon. Leaving it open in the meanwhile.

Most of the modules and functionalities have been added. I've not had a chance to work on this in a while, I hope to get back to it soon. Leaving it open in the meanwhile as some minor bits were still missing to include all the current functionalities of the reimage library.

Resolving as the known hosts backend in Cumin allows to use the already existing host list file present in the cumin hosts as part of the known hosts file.
In case we'd want to protect us from a combined failure of PuppetDB and the disappearance of the known hosts file we could resume the above patch and adapt it (as it's quite old and surely cannot be merged as is).

@crusnov what release/deployment is this task pending for?

One thing to take into account: we're using certificates signed by the Puppet CA in many places:

• the puppet client certificate exposed via puppet code, see base::expose_puppet_certs
• certificates in the private puppet repo generated via the utils/create_ecdsa_cert script and cergen

While you wait for @ayounsi I can maybe fill some gap. Homer is already a thing and Arzhel is using and testing it, but it doesn't have yet proper documentation for a wider usage (it will soon though). In the meanwhile, if you do manual changes to network devices is good in any case to have a patch for Homer's templates (when applicable) to keep things in sync.
So thanks for the patch, it's great to have it!

The only problem is how to keep adding those permissions every time a new Netbox release introduces some new model...

@crusnov should this be resolved?

As I've not yet fully understood the use case of those files given that AFAIK most of them cannot be re-imported as is into Netbox it's hard for me to give a feedback on the frequency of the backups and their retention.
If I have to ballpark it while keeping it simple then the standard hourly for a week, daily for the rest of the retention period might be a good compromise.

As chatted with Bryan earlier today yes it was caused by the 2.6 upgrade that included the above linked view permission.

@crusnov based on the recent change in the settings for the proxy wtr url scheme, do you think this issue no longer exists?

In Netbox we can already filter devices by purchase date, support expiry date and procurement ticket. That should be enough to pinpoint the batch as far as I can tell.
In this specific case for example picking the procurement ticket and the purchase date from Netbox for db1075 you could:

• search for the ticket: https://netbox.wikimedia.org/dcim/devices/?cf_ticket=T118174
• search for the date (need to be formatted compared to how is showed in the UI): https://netbox.wikimedia.org/dcim/devices/?cf_purchase_date=2016-03-02

Basic integration with Netbox has been developed and is now merged, pending the next release. Some improvements are already WIP and should be ready for CR later today.

Confirmed it's a puppetdb slowness:

2019-10-30 12:22:57,863 [DEBUG puppetdb.py:256 in _execute] Queried puppetdb for '["or", ["=", "certname", "cp5008.eqsin.wmnet"]]', got '0' results.

vs

2019-10-30 12:23:02,941 INFO  [p.p.command] [206393-1572438182779] [153ms] 'replace facts' command processed for cp5008.eqsin.wmnet

In this case the query to puppetdb returned no matching host. After a first look I think that it might be related to the queue size in Puppetdb that apparently has grown quite a lot in the last month, see:
https://grafana.wikimedia.org/d/000000477/puppetdb?panelId=19&fullscreen&orgId=1&from=1568392819818&to=1572273876961

This is all done, resolving. Feel free to re-open if any issue is found.

Closing as the host has been decommissioned as part of T236454

I can confirm this as it happened to me today. I'm seeing the fund raising banner on enwiki with "Hi, reader in France," while I'm in Italy. I've been in France recently but I left it ~20 days ago. I've two GeoIP cookies, both set with Session as expiration, one for .wikimedia.org and the other for .wikipedia.org.
We could force a refresh of the cookie if the IP doesn't match anymore the one in the cookie, if it's not a too heavy operation given that clients will surely change IP much more frequently than country.

@MoritzMuehlenhoff it failed the power off, as reported by the script, see https://phabricator.wikimedia.org/T208585#5599005

@jbond I had already opened T236345 for this. I guess that can probably be merged into this at this point.

There are also local modifications in the private repo fwiw.

@jcrespo the decommissioning cookbook supports VMs and can be use with them. The output will tell if there is any manual step to perform because not yet supported.
As for the wmf-auto-reimage scripts, those are not yet fully migrated to cookbooks and don't support VMs, because:

• VMs first installation is a bit different and basically self-done, so given the low number of manual steps was not a priority to automate
• VMs usually don't get reimaged, a new one is created and the old one decommissioned. At least those were the assumptions when the script were developed. If that has changed let me know.

In T234452#5593612, @crusnov wrote:

This should be fixed now.

@kaldari thanks for the offer, but I think this deserves some preparatory work that I cannot commit to at the moment, I'm sorry.

In T187709#5552423, @TheAnarcat wrote:

Why double? Curly braces don't need escape in the shell.

No? Here {foo,bar} means foo bar, and is equivalent to * if the content of the current directory is foo bar.

I'm marking this as resolved as the cookbook has been used many times at this point and both Phabricator templated and wikitech documentation have been updated accordingly. I'll send an email to ops as FYI to spread a bit more the word today.

In T187709#5544746, @TheAnarcat wrote:

On a more detailed level: there will be at least one conflict in syntax I can think of right now, { and } can appear in both Cumin and Prometheus. I'm assuming the latter will need escaping (?)

Ugh. Yeah. And double-escaping too, because we need to escape the shell first. That's rather inconvenient.

@CDanis my 2 cents are with Manuel to use es[123] on the dbctl side and have the mapping es->cluster in mediawiki-config code as it is right now (with comments in the db-$dc.php files).
One thing that I don't remember if it was mentioned is that the es1 cluster is read-only and doesn't have any replication setup between them. Some additional code/knob is probably needed to allow this setup IIRC.

Thanks @BBlack for the very detailed and precise summary.

Monving discussion from https://gerrit.wikimedia.org/r/c/operations/software/netbox-deploy/+/539013 here (+ brandon)

Updated Lifecycle page accordingly: https://wikitech.wikimedia.org/w/index.php?title=Server_Lifecycle&type=revision&diff=1839914&oldid=1837183

In T230449#5540161, @crusnov wrote:

This script has been released and appears to work correctly!

In T187709#5537925, @TheAnarcat wrote:

hi! i've been looking into this again and I think i might start looking at making a backend for Prometheus myself. It seems the first step would be to design a grammar for the Prometheus queries that wouldn't conflict with the existing selectors.

Yes that's kinda normal for many systems, I don't have a stats of them to be able to see if depends on generation or brand, etc... I could probably relax the check as the important parameter we want to check is just the force PXE boot.
Having a bit of time we could grab probably from the logs of the script on both hosts some stats on which hosts does it and find some pattern.

In T231066#5522792, @fgiunchedi wrote:

I tested the cookbook on ms-be1027 in T233289, the host is powered down and not coming back (faulty hw) and the cookbook stopped when trying to get to the host, whereas IMHO it should have continued (and/or prompt) with the remaining steps

In T233189#5512761, @faidon wrote:

This is approved.

Patch ready, pending approval.

From your description it seems that Netbox doesn't preserve the URL schema on pagination.

@wiki_willy the related patch above should already help a lot, but as you know I'm off those days and I cannot give it the necessary testing for merging it, but if anyone else want to volunteer to merge+test it is welcome ;) Otherwise I'll take care of it as soon as I'm back.
As a workaround clearly is possible to add more permissions to dcops, it's a trivial change in puppet that anyone can do, but it's not to me to decide, that should be considered an access request to be decided by the owners of the group (SRE, usually discussed in the weekly meeting).

@crusnov in case you missed my IRC ping yesterday, please re-install the two public ones before proceeding with the installation as they had no firewall (see above hotfix patch).

In T229686#5455125, @CDanis wrote:

Here's a mini-design proposal for the dbctl feature itself (@Volans and @Joe please review):

• Add a flavor enum to the section schema. The default will be regular, which will cause the section to be output in the config in sectionLoads, as usual.
• The flavor externalstore will cause the section to be output in externalLoads.
  • Any groups set in the instances of a section with flavor=externalstore will be ignored.

Thanks for the update.

@wiki_willy The data gathering failed because of host unreachable, but is this still a commissioned host? I cannot see the records in the DNS repo, just the management ones are there. Also see its decom task: T224475

It was a maintenance, tracked with GIN-1-2116159603, that was not present to the calendar because sent to noc@ and not the maint announce ML. We need to update their settings.

It seems that the session is misconfigured on their side:

Aug 27 09:10:38  cr2-eqord rpd[13953]: bgp_process_open:4072: NOTIFICATION sent to 2001:418:0:5000::a34 (External AS 2914): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: peer 2001:418:0:5000::a34 (External AS 2914) claims 65000, 2914 configured
Aug 27 09:10:42  cr2-eqord rpd[13953]: bgp_process_open:4072: NOTIFICATION sent to 128.241.2.53 (External AS 2914): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: peer 128.241.2.53 (External AS 2914) claims 65000, 2914 configured

In T229657#5431230, @Marostegui wrote:

@CDanis @Volans can you confirm this command will set wikitech (db1073 is its master) on read-only?:

# set read-only
dbctl --scope eqiad section wikitech ro "Maintenance on wikitech T229657 " && dbctl config commit -m "Set wikitech as read-only for maintenance T229657"

Thanks!

That's because we pass memory={memory}g to the gnt-instance add command. We could instead accept a float in the cookbook, convert it to MB and use m in the command.
I'm fine either way. CCing @elukey that used it a lot and @crusnov