Cumin: allow querying PuppetDB over HTTP
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	TheAnarcat
	Mar 15 2019, 9:11 PM

Description

Cumin version	3.0.2
Python version	3.5.3
Operating System	Debian stretch 9.8

Issue:

In our environment, the PuppetDB server isn't accessible from the network, only from localhost. We have therefore not bothered creating a valid HTTPS certificate for the server, and usually query it over HTTP locally, or through an SSH tunnel.

When we try to use PuppetDB against the server, it crashes with the following error:

Caught SSLError exception: HTTPSConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /pdb/query/v4/nodes (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)'),))

Note thta this error is impossible to work around with the urllib3_disable_warnings documented in the example config file: even if we add SSLError to the list, the exception still trips up to the user.

Therefore, I have made the following simple patch to allow the configuration to override the scheme as well:

https://github.com/anarcat/cumin/commit/508551f4f259cc5251c2d7f3c175bbbc21e2fe69

I'd be happy to submit that to Gerrit if it's useful and I figure out how it works.

Thanks for your time.

Details

	Subject	Repo	Branch	Lines +/-
	PuppetDB backend: allow to override URL scheme in config	operations/software/cumin	master	+5 -1

Customize query in gerrit

Related Objects

Mentioned In: rCUMIN8e49a21f5758: PuppetDB backend: allow to override URL scheme in config

Event Timeline

TheAnarcat created this task.Mar 15 2019, 9:11 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2019, 9:11 PM

First of all thanks a lot for the report and the patch @TheAnarcat.
The choice of hardcoding HTTPS as a protocol in the PuppetDB backend was done assuming that most likely Puppet, PuppetDB and Cumin would be installed on different hosts, and given the private nature of the data stored in PuppetDB, requiring HTTPS seems a natural choice. Of course if all of them are on the same host and listen only on localhost it doesn't really matter.

The patch looks ok, the only thing I'd add is to add the new scheme config to the doc/examples/config.yaml file, with an inline comment to mention that is [optional] and what is its default.

The docs to send a patch to Cumin are on [1], but I guess you've already found them. To get a Gerrit account just follow [2]. Let us know if you have any trouble.

[1] https://wikitech.wikimedia.org/wiki/Cumin#How_to_contribute_to_Cumin
[2] https://www.mediawiki.org/wiki/Developer_account

hi @Volans ! long time no see ;)

I understand the choice of HTTPS - I'd even say it's great to use that by default - but I'm not sure how it resolves the problem of privacy in this peculiar case. After all, if PuppetDB is accessible without authentication over HTTPS, it doesn't really matter if the content is encrypted in transit if anyone can just get it by asking nicely. So I assume there's *some* sort of other authentication at play here, which I'd be curious to hear more about. Maybe it *would* be better to implement *that* instead of crippling Cumin, but the current documentation doesn't make that very clear.

I did register for a Developer account with Gerrit, but I've struggled to make it work because I first cloned from GitHub (because finding those Gerrit URLs is hard). The patch now lives in Gerrit, with your suggested change:

https://gerrit.wikimedia.org/r/#/c/operations/software/cumin/+/497309/

In the Wikimedia setup only certain types of hosts are able to talk to the Puppet DB service. I did try to get client TLS certificate checking added in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/437640/ but, well, I'm not really sure what was going on in prod that stopped that working.

right, so I think I understand your setup - you use HTTPS to talk to the PuppetDB from various hosts and probably use something like IP-level blocking to ensure data integrity. that's fine and it's what we do for other stuff like monitoring (Prometheus) here as well. i just don't see why i should absolutely get a valid certificate to talk to the PuppetDB if I have authentication and privacy figured out some other way.

it does seem like puppetdb is certified by the PuppetCA, so in theory we could work around this by injecting the PuppetCA certificate into the Cumin trust chain. But i'm not sure how that should be done. I'm also not comfortable adding that CA to the global trust anchors either so I don't know if there's a good workaround.

if there *is* one, i'd be happy to use it but it should be documented more clearly (something that I can also patch).

thanks for any pointers on how to move this forward...

@TheAnarcat yes we open the firewall on the PuppetDB hosts only from specific hosts (puppetmasters and cumin masters basically) and use the Puppet certificate for the HTTPS part.

Volans moved this task from Backlog to In Progress on the SRE-tools board.Mar 21 2019, 6:41 PM

Volans moved this task from In Progress to In Code Review on the SRE-tools board.

Change 497309 had a related patch set uploaded (by Volans; owner: TheAnarcat):
[operations/software/cumin@master] PuppetDB backend: allow to override URL scheme in config

https://gerrit.wikimedia.org/r/497309

gerritbot added a project: Patch-For-Review.Mar 21 2019, 6:42 PM

Volans mentioned this in rCUMIN8e49a21f5758: PuppetDB backend: allow to override URL scheme in config.Mar 21 2019, 6:43 PM

Change 497309 merged by jenkins-bot:
[operations/software/cumin@master] PuppetDB backend: allow to override URL scheme in config

https://gerrit.wikimedia.org/r/497309

Volans closed this task as Resolved.Mar 21 2019, 7:23 PM

Volans assigned this task to TheAnarcat.

Aklapper edited projects, added Cumin; removed SRE-tools.May 10 2023, 11:15 AM

Maintenance_bot removed a project: Patch-For-Review.May 10 2023, 11:33 AM

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptMay 10 2023, 11:34 AM