The puppet db micro service puppetdb-api.discovery.wmnet does not handle large queries very well. This can be demonstrated by the following two queries
$ time curl -X POST http://localhost:8080/pdb/query/v4/resources --data '{"query": ["=", "type", "File"]}' -H 'Content-Type: application/json' &> /dev/null curl -X POST http://localhost:8080/pdb/query/v4/resources --data -H &> 2.03s user 2.85s system 1% cpu 7:17.45 total $ time curl -X POST https://puppetdb-api.discovery.wmnet:8090/pdb/query/v4/resources --data '{"query": ["=", "type", "File"]}' -H 'Content-Type: application/json' <html> <head><title>504 Gateway Time-out</title></head> <body bgcolor="white"> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx/1.14.2</center> </body> </html> curl -X POST https://puppetdb-api.discovery.wmnet:8090/pdb/query/v4/resources 0.02s user 0.02s system 0% cpu 1:00.06 total
The issue is that the flask service reads the data into memory iterates and modifies it before then sending it to the client, which causes a crash probably from OOM. It would be better if we could stream the data directly to the client and modify it on the fly.
The main affct this has is that cuminunpriv is unable to lookup very generic resources or classes that are used in a lot of places. I also suspect that genral queries will be slower on cuminunpriv vs cumin (which goes directly to puppetdb)