Page MenuHomePhabricator

Cumin fails on huge nodelists emitted by its own outputs
Closed, ResolvedPublic


I had started with a cumin command over * (all nodes) to separate all our machines into groupings based on a certain command output. Then I took one of the output nodelists from that command and tried to paste it back as an input nodelist for a further command on one sub-group, but the (fed-back from outputs) nodelist was too large, resulting in an HTTP 414 error:

bblack@neodymium:~$ sudo cumin 'bast[1001,3002],,,conf[1001-1003].eqiad.wmnet,copper.eqiad.wmnet,cp[3004-3008,3010].esams.wmnet,,,db[2010-2012,2016-2019,2023,2028-2030].codfw.wmnet,db[1001,1009,1011,1015-1016,1018,1020-1024,1026,1028-1031,1033-1039,1041,1043-1050].eqiad.wmnet,dbproxy[1001-1011].eqiad.wmnet,,es[2001-2004].codfw.wmnet,eventlog2001.codfw.wmnet,graphite1002.eqiad.wmnet,hafnium.eqiad.wmnet,helium.eqiad.wmnet,,,labcontrol[1001-1002],labnet[1001-1002].eqiad.wmnet,labnodepool1001.eqiad.wmnet,labsdb[1001,1003-1007].eqiad.wmnet,labstore[2001,2003-2004].codfw.wmnet,labstore1003.eqiad.wmnet,labtestneutron2001.codfw.wmnet,,labtestvirt2002.codfw.wmnet,lvs[3001-3004].esams.wmnet,lvs[1001-1006],,maps-test[2001-2004].codfw.wmnet,mc[1001-1016].eqiad.wmnet,,ms-be[3001-3004].esams.wmnet,ms-fe[3001-3002].esams.wmnet,,mw2017.codfw.wmnet,mw[1259-1260].eqiad.wmnet,,,oxygen.eqiad.wmnet,poolcounter1002.eqiad.wmnet,,rcs[1001-1002].eqiad.wmnet,rdb[1007-1008].eqiad.wmnet,,rhodium.eqiad.wmnet,ruthenium.eqiad.wmnet,,stat1002.eqiad.wmnet,tungsten.eqiad.wmnet,' 'ls /sys/devices/system/node/node*'
Caught HTTPError exception: 414 Client Error: Request-URI Too Large for url: https://nitrogen.eqiad.wmnet:443/v3/nodes?

Revisions and Commits

Event Timeline

@BBlack yes that is a puppetdb error when the limit is reached.
If you have already an authoritative list of hosts in NodeSet notation (the
one printed by cumin), you can use --backend direct to use that as is
without querying puppetdb.

Once we will upgrade to PuppetDB API v4 I will move the PuppetDB queries in Cumin from using GET to using POST to overcome this limit and see if we find any other limit. The v3 of PuppetDB API don't accept POST unfortunately.

In the meanwhile, if the --backend direct workaround is not enough, we could try to increase the Nginx and Jetty limits of the GET requests to overcame this limitation.


With the global grammar available in Cumin, the way to do it now is to use the direct backend syntax D{} inside the query.

So basically copy the long comma-separated list of hosts in the Cumin output (it's a printed NodeSet) you want to execute a command to and do:

cumin 'D{__LONG_LIST_OF_HOSTS_}' 'command'

I'm resolving it given that this is a limitation in PuppetDB's API and there is a working workaround.

Volans claimed this task.