Page MenuHomePhabricator
Create Task
Maniphest T166397

Cumin fails on huge nodelists emitted by its own outputs
Closed, ResolvedPublic
Actions

Description

I had started with a cumin command over * (all nodes) to separate all our machines into groupings based on a certain command output. Then I took one of the output nodelists from that command and tried to paste it back as an input nodelist for a further command on one sub-group, but the (fed-back from outputs) nodelist was too large, resulting in an HTTP 414 error:

bblack@neodymium:~$ sudo cumin 'bast[1001,3002].wikimedia.org,californium.wikimedia.org,chromium.wikimedia.org,conf[1001-1003].eqiad.wmnet,copper.eqiad.wmnet,cp[3004-3008,3010].esams.wmnet,cp1008.wikimedia.org,dataset1001.wikimedia.org,db[2010-2012,2016-2019,2023,2028-2030].codfw.wmnet,db[1001,1009,1011,1015-1016,1018,1020-1024,1026,1028-1031,1033-1039,1041,1043-1050].eqiad.wmnet,dbproxy[1001-1011].eqiad.wmnet,eeden.wikimedia.org,es[2001-2004].codfw.wmnet,eventlog2001.codfw.wmnet,graphite1002.eqiad.wmnet,hafnium.eqiad.wmnet,helium.eqiad.wmnet,hydrogen.wikimedia.org,iron.wikimedia.org,labcontrol[1001-1002].wikimedia.org,labnet[1001-1002].eqiad.wmnet,labnodepool1001.eqiad.wmnet,labsdb[1001,1003-1007].eqiad.wmnet,labstore[2001,2003-2004].codfw.wmnet,labstore1003.eqiad.wmnet,labtestneutron2001.codfw.wmnet,labtestservices2001.wikimedia.org,labtestvirt2002.codfw.wmnet,lvs[3001-3004].esams.wmnet,lvs[1001-1006].wikimedia.org,maerlant.wikimedia.org,maps-test[2001-2004].codfw.wmnet,mc[1001-1016].eqiad.wmnet,ms1001.wikimedia.org,ms-be[3001-3004].esams.wmnet,ms-fe[3001-3002].esams.wmnet,multatuli.wikimedia.org,mw2017.codfw.wmnet,mw[1259-1260].eqiad.wmnet,nescio.wikimedia.org,netmon1001.wikimedia.org,oxygen.eqiad.wmnet,poolcounter1002.eqiad.wmnet,radium.wikimedia.org,rcs[1001-1002].eqiad.wmnet,rdb[1007-1008].eqiad.wmnet,rhenium.wikimedia.org,rhodium.eqiad.wmnet,ruthenium.eqiad.wmnet,silver.wikimedia.org,stat1002.eqiad.wmnet,tungsten.eqiad.wmnet,uranium.wikimedia.org' 'ls /sys/devices/system/node/node*'
Caught HTTPError exception: 414 Client Error: Request-URI Too Large for url: https://nitrogen.eqiad.wmnet:443/v3/nodes?query=%5B%22or%22%2C+%5B%22%3D%22%2C+%22name%22%2C+%22bast1001.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22bast3002.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22californium.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22chromium.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22conf1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22conf1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22conf1003.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22copper.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22cp3004.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22cp3005.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22cp3006.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22cp3007.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22cp3008.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22cp3010.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22cp1008.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dataset1001.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2010.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2011.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2012.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2016.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2017.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2018.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2019.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2023.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2028.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2029.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db2030.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1009.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1011.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1015.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1016.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1018.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1020.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1021.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1022.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1023.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1024.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1026.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1028.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1029.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1030.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1031.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1033.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1034.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1035.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1036.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1037.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1038.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1039.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1041.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1043.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1044.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1045.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1046.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1047.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1048.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1049.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22db1050.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1003.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1004.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1005.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1006.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1007.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1008.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1009.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1010.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22dbproxy1011.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22eeden.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22es2001.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22es2002.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22es2003.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22es2004.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22eventlog2001.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22graphite1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22hafnium.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22helium.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22hydrogen.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22iron.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labcontrol1001.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labcontrol1002.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labnet1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labnet1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labnodepool1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labsdb1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labsdb1003.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labsdb1004.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labsdb1005.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labsdb1006.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labsdb1007.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labstore2001.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labstore2003.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labstore2004.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labstore1003.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labtestneutron2001.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labtestservices2001.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22labtestvirt2002.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs3001.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs3002.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs3003.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs3004.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs1001.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs1002.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs1003.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs1004.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs1005.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22lvs1006.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22maerlant.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22maps-test2001.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22maps-test2002.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22maps-test2003.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22maps-test2004.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1003.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1004.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1005.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1006.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1007.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1008.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1009.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1010.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1011.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1012.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1013.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1014.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1015.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mc1016.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ms1001.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ms-be3001.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ms-be3002.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ms-be3003.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ms-be3004.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ms-fe3001.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ms-fe3002.esams.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22multatuli.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mw2017.codfw.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mw1259.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22mw1260.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22nescio.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22netmon1001.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22oxygen.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22poolcounter1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22radium.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22rcs1001.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22rcs1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22rdb1007.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22rdb1008.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22rhenium.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22rhodium.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22ruthenium.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22silver.wikimedia.org%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22stat1002.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22tungsten.eqiad.wmnet%22%5D%2C+%5B%22%3D%22%2C+%22name%22%2C+%22uranium.wikimedia.org%22%5D%5D

Revisions and Commits

rCUMIN Cumin
rCUMIN687bddf39386 PuppetDB backend: add support for API v4

Event Timeline

BBlack created this task.May 26 2017, 5:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2017, 5:16 PM
Volans added a comment.May 26 2017, 5:29 PM

@BBlack yes that is a puppetdb error when the limit is reached.
If you have already an authoritative list of hosts in NodeSet notation (the
one printed by cumin), you can use --backend direct to use that as is
without querying puppetdb.

Volans added a comment.May 27 2017, 4:21 PM

Once we will upgrade to PuppetDB API v4 I will move the PuppetDB queries in Cumin from using GET to using POST to overcome this limit and see if we find any other limit. The v3 of PuppetDB API don't accept POST unfortunately.

In the meanwhile, if the --backend direct workaround is not enough, we could try to increase the Nginx and Jetty limits of the GET requests to overcame this limitation.

Thoughts?

Volans added a comment.Sep 28 2017, 12:06 PM

With the global grammar available in Cumin, the way to do it now is to use the direct backend syntax D{} inside the query.

So basically copy the long comma-separated list of hosts in the Cumin output (it's a printed NodeSet) you want to execute a command to and do:

cumin 'D{__LONG_LIST_OF_HOSTS_}' 'command'

I'm resolving it given that this is a limitation in PuppetDB's API and there is a working workaround.

Volans closed this task as Resolved.Sep 28 2017, 12:06 PM
Volans claimed this task.
Volans added a commit: rCUMIN687bddf39386: PuppetDB backend: add support for API v4.Jan 18 2018, 3:29 PM
Aklapper edited projects, added Cumin; removed SRE-tools.May 10 2023, 11:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits