Page MenuHomePhabricator

zuul-client encrypt fails to retrieve public key
Open, Needs TriagePublic

Description

$ zuul-client --zuul-url https://zuul.wikimedia.org/ encrypt --tenant wikimedia --project integration/config --infile <(echo -n bar)
zuulclient.api.ZuulRESTException: Insufficient privileges to perform the action.
ERROR    - Command encrypt completed with error(s)

The client attempt to retrieve the key using the API: GET /api/tenant/{tenant_name}/key/{project_name}.pub

The API can be tried from https://zuul.wikimedia.org/openapi:

  • scroll to that API
  • hit the button Try it out
  • File the fields:
  • tenant_name: wikimedia
  • project_name: integration/config

Results

curl
curl -X 'GET' \
  'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub' \
  -H 'accept: text/plain'
url
https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub

Response

404
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

That is Apache responding with its own code, it should probably have given the result emitted by Zuul which is expected to be Tenant or Project not found. The URL not found is probably because Apache shortcircuited the URL encoded or did a file state or whatever madness.

If I do the request transforming the %2F% to /, it works!

$ curl https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration/config.pub
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA4Y9UWgnNStoHtWIyjF71
/r+DW0VH8yElfQUt36kxITm+BFtmMctyc5PfgcPMNzM+U75HxrydZHgrFeMuMhot
3Dj4IeBMA/kq94UWzfeI8kgGrSqSbgRz14P1J6PPIcRpJfukvkNt8qTjgcVCDymj
...

Once I get the key, I can invoke zuul-client --public-key <file holding the key>.

We should figure out the root cause of the URL not being accepted. Something something about urlencoding, I am pretty sure Gerrit has a similar requirement.

Event Timeline

Is that curl command generated by zuul-client or something else or is that just you testing?

Where does "curl -X 'GET' \

'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub' \
-H 'accept: text/plain'"

that contains the %2F come from?

a claim is made that: '"This happens to prevent directory traversal attacks. Apache evaluates the URL and intercepts it before your proxy rules have a chance to decode and route it properly to Zuul. This is exactly why Gerrit and similar CI services encounter this same behavior."

We might want AllowEncodedSlashes On in httpd config if we want to allow them.

Change #1307209 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: allow encoded slashes and proxy them unchanged

https://gerrit.wikimedia.org/r/1307209

I tested this. When allowing encoded slashes using "On" there is still a 404 but now it does not come from Apache httpd anymore.

The CherryPi and Unknown project here tells us this is now zuul.

[zuul1001:~] $ curl -X 'GET'   'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub'   -H 'accept: text/plain'
<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>404 Not Found</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>404 Not Found</h2>
        <p>Unknown project</p>
        <pre id="traceback"></pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.dev">CherryPy 18.10.0</a>
      </span>
    </div>
    </body>
</html>

When we use AllowEncodedSlashes NoDecode to ensure it also does not try to decode them and just pass them on to zuul, it works :)

[zuul1001:~] $ curl -X 'GET'   'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub'   -H 'accept: text/plain'
-----BEGIN PUBLIC KEY-----

Change #1307209 merged by Dzahn:

[operations/puppet@production] zuul: allow encoded slashes and proxy them undecoded

https://gerrit.wikimedia.org/r/1307209

@hashar fixed!

[zuul1001:~] $ curl -X 'GET' \
  'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub' \
  -H 'accept: text/plain'

-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA4Y9UWgnNStoHtWIyjF71
...

On https://zuul.wikimedia.org/openapi the API parameters can be filed with spaces or slashes and given they are string parameters they are rightfully quoted space to %20, and / to %2F (I have inserted a space in the middle of wikimedia to highlight the quoting):

zuul_openapi_params.png (668×185 px, 18 KB)

Which yields the following URL:
https://zuul.wikimedia.org/api/tenant/wiki%20media/key/integration%2Fconfig.pub

AllowEncodedSlashes Off (doc), caused Apache to reject the request immediately.

With AllowEncodedSlashes NoDecode, the request is thus passed as-is to the backend which receives integration%2Fconfig.pub.

Due to requests being cached by the CDN (T430462) we can't test directly on zuul.wikimedia.org, but we can hit the web app directly. On zuul1001.eqiad.wmnet, we now get a 404 for that request:

$ curl http://127.0.0.1:9000/api/tenant/wikimedia/key/integration%2Fconfig.pub
...
        <h2>404 Not Found</h2>

The web application routes configuration for this specific API is:

route_map = cherrypy.dispatch.RoutesDispatcher()
route_map.connect('api', '/api/tenant/{tenant_name}/key/'
                  '{project_name:.*}.pub',
                  controller=api, action='key')

The project_name variable there is thus set to integration%2Fconfig. It is passed as is to the action=key which is:

    @openapi_response(404, 'Tenant or Project not found')
    def key(self, tenant_name, tenant, auth, project_name):
        project = self._getProjectOrRaise(tenant, project_name)
...

    def _getProjectOrRaise(self, tenant, project_name):
        _, project = tenant.getProject(project_name)
        if not project:
            raise cherrypy.HTTPError(404, "Unknown project")
        return project

And that yields because the passed parameter is never decoded.


In the Javascript web app if I head to https://zuul.wikimedia.org/t/wikimedia/project/gerrit.wikimedia.org/integration/config it emits the request without encoding the parameters:

The clients (web app or zuul-client) should quote the parameters and the backend should unquote them. But they do not.

We should thus use AllowEncodedSlashes On (that is what upstream does and we do it as well for Gerrit).

Change #1307442 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] zuul: allow encoded slashes

https://gerrit.wikimedia.org/r/1307442

When we use AllowEncodedSlashes NoDecode to ensure it also does not try to decode them and just pass them on to zuul, it works :)

[zuul1001:~] $ curl -X 'GET'   'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub'   -H 'accept: text/plain'
-----BEGIN PUBLIC KEY-----

This can not work, it is definitely a cached response from an earlier attempt using AllowEncodedSlashes On which got cached by the CDN (T430462). I have verified it by hitting the backend directly which rejects the project with a 404:

$ curl http://127.0.0.1:9000/api/tenant/wikimedia/key/integration%2Fconfig.pub
...
        <h2>404 Not Found</h2>

Another thing I found on Thursday evening is our CDN rejects zuul-client requests to the Zuul API. It forbids queries because the client uses the default user-agent from the Python requests library. I have sent a patch Upstream to have an user-agent set: https://review.opendev.org/c/zuul/zuul-client/+/995988 and have the client to report the rejection reason https://review.opendev.org/c/zuul/zuul-client/+/995986

Change #1307442 merged by Dzahn:

[operations/puppet@production] zuul: allow encoded slashes

https://gerrit.wikimedia.org/r/1307442

At this point; are we just waiting for upstream? Or was there anything else I could do here?

Change #1314120 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: fix API 404s for encoded project slashes

https://gerrit.wikimedia.org/r/1314120

@hashar I have a new fix! This does it:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1314120/1/modules/zuul/files/zuul.wikimedia.org.conf

request directly to httpd (port 80), so nothing cached and also NOT to port 9000, with encoded slashes:

[zuul1001:~] $ curl http://zuul.discovery.wmnet:80/api/tenant/wikimedia/key/integration%2Fconfig.pub
-----BEGIN PUBLIC KEY-----

same as above without encoded slashes:

[zuul1001:~] $ curl http://zuul.discovery.wmnet:80/api/tenant/wikimedia/key/integration/config.pub
-----BEGIN PUBLIC KEY-----

directly to the backend, port 9000, without encoded slashes:

[zuul1001:~] $ curl http://zuul.discovery.wmnet:9000/api/tenant/wikimedia/key/integration/config.pub
-----BEGIN PUBLIC KEY-----

directly to the backend, port 9000, with encoded slashes (fails AS EXPECTED)

[zuul1001:~] $ curl http://zuul.discovery.wmnet:9000/api/tenant/wikimedia/key/integration%2Fconfig.pub
..
        <h2>404 Not Found</h2>

original test command, from external, with encoded slashes:

curl -X 'GET' \
  'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconfig.pub' \
  -H 'accept: text/plain'
-----BEGIN PUBLIC KEY-----

original test command, from external without encoded slashes:

curl -X 'GET'   'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration/config.pub'   -H 'accept: text/plain'
-----BEGIN PUBLIC KEY-----

please review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1314120

There are a bunch of comments and comments to the comments on Gerrit itself.

It seems like the test URL getting a 404 is actually considered "as designed". But that would contradict the ticket description.

Mentioned in SAL (#wikimedia-operations) [2026-07-27T20:51:24Z] <mutante> zuul1001 - re-enabled puppet - revert "cherry-picked" gerrit:1314120 - T431003

Re-enabled puppet on zuul1001 where I had applied gerrit:1314120 to showcase it as a possible fix.

Antoine has indicated he wants to keep following the approach of upstream though, which uses RewriteRules rather than pure ProxyPass config. Afaict using rewrite rules caused the problem described in this ticket though.

Unless I misunderstood we are therefore just waiting for a fix in upstream.

Not entirely sure if this is blocking switch to new zuul or just a nice to have.

Dzahn removed Dzahn as the assignee of this task.Mon, Jul 27, 8:54 PM

Change #1314120 abandoned by Dzahn:

[operations/puppet@production] zuul: fix API 404s for encoded project slashes

https://gerrit.wikimedia.org/r/1314120