Jenkins Matrix builds containing a slash are not reachable
Closed, ResolvedPublic
Actions

Description

Jenkins matrix build lets one run several builds at once based on a list of parameters which are combined together. We use that for the *php-security jobs which are configured to generate build based on a list of projects.

An example is https://integration.wikimedia.org/ci/job/mediawiki-php-security/ which is made to run against MediaWiki core and vendor repositories against each of the master, REL1_31 and REL1_35 branches. The job page offers link to each of the combination which in the raw HTML looks like:

https://integration.wikimedia.org/ci/job/mediawiki-php-security/ZUUL_BRANCH=master,ZUUL_PROJECT=mediawiki%2Fcore,label=contint2001/

However that results in a 404 and in the Apache access.log we have:

http://integration.wikimedia.org/ci/job/mediawiki-php-security/ZUUL_BRANCH=master,ZUUL_PROJECT=mediawiki/core,label=contint2001/

The encoded slashes (%2F) are not preserved despite the Apache config having:

/etc/apache2/jenkins_proxy

ProxyPass       /ci http://localhost:8080/ci nocanon

The ZUUL_PROJECT variable is the Gerrit repository which uses slashes as a separator. That is also recognized as a path separator by Apache which I guess reject it.

Someone already cross filed bug reports to both Jenkins and Apache describing that exact same problem:

Seems we need to configure Apache with AllowEncodedSlashes On: https://httpd.apache.org/docs/2.4/en/mod/core.html#allowencodedslashes

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+7 -1	jenkins: allow path with encoded slashes

Customize query in gerrit

Event Timeline

hashar created this task.Jan 15 2021, 3:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2021, 3:45 PM

Jenkins has documentation about reverse proxying at https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-apache/ and they mention:

ProxyPass         /jenkins  http://localhost:8081/jenkins nocanon
ProxyPassReverse  /jenkins  http://localhost:8081/jenkins
ProxyRequests     Off
AllowEncodedSlashes NoDecode

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default
# See /etc/apache2/mods-enabled/proxy.conf in Ubuntu
<Proxy http://localhost:8081/jenkins*>
  Order deny,allow
  Allow from all
</Proxy>

And

Both the nocanon option to ProxyPass, and AllowEncodedSlashes NoDecode, are required for certain Jenkins features to work.

So looks like we simply need to use in Apache AllowEncodedSlashes NoDecode aka:

The AllowEncodedSlashes directive allows URLs which contain encoded path separators (%2F for / and additionally %5C for \ on accordant systems) to be used in the path info.
...
With the value NoDecode, such URLs are accepted, but encoded slashes are not decoded but left in their encoded state.

@Legoktm for information the builds of the php-security jobs are not reachable in Jenkins. They result in a 404 cause the ZUUL_PROJECT variable typically contains a / and that is decoded and rejected by Apache despite being properly url encoded by Jenkins :D

Sending an Apache config change right away.

Change 656443 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] jenkins: allow path with encoded slashes

https://gerrit.wikimedia.org/r/656443

gerritbot added a project: Patch-For-Review.Jan 15 2021, 4:10 PM

Change 656443 merged by Legoktm:
[operations/puppet@production] jenkins: allow path with encoded slashes

https://gerrit.wikimedia.org/r/656443

Mentioned in SAL (#wikimedia-operations) [2021-01-15T17:17:44Z] <legoktm> legoktm@contint2001:~$ sudo systemctl reload apache2 # for T272159

Confirmed the URL in the task description now works.

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2021, 6:10 PM

Jenkins Matrix builds containing a slash are not reachableClosed, ResolvedPublicActions

Description

Details

Event Timeline

Jenkins Matrix builds containing a slash are not reachable
Closed, ResolvedPublic
Actions