Page MenuHomePhabricator

Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs
Open, Needs TriagePublic

Description

About 20 minutes ago, I followed the console links of the last three builds for this Gerrit change (patch set 3) from Zuul; in two of them, I got an error like this:

image.png (926×1 px, 114 KB)

After reloading, I could view the logs as usual.

Possibly related upstream docs: https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-masters/instance-navigation-often-fails-with-crumb-issuer-errors

Which says:

In some environments, this check would fail and causes No valid crumb was included in request. For example when a proxy is not preserving the client IP or the X-FORWARDED-FOR header is set but mis-configured

On the Jenkins server side:

Jan 13 16:10:43 contint2001 jenkins[17543]: WARNING: [hudson.security.csrf.CrumbFilter doFilter] Found invalid crumb xxxxxxxx. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
Jan 13 16:10:43 contint2001 jenkins[17543]: WARNING: [hudson.security.csrf.CrumbFilter doFilter] No valid crumb was included in request for /ci/job/quibble-vendor-mysql-php74-noselenium-docker/76991/logText/progressiveHtml by Lucas Werkmeister (WMDE). Returning 403.

And there was a second request for /ci/job/wmf-quibble-selenium-php74-docker/15950/logText/progressiveHtml which had the same token given. That matches the two tabs which were opened when the error happened.

A potential repro is to idle on an in progress job console. Reported via duplicate T327988

HTTP ERROR 403 No valid crumb was included in the request

URI:	/ci/job/wmf-quibble-core-vendor-mysql-php74-docker/8321/logText/progressiveHtml
STATUS:	403
MESSAGE:	No valid crumb was included in the request
SERVLET:	Stapler

Powered by Jetty:// 10.0.11

Screenshot 2023-01-25 at 23.42.51.png (1×1 px, 663 KB)

Event Timeline

I think the three builds would have been 1, 2 and 3, and 2 worked right away while 1 and 3 gave me the error until I reloaded the tabs, but I’m not 100% sure.

hashar added a project: Jenkins.
hashar subscribed.

That might be an issue with our ATS/Varnish cache stripping the XFF header, not passing the client IP or stripping some header that Jenkins relies on. Might want to dig into logs to find how often it happens.

Just got another one, when loading https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium-docker/80984/console (within seconds of opening it – I middle-clicked the link in Zuul, and by the time I switched to the tab the error was already there). A reload resolved it.