Page MenuHomePhabricator

External Monitoring alerting on 400 Bad Request errors
Closed, ResolvedPublic

Description

The HTTP checks configured in the service formerly known as "watchmouse" are alerting.

They say we are issuing 400 responses to their probes:

Server Response:
HTTP/1.1 400 Bad Request

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I cannot find any indication that the 400s are originating from our servers either in webrequest log or turnilo.

I have temporarily disabled the alerts until this can be looked into further or other indications of issues on our systems are indicated.

colewhite triaged this task as Medium priority.

In the checks definition, under "Advanced" -> "Custom HTTP request headers" we used to specify a header with quotes, which was passed verbatim by CA App Synthetic Monitor.

Quotes removed, checks are now green. It seems that between Varnish 5 and Varnish 6 something changed in the handling of silly request headers:

Varnish 6:

$ printf 'GET /wiki/Main_Page HTTP/1.1\r\nHost: en.wikipedia.org\r\nX-Forwarded-Proto: https\r\n"Cookie: prova"\r\n\r\n' | nc localhost 80 | head -1
HTTP/1.1 400 Bad Request

Varnish 5:

$ printf 'GET /wiki/Main_Page HTTP/1.1\r\nHost: en.wikipedia.org\r\nX-Forwarded-Proto: https\r\n"Cookie: prova"\r\n\r\n' | nc localhost 80 | head -1
HTTP/1.1 200 OK

Notifications re-enabled.