Page MenuHomePhabricator

Tools Labs proxy should not overwrite error page
Open, NormalPublic

Description

When a tool is emitting an error page in response, the proxy is currently replacing the response with a error. The original headers and body are suppressed.

This obscures a lot of errors and makings debugging virtually impossible.

Example file:

<?php
http_response_code( 500 );
header( 'Content-Type: text/html; charset=utf-8' );
?><!DOCTYPE html>
<html>
  <meta charset="utf-8">
  <title>Custom error</title>
  <body>
    <h1>Cannot connect to example.db</h1>
  </body>
</html>

Hosted at https://tools.wmflabs.org/orphantalk/test.php

In response to a POST request, the proxy serves a generic nginx "400 Bad Request" page.

Example:

curl 'https://tools.wmflabs.org/orphantalk/test.php' --data 'x' -i -v
[..]
> POST /orphantalk/test.php HTTP/1.1
> User-Agent: curl/7.37.1
> Host: tools.wmflabs.org
> Accept: */*
> Content-Length: 1
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 1 out of 1 bytes
HTTP/1.1 500 Internal Server Error
Server: nginx/1.4.6 (Ubuntu)
Date: Wed, 24 Jun 2015 08:31:11 GMT
Content-Type: text/html
Content-Length: 349
Connection: close
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 - Bad Request</title>
 </head>
 <body>
  <h1>400 - Bad Request</h1>
 </body>
</html>

In response to a GET request, it serves a ToolLabs-style "Internal error" page.

> GET /orphantalk/test.php HTTP/1.1

HTTP/1.1 500 Internal Server Error
Server: nginx/1.4.6 (Ubuntu)
Date: Wed, 24 Jun 2015 08:57:37 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: close
X-Powered-By: PHP/5.5.9-1ubuntu4.9

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
  <head>
    <title>Tool Labs</title>
    <meta charset="utf-8" />
    <meta name="title" content="Tool Labs" />
    <meta name="description" content="This is the Tool Labs project for community-developed tools assisting the Wikimedia projects." />
[..]

Event Timeline

Krinkle created this task.Jun 24 2015, 8:35 AM
Krinkle raised the priority of this task from to High.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJun 24 2015, 8:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle renamed this task from Tools Labs proxy should not overwrite error page with generic "400 Bad Request" to Tools Labs proxy should not overwrite error page.Jun 24 2015, 8:58 AM
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.
scfc added a subscriber: scfc.Jun 24 2015, 2:49 PM

IIRC originally the reasoning was that with default error pages, all users would be pointed to webmaster@tools.wmflabs.org which would be unbearable. (And I think I wrote something like that a few weeks ago, so this task may be a duplicate of something?)

What would be nice:

  • Change the default lighttpd error pages in the tools' configurations to display the same information the "uber" error pages display currently.
  • Then rely on that and remove any overwrites.

I assume the 400 for POST comes from nginx trying to access tools.admin's web service for the Tools-style error page and somewhere in the pipeline only GETs are passed through.

If you pass the X-Wikimedia-Debug header and set it to 1 it'll pass them through as well, in the proxy.

valhallasw lowered the priority of this task from High to Normal.Jul 2 2015, 6:59 PM
valhallasw added a subscriber: valhallasw.
valhallasw moved this task from Triage to Backlog on the Toolforge board.Jul 2 2015, 7:09 PM

I've been digging a bit into the nginx lua module, and I think we can actually do something really cool: wrap the actual error page with a more generic message: you'd see the generic error message, with a fold-out with more detailed error information.
(http://wiki.nginx.org/HttpLuaModule#body_filter_by_lua / http://wiki.nginx.org/HttpLuaModule#ngx.status).

On the other hand, I think scfc's option makes more engineering sense, and is probably much easier to implement. After all, /data/project/tools.admin/public_html is readable for everyone, so it can just server the relevant .php file from there.

Problem with the lighttpd approach is not sure what that does for uwsgi, nodejs, etc.

valhallasw added a comment.EditedAug 28 2015, 7:42 PM

The disadvantage to that is that fcgi calls are never decorated with error pages in lighttpd, so a

<?php
header('HTTP/1.1 501 Error');

in php will just return an empty page.

For uwsgi, nodejs, etc, it does 'whatever that framework does', but that's not too unreasonable, I think.