Page MenuHomePhabricator

authoritative copy of 'root' files for upload.wikimedia.org is only in swift
Open, MediumPublic

Description

following up from T130449: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ at the moment files under upload.wikimedia.org/ are served from root container via rewrite.py middleware. ATM the only authoritative copy of those files is in swift, though we want the authoritative copy to be in version control/puppet

root@ms-fe1001:~# swift list root
crossdomain.xml
favicon.ico
index.html
robots.txt
root@ms-fe1001:~# swift download root crossdomain.xml
crossdomain.xml [auth 0.022s, headers 0.050s, total 0.051s, 0.007 MB/s]
root@ms-fe1001:~# swift download root robots.txt
robots.txt [auth 0.028s, headers 0.155s, total 0.155s, 0.000 MB/s]
root@ms-fe1001:~# swift download root index.html
index.html [auth 0.024s, headers 0.033s, total 0.034s, 0.481 MB/s]
root@ms-fe1001:~# cat crossdomain.xml 
<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.adobe.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
  <allow-access-from domain="*" />
</cross-domain-policy>
root@ms-fe1001:~# cat robots.txt 
User-agent: *
Disallow: /wikipedia/commons/archive/

root@ms-fe1001:~# head index.html 
<html>
       <head>
               <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
               <title>Wikimedia Upload</title>
               <link rel="shortcut icon" href="/favicon.ico">
               <style type="text/css">
                       * {
                               font-family: 'Gill Sans', 'Gill Sans MT', sans-serif;
                       }
                       a:link, a:visited {

Event Timeline

Change 278924 had a related patch set uploaded (by Filippo Giunchedi):
varnish: redirect upload.wikimedia.org to commons

https://gerrit.wikimedia.org/r/278924

Change 278924 merged by BBlack:
varnish: redirect upload.wikimedia.org to commons

https://gerrit.wikimedia.org/r/278924

akosiaris triaged this task as Medium priority.Apr 20 2016, 11:16 AM
BBlack claimed this task.
fgiunchedi claimed this task.
fgiunchedi added a subscriber: BBlack.

reopening since we've redirected the homepage which fixed the original issue, but not the accessory files listed above. sounds reasonable @BBlack ?

I guess you mean crossdomain.xml and robots.txt? Is there a plan for handling them outside of swift in general? I assume redirects aren't really a great idea for those files, so they need to come from somewhere internal?

I'm not sure I understand this task — when/how is rewrite.py is going to go away? We haven't really talked about that.

Even if that happens, why couldn't the thing that replaces it not be able to serve a few files for /? If the existence of the custom middleware is the issue, we could explore some of the stock middlewares (I vaguelly recall a static pages one somewhere).

Of all of the functionality rewrite.py implements, this is the least of our problems really. Having Swift serve its own robots.txt, crossdomain.xml etc. isn't really that bad, is it?

fgiunchedi renamed this task from don't serve upload.wikimedia.org 'root' from wmfrewrite/swift to authoritative copy of 'root' files for upload.wikimedia.org is only in swift.Apr 20 2016, 3:45 PM
fgiunchedi updated the task description. (Show Details)

@BBlack @faidon I've retitled/clarified a bit the scope of this task since it wasn't clear, originally in T130449 it was asked where those files live, and apparently the only copy is in swift afaics, whereas we'd want it in puppet for example. For that the static files middleware seems like a good idea, my comment re: rewrite.py going away sooner or later was referred to thumbor work but clearly nothing is set in stone

I'm not sure I see this as a problem, although having them in VCS would definitely be nice. Regardless, this vs. moving those files to be served directly out of our traffic infrastructure and the future deprecation of rewrite.py seem like orthogonal issues in nature.

definitely, I've removed Traffic since it got inherited from the parent and not relevant

They should be in VCS so people can propose changes to them, and have Puppet apply the changes when approved. It'd also be helpful for things like T64835: Setup a Swift cluster on beta-cluster to match production where we want other systems to act like production.