Page MenuHomePhabricator

Add 'Content-Length' in ws-export HTTP Response
Open, LowPublicFeature

Description

Feature summary (what you would like to be able to do and where):

Response currently does not include the Content Length of the file that is being downloaded. Adding this to response helps users know the file size prior.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

Readium's Kotlin toolkit has a "Download from the web" feature and it relies on Content Length.

Benefits (why should this be implemented?):

We will be able to know the file size before hand

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I've not looked into this deeply yet, but it appears it may be related to the Toolforge proxy. The app is sending Content-Length correctly on my local Apache:

$ curl -i 'http://localhost:8001?lang=en&page=Emma&credits=0'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=0, must-revalidate, private
Content-Description: File Transfer
Content-Disposition: attachment; filename=Emma.epub
Content-Length: 67079
Content-Type: application/epub+zip
Date: Mon, 27 Jan 2025 06:01:15 GMT
Expires: Mon, 27 Jan 2025 06:01:15 GMT
Host: localhost:8001
Last-Modified: Mon, 27 Jan 2025 06:01:15 GMT
Set-Cookie: PHPSESSID=--; expires=Wed, 29 Jan 2025 06:01:15 GMT; Max-Age=172800; path=/; httponly; samesite=lax
X-Debug-Token: 6a7707
X-Debug-Token-Link: http://localhost:8001/_profiler/6a7707
X-Powered-By: PHP/8.3.6
X-Robots-Tag: none

But not on the VPS:

$ curl -i 'https://ws-export.wmcloud.org?lang=en&page=Emma&credits=0'
HTTP/2 200 
server: nginx/1.18.0
date: Mon, 27 Jan 2025 06:01:56 GMT
content-type: application/epub+zip
cache-control: max-age=0, must-revalidate, private
x-robots-tag: none
content-description: File Transfer
content-disposition: attachment; filename=Emma.epub
accept-ranges: bytes
expires: Mon, 27 Jan 2025 06:01:56 GMT
set-cookie: PHPSESSID=--; expires=Wed, 29-Jan-2025 06:01:56 GMT; Max-Age=172800; path=/; secure; httponly; samesite=lax
vary: X-Forwarded-For
last-modified: Mon, 27 Jan 2025 06:01:56 GMT
strict-transport-security: max-age=31622400
x-clacks-overhead: GNU Terry Pratchett
permissions-policy: browsing-topics=()

I've not looked into this deeply yet, but it appears it may be related to the Toolforge proxy. The app is sending Content-Length correctly on my local Apache:

It's presumably because compression is enabled in nginx. Since the ePub is already compressed from the app, you could probably turn off compression in nginx for ws-export. But that requires tweaking nginx config, and making it different for just ws-export, so may not be worth the hassle. Or maybe it'd make sense (for several reasons) to serve the static files from a different domain instead, and have that configured for static assets and bulk downloads.

Yes, I was thinking something the same. I thought Content-Length would only be removed if we were sending e.g. Transfer-Encoding:gzip but I guess not.

I totally agree that a better fix would be to serve the files separately. In T265660 we were going to continue on to add caching of the generated files, but never finished that work. I've opened T385138 to discuss how that might work.

There's not much we can do about this task for now, I think.

Readium's Kotlin toolkit has a "Download from the web" feature and it relies on Content Length.

That seems like a shortcoming of that tool. Perhaps a bug report to them might be in order? There's nothing wrong with serving files without a Content-Length header.

taavi subscribed.

(Removing Toolforge since wmcloud.org is not Toolforge.)