Page MenuHomePhabricator

Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish)
Open, LowPublic

Description

Videos in the WebM format tend to load the first 1-2 seconds, and then stop due to a "network error".

Screenshot 2025-06-23 at 13.34.06.png (1,554×1,270 px, 128 KB)

Videos in the MP4 format don't play at all ("No video supported format and MIME type"). I'm guessing the MP4 format is such that the initial response has no poster or renderable video, and so it fully relies on the Range request (second request)

Screenshot 2025-06-23 at 13.44.27.png (840×340 px, 24 KB)

When opening the WebM video in a new tab, this is consistently reproducible.

Screenshot 2025-06-23 at 13.33.21.png (2,395×1,689 px, 557 KB)

It is also consistently reproducible from the command-line, e.g. with curl.

curl 'https://phab.wmfusercontent.org/file/data/jg3i2o6piawj22nzjqhs/PHID-FILE-gd62glvlpvbshlwwrqz4/edit-toast-click_to_disappear.webm' -H 'Range: bytes=1660261-' -i 
HTTP/2 503 
server: Varnish
…
x-cache: cp4039 int
x-cache-status: int-front
…
<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<title>Wikimedia Error</title>
…
Error: 503, Backend fetch failed at Mon, 23 Jun 2025 20:32:23 GMT

More information

I suspect the reason this works in Chrome is that it either has a larger buffer before it makes a Range request, or it might be setting different headers that case the server to fallback to sending the file in full if it doesn't support Range requests.

If the above example returns a 404 Error, then the temporary URL has expired. Use the below example to try a current URL.

MP4 and WebM video from T309222#9342639:

MP4:

WebM:

[…]

WebM video from T375315#10231154:

[…]

See also:

Event Timeline

FWIW I can't reproduce on Linux with firefox or chrome. With the provided curl reproducer I can see how the request hits the applayer and comes back to be discarded by varnish with the following error:

--- Error          Invalid content-range header

backend response looks like this:

--- BerespProtocol HTTP/1.1                                                             
--- BerespStatus   206                                                                  
--- BerespReason   Partial Content                                                      
--- BerespHeader   date: Tue, 24 Jun 2025 09:48:57 GMT                                  
--- BerespHeader   server: Apache                                                       
--- BerespHeader   x-powered-by: PHP/7.4.33                                             
--- BerespHeader   x-frame-options: Deny                                                
--- BerespHeader   strict-transport-security: max-age=0; includeSubdomains; preload     
--- BerespHeader   content-security-policy: default-src https://phab.wmfusercontent.org; img-src https://phab.wmfusercontent.org data:; style-src https://phab.wmfusercontent.org
 'unsafe-inline'; script-src https://phab.wmfusercontent.org; connect-src 'self'; frame-src 'self'; frame-ancestors 'none'; object-src 'none'; form-action 'self'; base-uri 'none
'                                                                                       
--- BerespHeader   referrer-policy: no-referrer                                         
--- BerespHeader   accept-ranges: bytes                                                 
--- BerespHeader   cache-control: max-age=2592000, private                              
--- BerespHeader   expires: Thu, 24 Jul 2025 09:48:57 GMT                               
--- BerespHeader   x-content-type-options: nosniff                                      
--- BerespHeader   content-range: bytes 1660261-1660260/1660261                         
--- BerespHeader   backend-timing: D=105618 t=1750758537589554
--- BerespHeader   content-length: 0                                                    
--- BerespHeader   content-type: video/webm                                             
--- BerespHeader   x-envoy-upstream-service-time: 106                                   
--- BerespHeader   Vary: X-Experiment-Enrollments                                       
--- BerespHeader   Age: 0                                                               
--- BerespHeader   Connection: keep-alive                                               
--- BerespHeader   X-Cache-Int: cp6011 miss                                             
--- BerespHeader   X-ATS-Timestamp: 1750758537

it looks like Varnish doesn't like the Content-Range value:

--- BerespHeader   content-range: bytes 1660261-1660260/1660261

From varnish source code:

	if (*lo > *hi)
		return (-2);
	assert(cl >= -1);
	if (*lo >= cl || *hi >= cl)
		return (-2);

in this particular response C-L is 0 (content-length: 0) and lo is 1660261, triggering:

	if (*lo >= cl || *hi >= cl)
		return (-2);

TL;DR varnish doesn't allow a response where C-L is smaller than the low or high range provided in C-R.

I have not managed yet to reproduce either. Is this still an issue that folks experience?

Yeah, I can still repro on a couple of devices: for me in Firefox on Windows I experience what's described in this task's description, and in iOS Safari the non-MP4 videos in this task's description don't seem to play anything at all.

Yes, it remains consistently broken in Firefox on Mac and Windows. It seems the Linux version does not trigger the issue.

Yes, it remains consistently broken in Firefox on Mac and Windows. It seems the Linux version does not trigger the issue.

I use Firefox + Ubuntu and I always get "Video playback aborted due to a network error" on the initial page load. If I refresh a few times, it will let me play the videos.

I've just been opening tasks in Chromium if I need to watch a video. A bit annoying, but a workaround nonetheless.

@Vgutierrez: Would you have any shareable opinions how to proceed? Can we do anything on our side here, and/or shall we (not)?
Or is there an issue we should forward to an upstream project? Thanks in advance!

Taking a second look at the curl reproducer, I'm seeing the following behavior:

First request using:

curl 'https://phab.wmfusercontent.org/file/data/jg3i2o6piawj22nzjqhs/PHID-FILE-gd62glvlpvbshlwwrqz4/edit-toast-click_to_disappear.webm?vgutierrez=100' 
-H 'Range: bytes=1660261-' -v -o /dev/null

(Note the ?vgutierrez=100 query parameter)

This triggers:

  • Cache miss on Varnish, the Range header is stripped from the backend request
  • Cache miss on ATS
  • 200 response from phabricator.discovery.wmnet
  • 200 response from ATS
  • 200 response from Varnish
  • Client receives ~1.6 MB of content

If I repeat the request, it triggers:

  • Hit-for-pass on Varnish, the Range header is not stripped
  • Cache miss on ATS
  • 206 response from phabricator.discovery.wmnet (with Content-Length: 0)
  • 206 response from ATS
  • 503 response from Varnish due to Content-Length: 0

If I bypass the CDN and hit phabricator.discovery.wmnet directly, it responds with an empty body and C-L: 0, so I'm wondering if that Range header is even valid for that specific URL.

So we have two issues here:

  • Inconsistent CDN behavior regarding whether the Range header is stripped (differs between cache miss and hit-for-pass)
  • Backend returning an empty response when the Range header is present

following up on my last comment, the webm file size is 1660261 bytes, so a request asking for a range starting at 1660261 should probably trigger a 416 Range Not Satisfiable response instead of a 503 but it still doesn't look like a valid request for me, it I use a valid range like 1000-1024, the request gets a valid response every time but inconsistent, the first one it gets a 1660261 bytes response back, and the following ones a 25 bytes response as requested on the Range header