Page MenuHomePhabricator

Receiving 413 error codes when trying to save notebook or upload certain files
Closed, ResolvedPublic

Description

This is two issues really that I assume are connected:

  • I tried to upload a 21.2 MB file to PAWS and got the following error code: Invalid response: 413 Request Entity Too Large. I don't know the exact threshold but I have uploaded larger files than this in the past without issue. If the new policy is to restrict uploads over e.g., 20MB, that's fine but currently it's confusing because when you try the upload, you first get a Large file size warning: The file size is 20 MB. Do you still want to upload it? confirmation box that suggests it's okay to do so.
  • Second issue is that my notebook was failing to save (similar to T163157). I don't know how to see how large the notebook was before saving but it was at 16KB and the difference was printing out a string with 1586800 characters (which is definitely long but still should only add about 1.5 MB to the notebook size assuming 1 byte per character which is reasonable because it's all ASCII)

Event Timeline

Perhaps related to https://github.com/jupyterlab/jupyterlab/issues/4214 rather than jupyter itself. It would appear that jupyter has a default maximum for notebooks (though not files) of 25M I'm getting a failure somewhere between 732K and 789K (732K works, 789K does not gives the error you mentioned above).

ingress-nginx did switch to a newer, and upstream, version a few weeks ago. So it seems likely that this is the cause considering the github issue. Updating client_max_body_size on the ingress-nginx to 0 may remedy this.

Ahh thanks for tracking down likely source @rook! 25M for notebook size sounds much more reasonable.

I am experiencing this issue as well. I can't upload a 6MB JSON file, when I have uploaded much larger files before—I think even hundreds of MB.

Changing the ingress nginx deploy from the current in https://wikitech.wikimedia.org/wiki/PAWS/Admin to:

helm upgrade --install ingress-nginx ingress-nginx \
  --version v4.4.0 \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.service.enableHttps=false \
  --set controller.service.nodePorts.http=30001 \
  --set-string controller.config.proxy-body-size="4m"

Allows me to upload a file that is over 100M (And may allow more, I stopped checking at that point), though we probably shouldn't upload things that large to PAWS, none the less goes beyond what seems like an excessively small limit. I can get this deployed to production PAWS tomorrow.

rook changed the task status from Open to In Progress.Jan 29 2023, 8:33 PM
rook claimed this task.

This is deployed and documentation is updated. I can upload the same file that I was able to in paws-dev to paws prod now.

Mentioned in SAL (#wikimedia-cloud) [2023-01-30T11:53:26Z] <Rook> updated ingress-nginx to allow larger file (more than 800K) uploads T328168