Page MenuHomePhabricator
Create Task
Maniphest T163157

Notebook fails to save, throwing "413 Request Entity Too Large" error
Closed, ResolvedPublic
Actions

Description

I have a particular PAWS notebook that I can't seem to save any more :-(

The notebook itself (in the browser tab) says "Autosave Failed!" "Not connected" "error" etc, and in the browser's JavaScript console I see a "413 Request Entity Too Large" error after a PUT request which seems to be an attempt to save the notebook.
Are there any size limits to be aware of? Or ways around that kind of problem?

The ipynb file is 900 kb large on disk.

The notebook is here (with the current version that is causing the problems backed up in my folder as "Mobile app edits on Wikidata.ipynb_toolarge").

Related Objects

Event Timeline

• Tbayer created this task.Apr 17 2017, 9:40 PM
• Tbayer added a comment.Apr 25 2017, 2:39 AM

PS: The notebook saved successfully after deleting parts of it, bringing the size of the .ipynb file down to 725K. Inspecting it in more detail, it looks like some matplotlib charts (in 300dpi resolution) took up most of the room; so that's what I'm going to look at for the purposes of circumventing this bug until it is fixed.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:37 PM
Chicocvenancio triaged this task as High priority.Feb 25 2018, 9:16 PM
zhuyifei1999 subscribed.Feb 26 2018, 1:48 AM

This 413 is uually an error thrown by nginx, but the nginx in proxy pod looks irrelevant:

P6740 PAWS nginx.conf
1# yuvipanda@tools-paws-master-01:~$ kubectl --namespace=prod exec -it proxy-5cd7d56555-tm4p6 /bin/bash
2# Defaulting container name to nginx.
3# Use 'kubectl describe pod/proxy-5cd7d56555-tm4p6 -n prod' to see all of the # containers in this pod.
4# root@proxy-5cd7d56555-tm4p6:/# cat /etc/nginx/nginx.conf
5
6daemon off;
7
8worker_processes 4;
9pid /run/nginx.pid;
10
11worker_rlimit_nofile 261120;
12
13worker_shutdown_timeout 10s ;
14
15events {
16 multi_accept on;
17 worker_connections 16384;
18 use epoll;
19}
20
21http {
22
23 real_ip_header X-Forwarded-For;
24
25 real_ip_recursive on;
26
27 set_real_ip_from 0.0.0.0/0;
28
29 geoip_country /etc/nginx/GeoIP.dat;
30 geoip_city /etc/nginx/GeoLiteCity.dat;
31 geoip_proxy_recursive on;
32
33 sendfile on;
34
35 aio threads;
36 aio_write on;
37
38 tcp_nopush on;
39 tcp_nodelay on;
40
41 log_subrequest on;
42
43 reset_timedout_connection on;
44
45 keepalive_timeout 75s;
46 keepalive_requests 100;
47
48 client_header_buffer_size 1k;
49 client_header_timeout 60s;
50 large_client_header_buffers 4 8k;
51 client_body_buffer_size 8k;
52 client_body_timeout 60s;
53
54 http2_max_field_size 4k;
55 http2_max_header_size 16k;
56
57 types_hash_max_size 2048;
58 server_names_hash_max_size 1024;
59 server_names_hash_bucket_size 32;
60 map_hash_bucket_size 64;
61
62 proxy_headers_hash_max_size 512;
63 proxy_headers_hash_bucket_size 64;
64
65 variables_hash_bucket_size 128;
66 variables_hash_max_size 2048;
67
68 underscores_in_headers off;
69 ignore_invalid_headers on;
70
71 include /etc/nginx/mime.types;
72 default_type text/html;
73
74 brotli on;
75 brotli_comp_level 4;
76 brotli_types application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
77
78 gzip on;
79 gzip_comp_level 5;
80 gzip_http_version 1.1;
81 gzip_min_length 256;
82 gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
83 gzip_proxied any;
84 gzip_vary on;
85
86 # Custom headers for response
87
88 server_tokens on;
89
90 # disable warnings
91 uninitialized_variable_warn off;
92
93 # Additional available variables:
94 # $namespace
95 # $ingress_name
96 # $service_name
97 log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';
98
99 map $request_uri $loggable {
100
101 default 1;
102 }
103
104 access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
105
106 error_log /var/log/nginx/error.log notice;
107
108 resolver 10.96.0.10 valid=30s;
109
110 # Retain the default nginx handling of requests without a "Connection" header
111 map $http_upgrade $connection_upgrade {
112 default upgrade;
113 '' close;
114 }
115
116 map $http_x_forwarded_for $the_real_ip {
117
118 default $remote_addr;
119
120 }
121
122 # trust http_x_forwarded_proto headers correctly indicate ssl offloading
123 map $http_x_forwarded_proto $pass_access_scheme {
124 default $http_x_forwarded_proto;
125 '' $scheme;
126 }
127
128 map $http_x_forwarded_port $pass_server_port {
129 default $http_x_forwarded_port;
130 '' $server_port;
131 }
132
133 map $http_x_forwarded_host $best_http_host {
134 default $http_x_forwarded_host;
135 '' $this_host;
136 }
137
138 map $pass_server_port $pass_port {
139 443 443;
140 default $pass_server_port;
141 }
142
143 # Obtain best http host
144 map $http_host $this_host {
145 default $http_host;
146 '' $host;
147 }
148
149 server_name_in_redirect off;
150 port_in_redirect off;
151
152 ssl_protocols TLSv1.2;
153
154 # turn on session caching to drastically improve performance
155
156 ssl_session_cache builtin:1000 shared:SSL:10m;
157 ssl_session_timeout 10m;
158
159 # allow configuring ssl session tickets
160 ssl_session_tickets on;
161
162 # slightly reduce the time-to-first-byte
163 ssl_buffer_size 4k;
164
165 # allow configuring custom ssl ciphers
166 ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
167 ssl_prefer_server_ciphers on;
168
169 ssl_ecdh_curve auto;
170
171 proxy_ssl_session_reuse on;
172
173 upstream upstream-default-backend {
174 # Load balance algorithm; empty for round robin, which is the default
175
176 least_conn;
177
178 keepalive 32;
179
180 server 10.244.7.45:8000 max_fails=0 fail_timeout=0;
181
182 }
183
184 ## start server _
185 server {
186 server_name _ ;
187
188 listen 80 default_server reuseport backlog=511;
189
190 listen [::]:80 default_server reuseport backlog=511;
191
192 set $proxy_upstream_name "-";
193
194 listen 443 default_server reuseport backlog=511 ssl http2;
195
196 listen [::]:443 default_server reuseport backlog=511 ssl http2;
197
198 # PEM sha: f4e0c71ae16dc74f4d8f71412a5d82a24579f42f
199 ssl_certificate /ingress-controller/ssl/default-fake-certificate.pem;
200 ssl_certificate_key /ingress-controller/ssl/default-fake-certificate.pem;
201
202 more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
203
204 location / {
205
206 set $proxy_upstream_name "upstream-default-backend";
207
208 set $namespace "";
209 set $ingress_name "";
210 set $service_name "";
211
212 port_in_redirect off;
213
214 client_max_body_size "64m";
215
216 proxy_set_header Host $best_http_host;
217
218 # Pass the extracted client certificate to the backend
219
220 proxy_set_header ssl-client-cert "";
221 proxy_set_header ssl-client-verify "";
222 proxy_set_header ssl-client-dn "";
223
224 # Allow websocket connections
225 proxy_set_header Upgrade $http_upgrade;
226 proxy_set_header Connection $connection_upgrade;
227
228 proxy_set_header X-Real-IP $the_real_ip;
229
230 proxy_set_header X-Forwarded-For $the_real_ip;
231
232 proxy_set_header X-Forwarded-Host $best_http_host;
233 proxy_set_header X-Forwarded-Port $pass_port;
234 proxy_set_header X-Forwarded-Proto $pass_access_scheme;
235 proxy_set_header X-Original-URI $request_uri;
236 proxy_set_header X-Scheme $pass_access_scheme;
237
238 # Pass the original X-Forwarded-For
239 proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
240
241 # mitigate HTTPoxy Vulnerability
242 # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
243 proxy_set_header Proxy "";
244
245 # Custom headers to proxied server
246
247 proxy_connect_timeout 5s;
248 proxy_send_timeout 60s;
249 proxy_read_timeout 60s;
250
251 proxy_redirect off;
252
253 proxy_buffering off;
254 proxy_buffer_size "4k";
255 proxy_buffers 4 "4k";
256 proxy_request_buffering "on";
257
258 proxy_http_version 1.1;
259
260 proxy_cookie_domain off;
261 proxy_cookie_path off;
262
263 # In case of errors try the next upstream server before returning an error
264 proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
265
266 proxy_pass http://upstream-default-backend;
267
268 }
269
270 # health checks in cloud providers require the use of port 80
271 location /healthz {
272 access_log off;
273 return 200;
274 }
275
276 # this is required to avoid error if nginx is being monitored
277 # with an external software (like sysdig)
278 location /nginx_status {
279 allow 127.0.0.1;
280 allow ::1;
281 deny all;
282
283 access_log off;
284 stub_status on;
285 }
286
287 }
288 ## end server _
289
290 # default server, used for NGINX healthcheck and access to nginx stats
291 server {
292 # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
293 # Changing this value requires a change in:
294 # https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
295 listen 18080 default_server reuseport backlog=511;
296 listen [::]:18080 default_server reuseport backlog=511;
297 set $proxy_upstream_name "-";
298
299 location /healthz {
300 access_log off;
301 return 200;
302 }
303
304 location /nginx_status {
305 set $proxy_upstream_name "internal";
306
307 access_log off;
308 stub_status on;
309
310 }
311
312 location / {
313
314 set $proxy_upstream_name "upstream-default-backend";
315 proxy_pass http://upstream-default-backend;
316 }
317
318 }
319}
320
321stream {
322 log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
323
324 access_log /var/log/nginx/access.log log_stream;
325
326 error_log /var/log/nginx/error.log;
327
328 # TCP services
329
330 # UDP services
331
332}
333

... having a client_max_body_size of 64m.

Chicocvenancio closed this task as Resolved.Jun 22 2018, 2:23 AM
Chicocvenancio claimed this task.
Chicocvenancio subscribed.

The paws-proxy-01 sits in front of paws right now, added client_max_body_size "64m"; there and it seems to solve the issue.

Isaac mentioned this in T328168: Receiving 413 error codes when trying to save notebook or upload certain files.Jan 27 2023, 8:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits