Page MenuHomePhabricator

[tools,harbor] failing to push images to harbor
Closed, ResolvedPublic

Description

When testing a release for builds-api, and testing a build, the build fails exporting the image to harbor:

[step-export] 2024-04-15T09:08:43.393187104Z Saving tools-harbor.wmcloud.org/tool-wm-lol/tool-wm-lol:latest...
[step-export] 2024-04-15T09:08:48.077025062Z *** Images (sha256:2e3b9992d19603e70335fc01e8b43f2f4678133da5f1e2095f163c568aafe2cb):
[step-export] 2024-04-15T09:08:48.077070117Z       tools-harbor.wmcloud.org/tool-wm-lol/tool-wm-lol:latest - PATCH https://tools-harbor.wmcloud.org/v2/tool-wm-lol/tool-wm-lol/blobs/uploads/0e1e89db-a807-47b5-9cab-28ab1356fa15?_state=REDACTED: unexpected status code 500 Internal Server Error: <html>
[step-export] 2024-04-15T09:08:48.077086008Z <head><title>500 Internal Server Error</title></head>
[step-export] 2024-04-15T09:08:48.077095143Z <body>
[step-export] 2024-04-15T09:08:48.077105465Z <center><h1>500 Internal Server Error</h1></center>
[step-export] 2024-04-15T09:08:48.077115784Z <hr><center>nginx/1.18.0</center>
[step-export] 2024-04-15T09:08:48.077124783Z </body>
[step-export] 2024-04-15T09:08:48.077133777Z </html>
[step-export] 2024-04-15T09:08:48.077142482Z 
[step-export] 2024-04-15T09:08:48.094211495Z ERROR: failed to export: failed to write image to the following tags: [tools-harbor.wmcloud.org/tool-wm-lol/tool-wm-lol:latest: PATCH https://tools-harbor.wmcloud.org/v2/tool-wm-lol/tool-wm-lol/blobs/uploads/0e1e89db-a807-47b5-9cab-28ab1356fa15?_state=REDACTED: unexpected status code 500 Internal Server Error: <html>
[step-export] 2024-04-15T09:08:48.094249412Z <head><title>500 Internal Server Error</title></head>
[step-export] 2024-04-15T09:08:48.094260024Z <body>
[step-export] 2024-04-15T09:08:48.094268554Z <center><h1>500 Internal Server Error</h1></center>
[step-export] 2024-04-15T09:08:48.094277661Z <hr><center>nginx/1.18.0</center>
[step-export] 2024-04-15T09:08:48.094323869Z </body>
[step-export] 2024-04-15T09:08:48.094336457Z </html>
[step-export] 2024-04-15T09:08:48.094346148Z ]

When going to the harbor web ui, it hangs loading, and on the container logs I see some errors (not very useful):

2024/04/15 09:03:19.286 [C] [config.go:500]  the request url is  /v2/toolforge/builds-api/blobs/sha256:c7ab89855e0287fbbb5f3d177ae80c1233d37d35f22034728615e376b87089ca                        
2024/04/15 09:03:19.287 [C] [config.go:501]  Handler crashed with error net/http: abort Handler                                                                                                
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/runtime/panic.go:914                                                                                                            
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/httputil/reverseproxy.go:519                                                                                           
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/repoproxy/proxy.go:78                                                                                               
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/repoproxy/proxy.go:54                                                                                               
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/metric/metric.go:91                                                                                                 
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/v2auth/auth.go:170                                                                                                  
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/router/router.go:92                                                                                                            
2024/04/15 09:03:19.287 [C] [config.go:507]  /go/pkg/mod/github.com/beego/beego/v2@v2.0.6/server/web/router.go:1149                                                                            
2024/04/15 09:03:19.287 [C] [config.go:507]  /go/pkg/mod/github.com/beego/beego/v2@v2.0.6/server/web/filter.go:83                                                                              
2024/04/15 09:03:19.287 [C] [config.go:507]  /go/pkg/mod/github.com/beego/beego/v2@v2.0.6/server/web/router.go:1002                                                                            
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:52                                                                                                    
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/security/security.go:75                                                                                             
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/security/security.go:62                                                                                             
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/artifactinfo/artifact_info.go:97                                                                                    
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.287 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:52                                                                                                    
2024/04/15 09:03:19.287 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/notification/notification.go:31                                                                                     
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/orm/orm.go:54                                                                                                       
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:52                                                                                                    
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/session/session.go:35                                                                                               
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/log/log.go:43                                                                                                       
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/requestid/requestid.go:46                                                                                           
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/metric/metric.go:62                                                                                                 
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/trace/trace.go:28                                                                                                   
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/mergeslash/mergeslash.go:31                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.288 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.288 [C] [config.go:507]  /harbor/src/server/middleware/url/parse.go:36                                                                                                     
2024/04/15 09:03:19.289 [C] [config.go:507]  /harbor/src/server/middleware/middleware.go:57                                                                                                    
2024/04/15 09:03:19.289 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2136                                                                                                         
2024/04/15 09:03:19.289 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2938                                                                                                         
2024/04/15 09:03:19.289 [C] [config.go:507]  /usr/local/go/src/net/http/server.go:2009                                                                                                         
2024/04/15 09:03:19.289 [C] [config.go:507]  /usr/local/go/src/runtime/asm_amd64.s:1650

Looking

Event Timeline

dcaro triaged this task as High priority.Apr 15 2024, 9:15 AM
dcaro created this task.

Restarting all the containers helped with the image pushing, that's restored.

For the UI, bypassing the instance proxy works, so it's probably related to the latest changes we made there.

From chrome I get:

ERR_HTTP2_PROTOCOL_ERROR

From curl:

HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

First search around the internet points to HTTP/2 not working, and fixing it by either disable proxy_buffering or forcing http/1.1:

https://stackoverflow.com/questions/56413290/getting-curl-92-http-2-stream-1-was-not-closed-cleanly-internal-error-err

It seems that the tmp directory was full on proxy-03.project-proxy.eqiad1.wikimedia.cloud, restarting nginx released the space and made things work again.

Closing this, will work on the tmp space on T354116