Maniphest T342084

Post-merge build failed due to Internal Server Error
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	kevinbazira
	Jul 18 2023, 8:47 AM

Description

The recommendation-api post-merge build failed in this patch due to unexpected HTTP status: 500 Internal Server Error as shown in this Jenkins post-merge build log.

We have tested the production image (created by blubber) that is supposed to be published to the wikimedia docker registry and it works well:

$ time docker build -t recommendation-api-prod20230718 .

...

Successfully built cff630cecacf
Successfully tagged recommendation-api-prod20230718:latest

real	6m11.289s
user	0m0.152s
sys	0m0.176s

$ docker images
REPOSITORY                                       TAG              IMAGE ID       CREATED          SIZE
recommendation-api-prod20230718                  latest           cff630cecacf   30 minutes ago   4.6GB

@elukey restarted the CI build hoping that this issue was transient but we got the same error as shown in this log.

What could be causing this Internal Server Error in the post-merge build?

Related Objects
Search...

Status	Assigned	Task
Open	None	T296994 Observations from research study for Section Translation on Thai Wikipedia
Open	None	T293648 Content Translation Recommendations API
Open	kevinbazira	T308164 Migrate Content Translation Recommendation API to Lift Wing
Open	kevinbazira	T339890 Host the recommendation-api container on LiftWing
Open	None	T288198 Pushes to docker-registry fail for images with compressed layers of size >1GB
Resolved	kevinbazira	T342084 Post-merge build failed due to Internal Server Error
Resolved	kevinbazira	T343576 Store and fetch the recommendation-api embedding from Swift

Event Timeline

kevinbazira created this task.Jul 18 2023, 8:47 AM

kevinbazira updated the task description. (Show Details)Jul 18 2023, 9:17 AM

On registry2003 I can only see this error:

level=error msg="response completed with error" err.code="name unknown" err.detail="map[name:wikimedia/research-recommendation-api]" err.message="repository name not known to registry"

@hashar hi! Are we missing any config in the integration repo by any chance?

calbon moved this task from Unsorted to In Progress on the Machine-Learning-Team board.Jul 18 2023, 2:27 PM

I built the image on contint1002 today. The image size is 4.6GB with one of the layers being 4.21GB. I tried pushing to the registry it kept failing when pushing the 4.21GB layer.

I browsed through the nginx logs via https://logstash.wikimedia.org/goto/ce91b70ec9e282c5ed7fbba0cc3fddd3 and found:

program:input-file-registry-nginx-access message:208.80.154.132 - ci-build [21/Jul/2023:16:17:20 +0000] "PATCH /v2/wikimedia/research-recommendation-api/blobs/uploads/d5a44664-d169-4552-b7f9-c08291670bae?_state=3cQcQ3IB-iq_IIZ4rnWKasAMOM6I5vwHzIRR_owNzVp7Ik5hbWUiOiJ3aWtpbWVkaWEvcmVzZWFyY2gtcmVjb21tZW5kYXRpb24tYXBpIiwiVVVJRCI6ImQ1YTQ0NjY0LWQxNjktNDU1Mi1iN2Y5LWMwODI5MTY3MGJhZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0wNy0yMVQxNjowNDoyNy40MDQ0OTM5NzdaIn0%3D HTTP/1.1" 500 193 "-" "docker/20.10.12 go/go1.16.12 git-commit/459d0df kernel/4.19.0-23-amd64 os/linux arch/amd64

program:input-file-registry-nginx-error message:2023/07/21 16:17:19 [crit] 4999#4999: *33472 pwritev() "/var/lib/nginx/body/0000000827" has written only 7884 of 8184, client: 208.80.154.132, server: , request: "PATCH /v2/wikimedia/research-recommendation-api/blobs/uploads/d5a44664-d169-4552-b7f9-c08291670bae?_state=3cQcQ3IB-iq_IIZ4rnWKasAMOM6I5vwHzIRR_owNzVp7Ik5hbWUiOiJ3aWtpbWVkaWEvcmVzZWFyY2gtcmVjb21tZW5kYXRpb24tYXBpIiwiVVVJRCI6ImQ1YTQ0NjY0LWQxNjktNDU1Mi1iN2Y5LWMwODI5MTY3MGJhZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0wNy0yMVQxNjowNDoyNy40MDQ0OTM5NzdaIn0%3D HTTP/1.1", host: "docker-registry.discovery.wmnet" @timestamp:Jul 21, 2023 @ 16:17:20.139 facility:local0 logsource:registry2004 normalized_message:2023/07/21 16:17:19 [crit] 4999#4999: *33472 pwritev() "/var/lib/nginx/body/0000000827" has written only 7884 of 8184, client: 208.80.154.132, server: , request: "PATCH /v2/wikimedia/research-recommendation-api/blobs/uploads/d5a44664-d169-4552-b7f9-c08291 @version:1 host:registry2004 type:syslog tags:rsyslog-shipper, kafka, es, syslog, es, normalized_message_trimmed timestamp:2023-07-21T16:17:19.991797+00:00 level:NOTICE _id:okQ9eYkBZQQsSJrNzE-a _type: - _index:logstash-syslog-1-7.0.0-1-2023.07.21 _score: -

I'm wondering how much free space is available in /var/lib/nginx/body on the nginx server.

Digging up T288198 reveals that /var/lib/nginx is hosted on a tmpfs of size 2GB. That would explain the problem. I'll try to revive the discussion on that ticket.

• dancy mentioned this in T288198: Pushes to docker-registry fail for images with compressed layers of size >1GB.Jul 21 2023, 7:33 PM

• dancy added a parent task: T288198: Pushes to docker-registry fail for images with compressed layers of size >1GB.Jul 21 2023, 7:43 PM

hashar unsubscribed.Jul 21 2023, 7:48 PM

Thank you for digging into this, @dancy! Enabling users to push to the docker-registry images with layers > 2GB (~4.21GB in our case) will unblock us on this issue.

To reduce image layer sizes, in T343576 we store and fetch the ~2.8GB recommendation-api embedding from Swift as recommended in T288198#9037109. This has enabled the post-merge build to succeed: https://integration.wikimedia.org/ci/job/trigger-recommendation-api-ng-pipeline-publish/4/console

kevinbazira mentioned this in T343951: Post-merge build succeeded but image not published to docker-registry.Aug 10 2023, 8:36 AM

kevinbazira mentioned this in T339890: Host the recommendation-api container on LiftWing.Aug 11 2023, 8:07 AM

kevinbazira closed this task as Resolved.Aug 11 2023, 8:49 AM

kevinbazira claimed this task.

isarantopoulos moved this task from In Progress to 2023-2024 Q3 Done on the Machine-Learning-Team board.Nov 2 2023, 10:47 PM

isarantopoulos closed subtask T343576: Store and fetch the recommendation-api embedding from Swift as Resolved.Nov 20 2023, 12:24 PM

isarantopoulos reopened subtask T343576: Store and fetch the recommendation-api embedding from Swift as Open.

isarantopoulos closed subtask T343576: Store and fetch the recommendation-api embedding from Swift as Resolved.