Inference Service pipeline intermittent failures
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ACraze
	Jan 11 2022, 7:40 PM

Description

The inference service pipelines seem to have an intermittent bug, where some of the images occasionally error when invoking pip during the image builds:

See: https://integration.wikimedia.org/ci/job/inference-services-pipeline-draftquality/36/execution/node/47/log/

/usr/bin/python3.7: Error while finding module specification for 'pip' (AttributeError: module '__main__' has no attribute '__file__')

It seems that this is due to a bug in a recent release of setuptools: https://github.com/pypa/setuptools/issues/3002#issuecomment-1006266993

The bug should be fixed in pip v60.3.1
https://github.com/pypa/setuptools/issues/3002#issuecomment-1006710343

This error doesn't happen on every build (I can do 'recheck' and sometimes it will pass). This makes me think that there is a node using an outdated version of pip somewhere.

Details

	Subject	Repo	Branch	Lines +/-
	outlink: update base image version	machinelearning/liftwing/inference-services	main	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: rMLISf1d1e12d24e2: outlink: update base image version
rMLIS23ccf337598b: draftquality: add transformer module

Event Timeline

• ACraze created this task.Jan 11 2022, 7:40 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2022, 7:40 PM

I updated the base image to the most recent version of buster and things seem to work again.

I think we may need to update the base image for all of the images (model-server and transformers) to make sure we are using the most recent version pip.

• ACraze mentioned this in rMLIS23ccf337598b: draftquality: add transformer module.Jan 14 2022, 5:43 AM

Change 754015 had a related patch set uploaded (by Accraze; author: Accraze):

[machinelearning/liftwing/inference-services@main] outlink: update base image version

https://gerrit.wikimedia.org/r/754015

gerritbot added a project: Patch-For-Review.Jan 14 2022, 6:54 PM

Change 754015 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] outlink: update base image version

https://gerrit.wikimedia.org/r/754015

• ACraze mentioned this in rMLISf1d1e12d24e2: outlink: update base image version.Jan 18 2022, 9:33 AM

Maintenance_bot removed a project: Patch-For-Review.Jan 18 2022, 10:10 AM

calbon moved this task from Parked to In Progress on the Machine-Learning-Team (Active Tasks) board.Jan 19 2022, 6:15 PM

Ok I think all images that need to be updated have been updated. Going to mark this as RESOLVED.