Page MenuHomePhabricator
Create Task
Maniphest T259611

The python-build images regenerate wheels even when matching ones are already available
Closed, ResolvedPublic
Actions

Description

When the deploy repository only has the artifacts.tar.gz, all wheels are build from scratch. When changing a single requirements, other unrelated wheels are modified when their build is not reproducible.

When the deploy repository has wheels committed in, a version bump would result in a second wheel for the same requirement. One can possibly delete the existing wheels but ends up with the same issue as above.

The python-build images have a script which takes a list of requirements (/deploy/frozen-requirements.txt), run pip wheel with resulting wheels written to /wheels. The images do not have an ability to be passed previously build wheels to skip building them.

The logic is in operations/docker-images/production-images.git images/python-build/run.sh

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Track wheels individuallyintegration/zuul/deploymaster+100 -27
python-build: set date of source files in the wheelsoperations/docker-images/production-imagesmaster+7 -1
Add LFS config for integration/zuul/deployAll-Projectsrefs/meta/config+5 -0
Reuse previous artifacts as a wheel cacheintegration/zuul/deploymaster+14 -1
python-build: reuse previously built wheelsoperations/docker-images/production-imagesmaster+87 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Aug 4 2020, 12:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2020, 12:38 PM
hashar added a parent task: T258630: Improve scheduling of CI jobs invoked by zuul .Aug 4 2020, 12:38 PM
gerritbot added a comment.Aug 4 2020, 12:40 PM

Change 605653 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/docker-images/production-images@master] python-build: allow reuse of existing wheels

https://gerrit.wikimedia.org/r/605653

gerritbot added a project: Patch-For-Review.Aug 4 2020, 12:40 PM
hashar moved this task from Should be empty (use Release-Engineering-Team) to Blocked externally on the Release-Engineering-Team-TODO board.Aug 4 2020, 12:40 PM
fgiunchedi triaged this task as Medium priority.Aug 17 2020, 8:32 AM
gerritbot added a comment.Feb 5 2021, 1:11 PM

Change 605653 abandoned by Hashar:
[operations/docker-images/production-images@master] python-build: reuse previously built wheels

Reason:
Will revive / redo if I ever had to rebuild wheels for Zuul

https://gerrit.wikimedia.org/r/605653

hashar closed this task as Declined.Feb 5 2021, 1:12 PM

Will revive / redo if I ever had to rebuild wheels for Zuul

hashar reopened this task as Open.Sep 28 2022, 11:47 AM
gerritbot added a comment.Sep 28 2022, 12:18 PM

Change 605653 restored by Hashar:

[operations/docker-images/production-images@master] python-build: reuse previously built wheels

https://gerrit.wikimedia.org/r/605653

thcipriani assigned this task to hashar.Oct 28 2022, 3:00 PM
thcipriani lowered the priority of this task from Medium to Low.
thcipriani edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team-TODO, Release-Engineering-Team (CI & Testing services).
thcipriani subscribed.

@hashar is this waiting on review, or are you making changes to your patch?

gerritbot added a comment.Nov 9 2022, 7:58 AM

Change 605653 abandoned by Hashar:

[operations/docker-images/production-images@master] python-build: reuse previously built wheels

Reason:

I have reopened this while pairing with Clément to update docker-pkg. We eventually found out that most dependencies on Pypi have wheels published which are downloaded directly. So it is probably less of a problem nowadays (short of redownloading wheels each time).

https://gerrit.wikimedia.org/r/605653

hashar closed this task as Declined.Nov 9 2022, 9:11 AM

Most dependencies on Pypi now provides wheels and our build download them. It is not really optimal but it is good enough. Maybe I will revisit this again later if I need it, meanwhile I am declining the task.

hashar mentioned this in T342346: Refresh integration/zuul/deploy to work on Debian Bullseye.Jul 20 2023, 1:30 PM
hashar reopened this task as Open.Jul 20 2023, 1:37 PM
hashar added a parent task: T342346: Refresh integration/zuul/deploy to work on Debian Bullseye.

To upgrade the OS on CI servers Buster → Bullseye (T334517) I have the use case of polishing up the Zuul deployment Makefile. I revisited the previous Gerrit changes I have opened and eventually figured out a system based on using PIP_FIND_LINKS to reuse previous build artifacts.

gerritbot added a comment.Jul 20 2023, 2:41 PM

Change 606242 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Reuse previous artifacts as a wheel cache

https://gerrit.wikimedia.org/r/606242

gerritbot added a comment.Jul 25 2023, 2:45 PM

Change 606242 merged by jenkins-bot:

[integration/zuul/deploy@master] Reuse previous artifacts as a wheel cache

https://gerrit.wikimedia.org/r/606242

gerritbot added a comment.Feb 8 2024, 2:32 PM

Change 998932 had a related patch set uploaded (by Hashar; author: Hashar):

[All-Projects@refs/meta/config] Add LFS config for integration/zuul/deploy

https://gerrit.wikimedia.org/r/998932

gerritbot added a comment.Feb 8 2024, 2:33 PM

Change 998932 merged by Hashar:

[All-Projects@refs/meta/config] Add LFS config for integration/zuul/deploy

https://gerrit.wikimedia.org/r/998932

gerritbot added a comment.Feb 8 2024, 2:34 PM

Change 998933 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Track wheels individually

https://gerrit.wikimedia.org/r/998933

gerritbot added a comment.Feb 13 2024, 2:38 PM

Change 1002997 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: default to run as nobody from /deploy

https://gerrit.wikimedia.org/r/1002997

gerritbot added a comment.Feb 13 2024, 2:52 PM

Change 1003002 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: add make and virtualenv

https://gerrit.wikimedia.org/r/1003002

gerritbot added a comment.Feb 14 2024, 8:50 AM

Change 940157 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: set date of source files in the wheels

https://gerrit.wikimedia.org/r/940157

gerritbot added a comment.Feb 16 2024, 11:59 AM

Change 940157 merged by Clément Goubert:

[operations/docker-images/production-images@master] python-build: set date of source files in the wheels

https://gerrit.wikimedia.org/r/940157

gerritbot added a comment.Feb 19 2024, 12:47 PM

Change 998933 merged by jenkins-bot:

[integration/zuul/deploy@master] Track wheels individually

https://gerrit.wikimedia.org/r/998933

hashar added a comment.Feb 22 2024, 8:56 AM

That has been solved by setting PIP_FIND_LINKS to point to the directory holding the existing wheels. I also wanted the wheels to be more or less reproducible which got done by setting SOURCE_DATE_EPOCH (Change 940157).

Ideally those tricks should be "upstreamed" from the integration/zuul/deploy toward the python-build images. Possibly by having the build image to generate the Makefile in addition of the wheels.

hashar closed this task as Resolved.May 17 2024, 3:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits