[builds-builder] Cache .m2 folder (local maven repository) between builds
Closed, ResolvedPublic
Actions

Assigned To

Slst2020

Authored By

	Don-vip
	Nov 1 2023, 8:45 PM

Project Tags

Referenced Files

None

Subscribers

Description

Affected tool: spacemedia

I tried to do

After a successful first build (spacemedia-buildpacks-pipelinerun-56fz5) that took 7 minutes, I tried to start a new one to check that the next builds are faster.
As per documentation the .m2 folder (local maven repository) should be cached between builds, so the new build shouldn't spend ~6 minutes to download all dependencies again.

And this happened

The new build (spacemedia-buildpacks-pipelinerun-b4p75) redownloaded all dependencies and spent 7 minutes again:

Build ID: spacemedia-buildpacks-pipelinerun-b4p75
Start Time: 2023-11-01T20:14:54Z
End Time: 2023-11-01T20:21:42Z
Status: ok
Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0
Parameters:
    Source URL: https://gitlab.wikimedia.org/toolforge-repos/spacemedia
    Ref: N/A
Destination Image: tools-harbor.wmcloud.org/tool-spacemedia/tool-spacemedia:latest

But I was expecting

Previously downloaded Maven artifacts should not be downloaded again, and the build time should have been around 1 minute max.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Slst2020	T350307 [builds-builder] Cache .m2 folder (local maven repository) between builds
		Open		None	T350689 [builds-builder] Explore adding caching support

Event Timeline

Don-vip created this task.Nov 1 2023, 8:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 1 2023, 8:45 PM

Don-vip updated the task description. (Show Details)Nov 1 2023, 8:46 PM

As per documentation the .m2 folder (local maven repository) should be cached between builds, so the new build shouldn't spend ~6 minutes to download all dependencies again.

Caching builds is a feature specific to Heroku as a platform, not something that Heroku buildpacks come with as a default. Agree it would be a nice feature to have on Toolforge though.

Slst2020 renamed this task from [buildservice] Bug - .m2 folder (local maven repository) is not cached between builds to [buildservice] Cache .m2 folder (local maven repository) between builds.Nov 7 2023, 2:42 PM

Slst2020 added a parent task: T350689: [builds-builder] Explore adding caching support.

Slst2020 changed the subtype of this task from "Bug Report" to "Task".

There is "some" support for caching in tekton too (https://github.com/tektoncd/catalog/blob/main/task/buildpacks-phases/0.2/buildpacks-phases.yaml#L23), but we'll have to figure out how to do it, as it also depends on us having persistent volumes on k8s (that we do not yet) or finding some other solution (caching proxy/etc.).

taavi edited projects, added: Toolforge Build Service; removed: Toolforge Build Service (Beta release).Dec 12 2023, 7:16 PM

dcaro removed a parent task: T350689: [builds-builder] Explore adding caching support.Mar 5 2024, 10:30 AM

dcaro added a subtask: T350689: [builds-builder] Explore adding caching support.

dcaro triaged this task as Low priority.Mar 5 2024, 10:40 AM

dcaro edited projects, added: Toolforge; removed: Toolforge Build Service.

dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.Mar 12 2024, 11:05 AM

dcaro renamed this task from [buildservice] Cache .m2 folder (local maven repository) between builds to [builds-builder] Cache .m2 folder (local maven repository) between builds.Apr 3 2024, 1:28 PM

Don-vip awarded a token.Jun 2 2024, 12:43 PM

dcaro mentioned this in T369563: fagiani/apt buildpack very slow when processing a large collection of packages.Jul 30 2024, 2:28 PM

@dcaro @Slst2020 have you enabled the cache already? I switched to Toolforge Build Service tonight in order to get Java 21 and the build is really fast! I don't need anything more to increase build performance, I'm very happy with the current situation :)

In T350307#10120054, @Don-vip wrote:

@dcaro @Slst2020 have you enabled the cache already? I switched to Toolforge Build Service tonight in order to get Java 21 and the build is really fast! I don't need anything more to increase build performance, I'm very happy with the current situation :)

Hmm... not that I know! For the record, how long does the build take?

@Don-vip is this something you'd still want, or could we close this task?

Restricted Application added a project: cloud-services-team. · View Herald TranscriptNov 7 2024, 11:12 AM

Hi @Slst2020! I'm sorry I didn't see your last reply. The Maven part of the build currently takes less than a minute, and the whole build less than 2 minutes, it's perfect:

[prepare] 2024-11-07T18:38:26.782272903Z 2024/11/07 18:38:26 Entrypoint initialization
[place-scripts] 2024-11-07T18:38:27.921146994Z 2024/11/07 18:38:27 Decoded script /tekton/scripts/script-0-kwbcw
...
[step-analyze] 2024-11-07T18:38:44.398673490Z Image with name "tools-harbor.wmcloud.org/tool-spacemedia/tool-spacemedia:latest" not found
[step-detect] 2024-11-07T18:38:45.315205239Z 3 of 5 buildpacks participating
[step-detect] 2024-11-07T18:38:45.315265529Z heroku/jvm      4.0.2
[step-detect] 2024-11-07T18:38:45.315272975Z heroku/maven    4.0.2
[step-detect] 2024-11-07T18:38:45.315278298Z heroku/procfile 2.0.2
...
[step-build] 2024-11-07T18:38:45.887829421Z
[step-build] 2024-11-07T18:38:45.887931117Z [Installing OpenJDK]
[step-build] 2024-11-07T18:38:51.102257966Z
[step-build] 2024-11-07T18:38:51.102340764Z [Installing Heroku JVM metrics agent]
[step-build] 2024-11-07T18:38:51.178324443Z
[step-build] 2024-11-07T18:38:51.178394973Z [Installing Maven]
[step-build] 2024-11-07T18:38:51.178409938Z Maven wrapper detected, skipping installation.
[step-build] 2024-11-07T18:38:51.178420005Z
[step-build] 2024-11-07T18:38:51.178429013Z [Executing Maven]
[step-build] 2024-11-07T18:38:51.178439506Z $ ./mvnw -Pweb -Pversion -ntp -DskipTests -Dspring.profiles.active clean install
[step-build] 2024-11-07T18:38:51.637239507Z Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
[step-build] 2024-11-07T18:38:53.066766988Z [INFO] Scanning for projects...
[step-build] 2024-11-07T18:39:06.043472514Z [INFO]
[step-build] 2024-11-07T18:39:06.043996998Z [INFO] --------------< org.wikimedia.commons.donvip:spacemedia >---------------
[step-build] 2024-11-07T18:39:06.044104381Z [INFO] Building spacemedia 0.5.0-SNAPSHOT
[step-build] 2024-11-07T18:39:06.044152193Z [INFO]   from pom.xml
[step-build] 2024-11-07T18:39:06.044164792Z [INFO] --------------------------------[ jar ]---------------------------------
...
[step-build] 2024-11-07T18:39:43.496836298Z [INFO] ------------------------------------------------------------------------
[step-build] 2024-11-07T18:39:43.496891714Z [INFO] BUILD SUCCESS
[step-build] 2024-11-07T18:39:43.496905135Z [INFO] ------------------------------------------------------------------------
[step-build] 2024-11-07T18:39:43.497976258Z [INFO] Total time:  50.455 s
[step-build] 2024-11-07T18:39:43.498139747Z [INFO] Finished at: 2024-11-07T18:39:43Z
[step-build] 2024-11-07T18:39:43.498178616Z [INFO] ------------------------------------------------------------------------
...
[step-export] 2024-11-07T18:40:25.483616971Z Adding cache layer 'heroku/maven:repository'
[step-results] 2024-11-07T18:40:27.069597300Z Built image tools-harbor.wmcloud.org/tool-spacemedia/tool-spacemedia:latest@sha256:117d6eb31a7a591eec9bebc14e1c07497c86bf10d73e28519cc706aa372e006a

So you can close the task, performances are great :)

That's great to hear, thank you!

[builds-builder] Cache .m2 folder (local maven repository) between buildsClosed, ResolvedPublicActions