Page MenuHomePhabricator
Create Task
Maniphest T147635

Investigate again a central cache for package managers
Closed, DeclinedPublic
Actions

Description

Package managers need to hit some cache to speed them up. We have a homegrown system based on rsync nicknamed Castor. I would like to replace it with something more robust.

Some past investigation is on T112560 and sub tasks.

Sonatype Nexus 2.x did not support npm/rubygems/pip. Version 3 does. On https://www.sonatype.com/download-oss-sonatype

Nexus Repository Manager OSS 3.xx

Manage these formats:

Docker NuGet npm Bower PyPI Ruby Gems

Related docs:

npmhttps://books.sonatype.com/nexus-book/3.0/reference/npm.html
pypihttps://books.sonatype.com/nexus-book/3.0/reference/pypi.html
gemhttps://books.sonatype.com/nexus-book/3.0/reference/rubygems.html

Wikimedia has Apache Archiva for Maven artifacts but it is not in proxy mode and I dont think it supports npm/pypi/rubygems.

There is a feature matrix that is apparently kept up-to-date by maintainers of Archiva, Artifactory, Sonatype Nexus, Eclipse Package Drone, and ProGet: https://binary-repositories-comparison.github.io/

Details

SubjectRepoBranchLines +/-
(WIP) contint: Sonatype Nexus (WIP)operations/puppetproduction+118 -0
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Oct 7 2016, 12:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2016, 12:34 PM
gerritbot subscribed.Oct 7 2016, 8:44 PM

Change 314751 had a related patch set uploaded (by Hashar):
(WIP) contint: Sonatype Nexus (WIP)

https://gerrit.wikimedia.org/r/314751

gerritbot added a project: Patch-For-Review.Oct 7 2016, 8:44 PM
hashar added a comment.Oct 7 2016, 10:04 PM

Did a basic puppet sprint and applied it on repository.integration.eqiad.wmflabs

Nexus got installed from the tar.gz

I have done a basic configuration which has a proxy for each of maven/npm/rubygems and pypi.

• mmodell awarded a token.Oct 8 2016, 1:57 AM
hashar added a comment.Oct 10 2016, 2:15 PM

I crafted a very lame puppet manifest. Sonatype provides their software as a tar.gz that has 266 .jar probably not worth trying to make a Debian package out of it.

Marko pointed out that it would be a good use case for scap3 deployment which would ship the code AND contain the setting files. So in theory we could have the instance to be a deploy server then just git pull && scap deploy to upgrade Nexus.

hashar triaged this task as Medium priority.Oct 11 2016, 8:48 AM
hashar moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.
hashar mentioned this in T106519: [Bug] github.com is 403ing downloads from Wikimedia CI during composer update.Oct 11 2016, 9:08 AM
gerritbot added a comment.Feb 17 2017, 2:06 PM

Change 314751 abandoned by Hashar:
(WIP) contint: Sonatype Nexus (WIP)

Reason:
The instance works on labs, I used with a few jenkins jobs and it works properly in the few use cases I tried.

However I don't have time to figure out how to deploy the Nexus java application itself. So abandoning until one has time to allocate to that.

https://gerrit.wikimedia.org/r/314751

Stashbot subscribed.Feb 17 2017, 2:07 PM

Mentioned in SAL (#wikimedia-releng) [2017-02-17T14:07:58Z] <hashar> integration: deleting "repository" instance. No time to figure out how to ship Sonatype Nexus to it. T147635

hashar added a comment.Feb 17 2017, 2:08 PM

The instance worked on labs, I used with a few jenkins jobs and it works properly in the few use cases I tried.

However I don't have time to figure out how to deploy the Nexus java application itself. So abandoning until one has time to allocate to that.

I have deleted the instance.

hashar changed the task status from Open to Stalled.Feb 17 2017, 2:09 PM

Stalled until we get bandwidth to figure out how to ship Sonatype Nexus in a better way than wget && tar -xvf.

greg moved this task from INBOX to Backlog on the Release-Engineering-Team board.May 20 2017, 12:15 PM
greg edited projects, added Release-Engineering-Team (Backlog); removed Release-Engineering-Team.
EBernhardson subscribed.Jul 17 2018, 11:03 PM
hashar mentioned this in T216039: jenkins / zuul backing up due to jenkins slaves down.Feb 20 2019, 2:30 PM
• Phabricator_maintenance edited projects, added Release-Engineering-Team-TODO; removed Release-Engineering-Team (Backlog).Jun 12 2019, 11:52 PM
• Phabricator_maintenance moved this task from Should be empty (use Release-Engineering-Team) to Later / Need volunteer on the Release-Engineering-Team-TODO board.Jun 12 2019, 11:55 PM
greg added a project: Release-Engineering-Team.Jun 21 2019, 10:35 PM
greg edited projects, added Release-Engineering-Team (CI & Testing services); removed Release-Engineering-Team.Jun 24 2019, 7:51 PM
hashar mentioned this in T188375: castor rsync's taking 3-5 minutes for mwgate-npm jobs.Sep 11 2019, 6:19 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 11 2019, 7:12 PM
hashar mentioned this in T234075: quibble-composer-sqlite-php72-docker does not use cache for composer downloads.Sep 27 2019, 8:45 PM
WMDE-leszek subscribed.Nov 30 2019, 11:02 AM
Aklapper added a comment.Jun 22 2020, 7:50 PM
In T147635#3036488, @hashar wrote:

Stalled until we get bandwidth to figure out how to ship Sonatype Nexus in a better way than wget && tar -xvf.

@hashar: Three years later, is this still the same situation and still wanted?

dancy changed the task status from Stalled to Open.Jul 27 2020, 7:12 PM
dancy claimed this task.
dancy added a project: User-dancy.
hashar added a project: Castor.Aug 5 2020, 5:31 PM
dancy removed dancy as the assignee of this task.Aug 10 2020, 3:51 PM
dancy subscribed.
thcipriani removed a project: Release-Engineering-Team (CI & Testing services).Apr 20 2021, 1:09 AM
thcipriani edited projects, added Release-Engineering-Team (thcipriani-workboard-fiddling); removed Release-Engineering-Team-TODO.Apr 20 2021, 3:42 AM
thcipriani moved this task from thcipriani-workboard-fiddling to Seen (ARCHIVE) on the Release-Engineering-Team board.Apr 20 2021, 3:58 AM
thcipriani edited projects, added Release-Engineering-Team; removed Release-Engineering-Team (thcipriani-workboard-fiddling).
thcipriani edited projects, added Release-Engineering-Team (Seen); removed Release-Engineering-Team.Apr 20 2021, 3:23 PM
hashar closed this task as Declined.Oct 22 2021, 1:02 PM

This is surely still wanted but there is little incentive or resource to drive the search or setup a new infrastructure. I am declining the task to reflect the reality.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits