Page MenuHomePhabricator
Create Task
Maniphest T125760

Create apt repo for MediaWiki-Vagrant to pull packages from
Closed, ResolvedPublic
Actions

Description

MediaWiki-Vagrant uses the WMF's apt repository to get most packages, but there are some roles that use software that for one reason or another is not in use on the WMF servers. Getting arbitrary packages imported into the WMF apt repo is a difficult (and often undesirable) process. To work around this there are roles that install software from various 3rd party apt repos or even resort to downloading source tarballs and compiling within the VM.

Introducing an apt repo dedicated to packages desired by MediaWiki-Vagrant would be easier to manage and also could provide some additional level of security for MediaWiki-Vagrant users. @yuvipanda mentioned that creating such a system should be possible and that there is a bit of a precedent in the apt repo which exists to support Tool Labs.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add mwv-apt.wmflabs.orgmediawiki/vagrantmaster+15 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bd808 created this task.Feb 4 2016, 1:58 AM
bd808 raised the priority of this task from to Needs Triage.
bd808 updated the task description. (Show Details)
bd808 added a project: MediaWiki-Vagrant.
bd808 added subscribers: bd808, yuvipanda.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 4 2016, 1:58 AM
bd808 added a comment.Feb 4 2016, 1:59 AM

T124996: Introduce Node 4.2 into MediaWiki-Vagrant and T54302: Varnish vagrant role are examples of roles/modules that could benefit from this.

• mobrovac awarded a token.Feb 4 2016, 2:20 AM
• mobrovac subscribed.Feb 4 2016, 2:22 AM

I second this idea. For example, for T124996 it is currently not possible to import Node 4 packages into our prod APT repo since that could break services that still run on Trusty and Node 0.10 in prod, most notably Parsoid, CXServer and possibly Etherpad and a bunch of others I'm not aware of or have deliberately forgotten.

bd808 added a comment.Feb 4 2016, 3:06 AM

Things that could be in apt:

yuvipanda mentioned this in T125884: Create a labs project for apt repository for mediawiki-vagrant installs.Feb 4 2016, 8:57 PM
yuvipanda claimed this task.Feb 4 2016, 8:59 PM

I'll setup the aptly setup, and then someone else can take over the policy of who gets access and how packages are maintained :)

bd808 added a subtask: T125884: Create a labs project for apt repository for mediawiki-vagrant installs.Feb 4 2016, 9:11 PM
yuvipanda added a comment.Feb 4 2016, 9:12 PM

I've setup it on mwv-apt-01 instance, with aptly (see wikitech.wikimedia.org/wiki/Aptly for docs). You can check it locally by adding the following line to your MWV install:

deb [trusted=yes] http://mwv-apt.wmflabs.org/repo trusty-mwv-apt main

We can get rid of the [trusted=yes] once we figure out signing.

yuvipanda added a comment.Feb 4 2016, 9:13 PM

I've imported the node package from nodesource into it.

yuvipanda added a comment.Feb 4 2016, 9:46 PM

Ok, so it supports https, so maybe we can skip gpg signing? :D

scfc subscribed.EditedFeb 5 2016, 3:50 PM

What I like about Ubuntu's PPAs (I have not checked whether they allow accounts created by groups) is that the binary package is recreated from scratch every time, thus ensuring that it can be reproduced, and that the source files are "bundled" with the binary packages. With aptly, AFAIK you cannot store sources in any way, and you have to have a convention how binary packages are produced, i. e. strictly only unchanged checkouts from some master branch, a pbuilder environment, etc., and everyone must abide by that.

yuvipanda removed yuvipanda as the assignee of this task.Feb 16 2016, 11:17 PM

I think we can skip gpg signing for this one. Unassigning from myself so someone involved with the MediaWiki-Vagrant project can take-over to co-ordinate.

I think we should have some rules as to what packages are welcome here - possibly only packages that are directly imported from an upstream source (which could be wikimedia?)!

bd808 mentioned this in T121326: Pywikibot's requirements.txt needs pip 6+ which is not present on Ubuntu Trusty.Mar 10 2016, 8:40 PM
yuvipanda unsubscribed.Jun 30 2016, 2:01 PM
bd808 added a comment.Jun 30 2016, 4:46 PM

Example of backporting a package from jessie to trusty using this repo:

  1. Download the orig tar and debian tar from packages.debian.org
  2. Untar *.orig.tar.gz
  3. cd orig
  4. Untar ../*.debian.tar.gz
  5. dch --local ~wmf+ --distribution trusty "Rebuild for trusty"
  6. dpkg-buildpackage -us -uc
  7. cd ..
  8. scp *.deb mwv-apt-01.mwv-apt.eqiad.wmflabs:
  9. ssh mwv-apt-01.mwv-apt.eqiad.wmflabs
  10. sudo aptly repo add trusty-mwv-apt $DEB
  11. sudo aptly publish --skip-signing update trusty-mwv-apt
gerritbot subscribed.Jun 30 2016, 7:22 PM

Change 296804 had a related patch set uploaded (by BryanDavis):
Add mwv-apt.wmflabs.org

https://gerrit.wikimedia.org/r/296804

gerritbot added a project: Patch-For-Review.Jun 30 2016, 7:22 PM
gerritbot added a comment.Jul 6 2016, 5:26 PM

Change 296804 merged by jenkins-bot:
Add mwv-apt.wmflabs.org

https://gerrit.wikimedia.org/r/296804

bd808 closed this task as Resolved.Jul 11 2016, 5:28 PM
bd808 claimed this task.

I'm going to mark this as resolved now. We have a Labs VM that is hosting an aptly repo and it is wired into the MediaWiki-Vagrant Puppet process. Adding a package to the repo is documented in T125760#2418707 and follows the normal aptly process.

As for what to put in this repo, I think we can take it on a case by case basis. MediaWiki-Vagrant is much more lax than production or even Labs. Its primary utility will probably be for making WMF managed debs available to MediaWiki-Vagrant before they are ready to be put in the main Wikimedia repo.

Restricted Application added a project: User-bd808. · View Herald TranscriptJul 11 2016, 5:28 PM
bd808 moved this task from To Do to Archive on the User-bd808 board.Jul 19 2016, 11:41 PM
Tgr mentioned this in T367542: Cloud VPS "mwv-apt" project Buster deprecation.Jul 10 2024, 10:01 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 10 2024, 10:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits