https://www.mediawiki.org/wiki/Requests_for_comment/Streamlining_Composer_usage
Next steps in any order:
T125343: Upgrade integration/composer to 1.6.5 stable
Create a CI job that creates an update to a branch including a build of vendor (T101123) that can be deployed to beta which is similar to what for production a new wmf branch normal is done for. See the work flow example in the RFC.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T101123 automatically build and commit mediawiki/vendor (composer) | |||
| Declined | None | T75940 Stop using an older release of SMW and get back to tracking their master branch | |||
| Declined | None | T105638 RFC: Streamlining Composer usage |
Currently I don't have enough time to catch up to the existing input. So I think we should wait with discussing this RFC again.
A message to all open tasks related to the #Wikimania-Hackathon-2015. What do you need to complete this task? Do you need support from the Wikimedia Foundation to push it forward? Help promoting this project? Finding an intern to work on it? Organizing a developer sprint? Pitching it to WMF teams? Applying for a grant? If you need support, share your request at T107423: Evaluate which projects showcased at the Wikimania Hackathon 2015 should be supported further or contact me personally. Thank you!
My preferred way forward here is adding signature support to composer and packagist.org, that would if the signature implementation is not broken and the persons trusted with signing are not compromised prevent all from T105638#1515362 except 5.
Does it make sense to discuss this RfC in the context of the Wikimedia-Developer-Summit-2016? See https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016#Call_for_participation
It might, if I have time to write up a sketch/plan how to implement this in composer upstream and if the right people are interested and there.
The right people would be the people doing the branch cut before deployment, doing the deployment, creating composer components, using composer components in Mediawiki.
Thank you for proposing a session for the Wikimedia Developer Summit. Please complete your description -- see "Expected fields" at https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016#Call_for_participation
@Qgil as far as I see it is complete, except that I need to update the RFC document.
definition of the problem
See RFC.
expected outcome at the Summit
Progress on the RFC. It would be nice if the WMF would make it a goal, but that is not strictly an expected outcome.
current status of the discussion
See RFC. But needs update.
links to background information
See this tasks blocker and the RFC.
related tasks in Phabricator
See this tasks blocker and the RFC.
Congratulations! This is one of the 52 proposals that made it through the first deadline of the Wikimedia-Developer-Summit-2016 selection process. Please pay attention to the next one: > By 6 Nov 2015, all Summit proposals must have active discussions and a Summit plan documented in the description. Proposals not reaching this critical mass can continue at their own path out of the Summit.
@Qgil I haven't seen any response from you, so I took the liberty to change the column on the work board.
We held E85: RFC Meeting: Streamlining Composer usage (2015-11-04), see Meeting summary. From that:
Here is an idea for a workflow-based solution that would work for nodejs as well:
This workflow is very close to what we are currently doing for node services, except that step 2) is currently performed manually, using a docker script.
@GWicke: very interesting suggestion. I like it, though I think I need to think it over a bit more before I can say for sure that it would work for mediawiki deployments.
TODO: See if something better than plain gpg1/2 verify exists and if git verify can be enhanced to verify if the key that made the signature is trusted, e.g. to have less of the implementation in composer and make it possible to share this with other package managers.
The librarization project is somewhat blocked on this (or rather on the Composer security problems outlined here), see comments in https://gerrit.wikimedia.org/r/#/c/248661/
Copy of the etherpad:
(missing start of talk, a minute or so) Step 4: CI job that automatically triggers after changes to core or ext * run composer update and push changes Step 5: Update BC from steps 3-4 Are these sufficient examples? Can people imagine difficulties? * If files are added to vendor, how do we deal with this in code review/security patches? ** Deployed as a security patch * We would keep this strategy, no automatic composer run Questions about the RfC in general? * Bryan: one of the issues we got to when discussing this last was which use case is being optimized * Relying on GPG tags for repos doesn't cover many vendor things ** System that Jan is proposing is a Debian style web of trust to composer usage where we rely on GPG hashes to authenticate code ** For composer things that aren't in that web of trust there would need to be a legacy system. Maybe run mirrors. Is that least sane? ** There's an inherent problem with trust if you're downloading vendor packages (?) If we want to make sure we're only using code we intended to, we need to sign it * Is there an open upstream issue for this? ** Basic sentiment is that composer is not designed to support ** all issues regarding verification/authentication of packages are lumped together *** a lot of ideas and no direction *** Some like Jan have given a proof of concept but it went nowhere ** Not a lot of interest upstream There are components already that are signed * e.g. phpunit We would need to sign some vendor components but most are ours already Can we just run some proxy for upstream components? * Same problems with verification What do you do now? * I don't update vendor. Main driver use case is WikiData/WikiBase * double code review problems Problem with moving to composer components is the overhead of more work * much easier to develop a new MW ext than to create a component * making this easier is key to adopting new conventions Counter-proposal was drop GPG * From security perspective would be insufficient. Jan wouldn't want to advertise this solution Was proposal to start doing tag signing? What? * Automate vendor updates using signed git tags, modifying composer ** control of keys etc. * Would it bea Jenkins job? Yes, updates to core or anything that has composer.json ** Possibly before BC deployment There are packages we don't control and fundamental rewrite of composer * It's not a total rewrite. There are a few places to change to support this (around finishing of download; introduce verification) ** There are problems current with the implementation of verification in composer (succeeds when it shouldn't) Getting support into composer is hardest part but signing should be easier * Signing might be easier for repos we control but not 3rd parties * We can trust 3rd party keys or make clone (mirror) of repos and add signing following security review * e.g., phpunit ** on WMF servers we'd maintain clone of source repo ** sign it ourselves (after review?) ** in MW vendors composer.json use stanza for copying it from our mirror We could package? ** When debian packager packages something (e.g. phpunit), where do they get the code form? *** Disconnect. You can verify at the point of installation once or have to verify it every time. Verifying every time has more potential for man-in-the-middle attacks We have these same problems for NPM modules in *oids * You find crazy things like libssl in vendor Python/pip as well. * Verification isn't required unless you tell it In packaging (Debian for example) * Every package you install has been verified ** Biggest problems are that packages are old, or don't exist Jan is trying to solve WMF specific problem * What does he need? ** Someone to build it. From security perspective, what are you solving? * Attack vector? * Requiring HTTPS and checking validity of cert is a start * Verifying packages is a step further. Requires more work in signing * Starting with a patched version of composer that requires HTTPS Tags are a terrible thing to trust. Trust only SHA1s For everything you're going to pull in from composer, fork/mirror
Steps mentioned at the beginning of the session are https://www.mediawiki.org/wiki/Requests_for_comment/Streamlining_Composer_usage#Workflow_example .
Wikimedia Developer Summit 2016 ended two weeks ago. This task is still open. If the session in this task took place, please make sure 1) that the session Etherpad notes are linked from this task, 2) that followup tasks for any actions identified have been created and linked from this task, 3) to change the status of this task to "resolved". If this session did not take place, change the task status to "declined". If this task itself has become a well-defined action which is not finished yet, drag and drop this task into the "Work continues after Summit" column on the project workboard. Thank you for your help!
T133995#2254431 @adrianheine I upgraded composer in our CI. Next T125343 we probably need to figure out the target environment .
Adjusting priority to what I perceive to be the actual demand. Sorry, overestimated the amount of my free time.
This seems worth discussing in the ArchCom RFC triage process (discussion: Z425 or wikitech-l)
We discussed this at E227 last week, and @Krinkle suggested a more substantive response to @csteipp's T105638#1515362 review would be helpful.
This also needs a shepherd to move forward. @daniel, is that something you can take on?
Is this blocked on something specific or just lack of interest? As I understood the original blocker was proper HTTPS support in Composer and that was fixed upstream some months ago.
Lack of commitment, lack of resources. I still think it would be a good idea to do it, but i'm not sure that counts as "interest".
We should get rid of mediawiki/vendor at some point...
Just going to poke this to see what people consider the state of this?
Personally I think out composer usage is rather streamlined right now (except in core we have super strict version dependencies)
Indeed. Our conventions around how Composer should be used in MediaWiki development, release and deployment have settled. I don't think we're in an ideal or perfect situation but it seems to be working well enough right now and meets the original criteria for adopting Composer
However, the RFC about agreeing how to use Composer (initially) is a separate one:https://www.mediawiki.org/wiki/Requests_for_comment/Composer_managed_libraries_for_use_on_WMF_cluster
Despite its generic name, this RFC (T105638) was not so much about setting up how we use Composer, but changing how we use Composer to be different. Specifically to address complexity with regards to merging composer.json files from multiple sources whilst still making it easy and consistent to test and install separately. This was mainly for the use case of the Wikidata build, which is being changed to not have its own vendor install anymore, thus making this an RFC without a shepherd/stakeholder.
The idea of automating mediawiki/vendor and/or to move the Composer step to be during CI or deployment, instead of in Git, remains interesting for the future. But following a reevaluation from TechCom, I'm closing this RFC given the original use case is becoming obsolete, and there is no longer resourcing from the affected teams for this problem.
Affected teams (e.g. Release Engineering or MediaWiki Platform Team) are free to create a new RFC at any time, possibly borrowing some of the ideas from this RFC.