Page MenuHomePhabricator

[Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs
Open, MediumPublic

Description

Make usage of Toolforge more familiar to users of commercial Platform as a Service (PaaS) products (Heroku, DigitalOcean, Google App Engine, etc) by implementing a Container based PaaS product that provides "build pack" and "push to deploy" workflows for deploying and scaling Tool owned processes.

  • roughly estimated 9-12 month project
  • initial: 1 root, 1 SWE, .25 tech writer, .25 community liaison
  • sustaining: .5 root, .5 SWE, .10 tech writer, .10 community liaison

https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_push_to_deploy

Design Documentation: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Buildpack_Implementation

Related Objects

StatusSubtypeAssignedTask
Opendcaro
DeclinedNone
DeclinedNone
DeclinedBstorm
OpenNone
Resolvedbd808
Resolvedbd808
Resolvedmmodell
Resolvedmmodell
Resolvedbd808
Resolved dpatrick
Resolvedbd808
Resolvedmmodell
Resolvedjcrespo
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Opendcaro
Opendcaro
ResolvedBstorm
Opendcaro
In Progressdcaro
ResolvedAndrew
In Progressdcaro
OpenNone
OpenNone
Resolveddcaro
OpenNone
Declineddduvall
ResolvedLegoktm
OpenNone
DuplicateNone
OpenNone
OpenNone
ResolvedLegoktm
OpenNone
ResolvedLegoktm
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone

Event Timeline

Vvjjkkii renamed this task from [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs to 78caaaaaaa.Jul 1 2018, 1:11 AM
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from 78caaaaaaa to [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs.Jul 1 2018, 3:03 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.

Oh hey, Heroku and friends are actually making this easier https://buildpacks.io/

As an update, I'm now working on this for the Cloud Services team. We're starting with a PoC of having a tool running using a buildpack-based image rather than one of the fixed ones: T265681: Deploy a tool using a buildpack-based image on Toolforge. Then will come setting up a CD pipeline (T265684: Figure out CI solution for building buildpack-based images for Toolforge) and then hooking it up to a webhook endpoint of some kind.

Started working on design documentation to organize the project and get things moving.

Change 683415 had a related patch set uploaded (by Bstorm; author: Bstorm):

[labs/tools/registry-admission-webhook@master] toolsbeta: For testing, I'd rather not upload tekton to the internal reg

https://gerrit.wikimedia.org/r/683415

Change 683415 merged by jenkins-bot:

[labs/tools/registry-admission-webhook@master] toolsbeta: For testing, I'd rather not upload tekton to the internal reg

https://gerrit.wikimedia.org/r/683415

Does the push-to-deploy pipeline accept early adopters? I’d gladly volunteer as a guinea pig.

@Sascha That would be wonderful to get feedback! What tool / language are you hoping to try out?

@nskaggs, it’s in Go, although I’m working on a web frontend that’ll eventually have a part written in JavaScript/React. Here’s my current “release process” that I’m hoping to make less manual. The environment variable GOOS=linux can be omitted when the compiler runs on a Linux machine.

git clone https://github.com/brawer/wikidata-qrank && cd wikidata-qrank
go test ./cmd/...
for tool in `ls cmd` ; do GOOS=linux go build -ldflags="-s -w" ./cmd/$tool && scp ./$tool sascha@bastion.toolforge.org:/data/project/qrank/bin/$tool ; done

Hi @Sascha, there's no timeline yet, but I'll ping you whenever the pipeline is ready to try, I'll really appreciate your input :)

We will support three main builders, paketo that supports building golang apps (you code might need some tweaking for this one), google cloud, that also supports golang, and our own toolforge one that does not support golang yet, though might in the future.

If you have any other needs please share and we'll try to accommodate for them too.