Page MenuHomePhabricator

[Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs
Open, MediumPublic

Description

Make usage of Toolforge more familiar to users of commercial Platform as a Service (PaaS) products (Heroku, DigitalOcean, Google App Engine, etc) by implementing a Container based PaaS product that provides "build pack" and "push to deploy" workflows for deploying and scaling Tool owned processes.

  • roughly estimated 9-12 month project
  • initial: 1 root, 1 SWE, .25 tech writer, .25 community liaison
  • sustaining: .5 root, .5 SWE, .10 tech writer, .10 community liaison

https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_push_to_deploy

Design Documentation: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Buildpack_Implementation

Related Objects

StatusSubtypeAssignedTask
StalledLucasWerkmeister
Stalledmatmarex
OpenNone
OpenNone
Opendcaro
DeclinedNone
DeclinedNone
DeclinedBstorm
Resolveddcaro
Resolvedbd808
Resolvedbd808
Resolvedmmodell
Resolvedmmodell
Resolvedbd808
Resolved dpatrick
Resolvedbd808
Resolvedmmodell
Resolvedjcrespo
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
ResolvedBstorm
Resolvedfnegri
ResolvedBUG REPORTdcaro
ResolvedLegoktm
ResolvedLegoktm
Opendcaro
ResolvedLegoktm
OpenNone
OpenNone
OpenNone
Resolveddcaro
In Progressdcaro
Resolveddcaro
Resolveddcaro
OpenNone
Declineddduvall
OpenNone
OpenNone
DuplicateNone
Resolveddcaro
ResolvedRaymond_Ndibe
Resolveddcaro
ResolvedSlst2020
ResolvedSlst2020
DuplicateNone
DuplicateNone
OpenNone
DuplicateNone
Resolveddcaro
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
Resolveddcaro
Resolvedfnegri
ResolvedRaymond_Ndibe
ResolvedSlst2020
ResolvedSlst2020
ResolvedKHernandez-WMF
ResolvedRaymond_Ndibe
Resolvedkomla
ResolvedRaymond_Ndibe
ResolvedSlst2020
Resolveddcaro
Resolveddcaro
Resolveddcaro
ResolvedRaymond_Ndibe
Resolveddcaro
OpenNone
StalledNone
OpenNone
OpenNone
OpenKHernandez-WMF
OpenNone
ResolvedSlst2020
OpenNone
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
ResolvedNone
Resolveddcaro
OpenNone
ResolvedRaymond_Ndibe
Stalleddcaro
ResolvedRaymond_Ndibe
ResolvedSlst2020
ResolvedSlst2020
ResolvedSlst2020
ResolvedSlst2020
ResolvedRaymond_Ndibe
ResolvedSlst2020
ResolvedSlst2020
ResolvedSlst2020
Resolveddcaro
Resolveddcaro
In Progressdcaro
Resolveddcaro
Resolveddcaro
Resolveddcaro
ResolvedRaymond_Ndibe
OpenNone
ResolvedRaymond_Ndibe
StalledRaymond_Ndibe
Resolveddcaro
ResolvedRaymond_Ndibe
In ProgressRaymond_Ndibe
Resolvedfnegri
Resolvedfnegri
OpenNone
OpenNone
ResolvedRaymond_Ndibe
Resolveddcaro
ResolvedAndrew
ResolvedSlst2020
Resolveddcaro
ResolvedRaymond_Ndibe
InvalidNone
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
Resolveddcaro
OpenNone
OpenNone
Resolveddcaro
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
OpenNone
OpenNone
OpenNone
Openkomla
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Resolveddcaro
Resolveddcaro
OpenNone
ResolvedSlst2020
ResolvedRaymond_Ndibe
ResolvedSlst2020
OpenNone
In ProgressNone
In ProgressNone
ResolvedSlst2020
OpenNone
OpenNone
Resolveddcaro
Opendcaro
In Progressdcaro
Resolveddcaro
Opendcaro
Resolveddcaro
Opendcaro
In ProgressRaymond_Ndibe
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
In ProgressRaymond_Ndibe
OpenRaymond_Ndibe
Openfnegri
Openfnegri
Openfnegri
Openfnegri
Openfnegri
In ProgressSlst2020
ResolvedSlst2020
ResolvedSlst2020
ResolvedSlst2020
In ProgressSlst2020
Opendcaro
Resolveddcaro
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Oh hey, Heroku and friends are actually making this easier https://buildpacks.io/

As an update, I'm now working on this for the Cloud Services team. We're starting with a PoC of having a tool running using a buildpack-based image rather than one of the fixed ones: T265681: [tbs.tools] Deploy a tool using a buildpack-based image on Toolforge. Then will come setting up a CD pipeline (T265684: Figure out CI solution for building buildpack-based images for Toolforge) and then hooking it up to a webhook endpoint of some kind.

Started working on design documentation to organize the project and get things moving.

Change 683415 had a related patch set uploaded (by Bstorm; author: Bstorm):

[labs/tools/registry-admission-webhook@master] toolsbeta: For testing, I'd rather not upload tekton to the internal reg

https://gerrit.wikimedia.org/r/683415

Change 683415 merged by jenkins-bot:

[labs/tools/registry-admission-webhook@master] toolsbeta: For testing, I'd rather not upload tekton to the internal reg

https://gerrit.wikimedia.org/r/683415

Does the push-to-deploy pipeline accept early adopters? I’d gladly volunteer as a guinea pig.

@Sascha That would be wonderful to get feedback! What tool / language are you hoping to try out?

@nskaggs, it’s in Go, although I’m working on a web frontend that’ll eventually have a part written in JavaScript/React. Here’s my current “release process” that I’m hoping to make less manual. The environment variable GOOS=linux can be omitted when the compiler runs on a Linux machine.

git clone https://github.com/brawer/wikidata-qrank && cd wikidata-qrank
go test ./cmd/...
for tool in `ls cmd` ; do GOOS=linux go build -ldflags="-s -w" ./cmd/$tool && scp ./$tool sascha@bastion.toolforge.org:/data/project/qrank/bin/$tool ; done

Hi @Sascha, there's no timeline yet, but I'll ping you whenever the pipeline is ready to try, I'll really appreciate your input :)

We will support three main builders, paketo that supports building golang apps (you code might need some tweaking for this one), google cloud, that also supports golang, and our own toolforge one that does not support golang yet, though might in the future.

If you have any other needs please share and we'll try to accommodate for them too.