Page MenuHomePhabricator

Meeting with WMF release engineering team
Closed, ResolvedPublic2 Estimated Story Points

Description

Monday, April 27, 18:30.

Will discuss our Blubber strategy and see if it fits WMF. Will they deploy what we have now? This is especially in regard our use of software owned by third party hosted on Github and what not. Are our scripts (prepare/build) OK? Do we need to live within Gerrit? Do we trust the master branch at all times, what happens when projects are updated, do we need to review before updating? Should we run of a given commit revision?

Event Timeline

Reedy renamed this task from Meeting with WMF release engineer team to Meeting with WMF release engineering team.Apr 27 2020, 1:27 PM
kalle moved this task from 😘 Review to 🤠 This week on the User-kalle board.
kalle set the point value for this task to 2.
  • Our Blubber build scripts are fine, but it's a little problematic that we keep them in different repository than the code as they should be in sync.
    • Personal note after the meeting: It might be OK if we stay in separate Blubber repo and clone specific revision identities?
    • All services should (i.e. not must) reside in Gerrit repositories.
  • Blubber is a red herring.
    • How the Docker images are built is not important. It's the content of the Docker images.
    • As long as we use WMF base Docker images it's fine that we pull in binaries on the side, i.e. GoLang and Gradle.
    • As long as the content of on our images are open source, there is no problem. This includes third party code such as Mishkal and Mary TTS. We will not have to review their code when we merge with upstream.
    • If we don't like Blubber (but I think we do) we can take a look at https://gerrit.wikimedia.org/g/operations/docker-images/production-images.
    • Service pipeline is the next step if we continue down the Blubber path. https://wikitech.wikimedia.org/wiki/PipelineLib
    • If we have funky build requirements, consider Blubber multi stage builds. It allows to setup and build using one Docker base image and then run using another with less dependencies.
    • It's not the end of the world that our services are non secure (allowed to write to /srv/) but we need to talk to Service ops about this.
      • Several of our services are unsecure due to requiring logging to disk. If we can turn this off (in my view a bad idea for tracing errors in deployment) or redirect them to syslogd, that would be great. Doesn't seem impossible if we manage some logrotate or something instead.
  • Services will be running on Kubernetes.
    • Service ops will be able to help us understand resource allocation.
    • Personal note after the meeting: This might include automatically scaling up and down on heavy load, but we'll probably have to report back to the k8s-cluster from the services regarding that.
    • Service ops are available on Freenode IRC network.
  • We need to supply a point of contact whom the Release Engineering and Service Ops teams can get in touch with when something goes wrong.
    • When something goes wrong, fix and really make sure it doesn't happen again. It will make life so much easier for everybody involved.
kalle moved this task from 😘 Review to 🤯 Done on the User-kalle board.
  • Our Blubber build scripts are fine, but it's a little problematic that we keep them in different repository than the code as they should be in sync.
    • Personal note after the meeting: It might be OK if we stay in separate Blubber repo and clone specific revision identities?
    • All services should (i.e. not must) reside in Gerrit repositories.

If we should have Gerrit forks of these repos anyway it would be fairly simple to add our blubber code to that repo.