Related task: T214158

I thought of some more guidelines we could use to help migrate our configuration. Open to feedback.

In T264244#6548155, @nnikkhoui wrote:

Based on some trial and error, I don't believe checkbox 1 Publish WVUI prerelease to NPM as part of Jenkins CI when a new patch is merged into master can be achieved in CI as Blubber stands.

When publishing, we want to run some commands prior (e.g. the npm preversion script) which executes some git commands (e.g. git status). The user running the command (in Blubber's case, it is runuser) doesn't have permission to execute these commands, and I think that is by design. This could technically be achieved by adding runs: insecurely: true to the Blubber config, but that config options is only meant for dev environments.
It seems like we will need to run those /bin/release-* scripts locally for the time being.

My concern is that this transition step becomes a permanent step.

In T263038#6490646, @akosiaris wrote:

In T263038#6490068, @jeena wrote:

Considering the two types of images we've discussed publishing:

1. Images with dev dependencies built into them so that developers can use them to test changes or debug something. These images would be built by CI after every merge. I don’t think we need to keep around old images when a new one is published.

I don't think we 've discussed this yet, have I missed something? In this task I think we 've discussed:

• the issue of developing something (e.g. a mediawiki extension) that's dependent on an image (e.g. like termbox pointed out in T263038#6470530). This does not require dev dependencies and we already support this scenario. We could improve the UX however.
• the issue of needing images showcasing/manual QA testing. We 'll need work on this but most of it is tangential to the images themselves and it highly depends on what kind of UX (my vision is something akin to https://patchdemo.wmflabs.org/) we want to provide and use cases we want to cover. So a bigger discussion overall. But I get the idea that 1 simple tag to refer to something showcasable and somewhat (I use this very liberally, I am trying to rule out race conditions and assumptions about delays between merging and showcasing) recent makes sense in the meanwhile. But this does not require dev dependencies.
• the issue of automatic testing for patchsets. I believe the work being done on the kask integration tests already goes that way, still some work to be done and hurdles to be lifted but we are on the path already. But still, this does not require dev dependencies

I 've tried making some sense out of the 2 gerrit changes posted in the taks, but the first one was abandoned by the owner with the decision to use the pipeline images, and the second one references a blubber variant ("dev") that does not seem to exist, so I am not sure what it refers to. I get the feeling however that these 2 lines from the first patchset is what you are referring to?

RUN apt-get update && apt-get install -y "git" "build-essential" "python-dev" "librdkafka-dev" "librdkafka1" "librdkafka++1" "kafkacat" "telnet" "iputils-ping" "procps" "curl" "vim" && rm -rf /var/lib/apt/lists/*
RUN mkdir -p /srv/service/schemas/event && git clone --single-branch -- https://gerrit.wikimedia.org/r/schemas/event/primary /srv/service/schemas/event/primary && cd /srv/service/schemas/event/primary && git reset --hard 486912f && git clone --single-branch -- https://gerrit.wikimedia.org/r/schemas/event/secondary /srv/service/schemas/event/secondary && cd /srv/service/schemas/event/secondary && git reset --hard d300355

Am I correct in this assumption?

I apologize - the comment about abandoning the patch in favor of the existing pipeline images slipped by me. But to answer your question, the Dockerfile from the patchset lines you referenced above is (with some modification due to not having already cloned the eventgate repo) what would be outputted from running blubber against the development variant for that repo (https://gerrit.wikimedia.org/r/plugins/gitiles/eventgate-wikimedia/+/refs/heads/master/.pipeline/blubber.yaml)

Is this accomplished by https://gerrit.wikimedia.org/r/c/mediawiki/tools/cli/+/622790?

Considering the two types of images we've discussed publishing:

My mistake, I did not realize that the docker-override.yml file creation was already accomplished. :)

We could do task T262370 and update the instructions, while keeping @Addshore 's idea in mind for after we accomplish some of the other tasks. How does that sound?

Thanks!

Various jobenqueue errors happened today in the past 6 hours with spikes of over 10k. I didn't see any k8s deployments correlated with them.

It looks like the helm test part is failing. This wasn't being tested with helm before so I can remove that part for now.

In T263038#6470530, @Ladsgroup wrote:

• Developer productivity, For example we have a detailed section on how to set up termbox in docker (out of the box) in README of termbox: https://github.com/wikimedia/wikibase-termbox#using-official-automatically-created-images the problem is that the docker image pointed there doesn't exists anymore (and it's one-year old: docker-registry.wikimedia.org/wikimedia/wikibase-termbox:2019-08-24-040743-production. Having something for devs or users to use would be amazing

Thanks for the replies all.

Proposal was written in Google docs. I've now moved it to my user page: https://wikitech.wikimedia.org/wiki/User:Jeena_Huneidi/Complex_Dev_Environment
But really, it's been abandoned in favor of docker-compose - https://phabricator.wikimedia.org/project/view/4585/

Hi, just catching up on the comments here and clarifying that originally, the .6 version was copied into the .8 folder (talking this over later with thcipriani, we conjectured that this was because there was actually no .7 deploy)

I added php support to blubber. An example of what would be added to the variant in your blubberfile for php services:

php:
  requirements: [composer.json]
  production: true # optional for adding the --no-dev flag

In T207535#6431138, @SalixAlba wrote:

Seeing as I have been summoned by invocation of my name, I must admit a bit of confusion with the process.

Is it possible for the fix to make it into production which a bit of manual adjustment? Or will we have to wait for the automated process to work before it goes live?

I can understand the frustration.
If you want to keep updated on the automation progress, T255835 and probably T214158 are the tasks I know of at the moment.

Just want to let you know we've been working on improvements towards the direction of automated deploys :)

@Physikerwelt Hi, I believe the problem is related to the pipelinelib issue as you mentioned. What happened is that the helm chart registry changed locations, so the deployment test failed since it's still trying to get the chart from the old location. T261346 attempts to fix this by adding the registry location to PipelineLib, but we still need to update the CI configuration to reflect those changes, which is why I created https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/623480 .

@Physikerwelt sorry you got added to the change. You were added by reviewer-bot automatically, I assume since you tend to make changes to mathoid. I'll reply to your comment on [T207535#6421053]. As far as using the latest mathoid version, I'm not sure about restbase, but for CI, I've made it the default in my change to use the latest mathoid chart. If that's not desired please let me know and we can go back to using the previously specified version.

Actually I think this is a blubber thing, right?

Hi, I'm working on https://phabricator.wikimedia.org/T255835, but I've only added the ability to specify image updates for an environment in the .pipeline/config.yaml. I'm wondering if I should also add this ability for releases as well...Is it likely we'll want to have an image version override in the release values.yaml?

Failing because kube_env is located in /etc/profile.d, so only loaded for interactive shell.

We decided to add a new step to pipelinelib called 'promote' to do this task. It will clone the deployment-charts repo and run an update script to update the versions, then push a patch to gerrit.

I thought I'd post some information about how RelEng supports docker images in case it is of interest to you.

Thanks for taking a look. I should have confirmed the service-checker version!