Page MenuHomePhabricator
Create Task
Maniphest T297673

Build MediaWiki images for kubernetes on the deployment servers
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Our deployment servers is where the code for mediawiki is "prepared", including:

  • mediawiki releases
  • security patches
  • private settings
  • localization cache

using scap. We would like to allow building our images outside of CI and as an integral part of preparing the code for deployment.

To this end, I think the easiest way to go about this is:

  • Install docker on the deployment servers (check for space/filesystem constraints)
  • Install the docker-pusher wrapper to allow pushing images via sudo from a mwbuilder user
  • Install a copy of mediawiki/tools/release under /srv/mwbuilder/release
  • Create a sudo rule allowing people in the deployment-ci-admins group to run /usr/local/bin/update-mediawiki-tools-release as mwbuilder (to allow updating the tools/release source code) - this is the same group as contint-admins.
  • Create a sudo rule allowing people in the deployment group to run /usr/bin/make -C /srv/mwbuilder/release/make-container-image -f Makefile *
  • Make scap (eventually, when it will be how we fetch code in production) and/or a git hook trigger the rebuild sudo'ing to the correct user
  • Possibly write the file with the image versions to consume for deployments.
  • Reduce verbosity of image build process. It's overwhelming.
  • Measure impact of image build time (first train of the wiki and followup commits).
  • Figure out interaction with sync-dir and sync-file (possibly deprecating these commands).
  • Set build_mw_container_image to True in /etc/scap.cfg

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Joe created this task.Dec 14 2021, 7:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2021, 7:44 AM
Joe claimed this task.Dec 20 2021, 10:42 AM
Joe triaged this task as Medium priority.
Legoktm mentioned this in T298165: mediawiki-multiversion image builder should also poll private and security patches git repositories.Dec 21 2021, 10:32 PM
gerritbot added a comment.Dec 22 2021, 9:05 AM

Change 749508 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] deployment_server: add docker engine

https://gerrit.wikimedia.org/r/749508

gerritbot added a project: Patch-For-Review.Dec 22 2021, 9:05 AM
gerritbot added a comment.Dec 22 2021, 4:19 PM

Change 749552 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] profile:k8s::deployment_server::mediawiki: split in subprofiles

https://gerrit.wikimedia.org/r/749552

gerritbot added a comment.Dec 22 2021, 4:19 PM

Change 749553 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] kubernetes::deployment_server::mediawiki: add builder user/role

https://gerrit.wikimedia.org/r/749553

gerritbot added a comment.Jan 3 2022, 10:25 AM

Change 749508 merged by Giuseppe Lavagetto:

[operations/puppet@production] deployment_server: add docker engine

https://gerrit.wikimedia.org/r/749508

gerritbot added a comment.Jan 3 2022, 11:39 AM

Change 749552 merged by Giuseppe Lavagetto:

[operations/puppet@production] profile:k8s::deployment_server::mediawiki: split in subprofiles

https://gerrit.wikimedia.org/r/749552

gerritbot added a comment.Jan 3 2022, 2:04 PM

Change 749553 merged by Giuseppe Lavagetto:

[operations/puppet@production] kubernetes::deployment_server::mediawiki: add builder user/role

https://gerrit.wikimedia.org/r/749553

Maintenance_bot removed a project: Patch-For-Review.Jan 3 2022, 2:10 PM
Joe updated the task description. (Show Details)Jan 3 2022, 3:00 PM
gerritbot added a comment.Jan 3 2022, 4:36 PM

Change 751166 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] deployment_server: fix permissions for mwbuilder/other

https://gerrit.wikimedia.org/r/751166

gerritbot added a project: Patch-For-Review.Jan 3 2022, 4:36 PM
gerritbot added a comment.Jan 4 2022, 4:20 PM

Change 751166 merged by Giuseppe Lavagetto:

[operations/puppet@production] deployment_server: fix permissions for mwbuilder/other

https://gerrit.wikimedia.org/r/751166

Joe removed Joe as the assignee of this task.Jan 4 2022, 4:28 PM
Joe updated the task description. (Show Details)

The deployment servers should now be able to build the images for mediawiki. I'll de-assign the task from myself for now, please ping me If I'm needed for the next steps!

Maintenance_bot removed a project: Patch-For-Review.Jan 4 2022, 5:10 PM
thcipriani edited projects, added Release-Engineering-Team (Done by Feb 23 🧟); removed Release-Engineering-Team.Feb 9 2022, 5:59 PM
thcipriani set the point value for this task to 3.
dancy claimed this task.Feb 11 2022, 9:53 PM
dancy moved this task from Backlog to Priority Backlog 💡 on the Release-Engineering-Team (Done by Feb 23 🧟) board.
dancy moved this task from Priority Backlog 💡 to In progress on the Release-Engineering-Team (Done by Feb 23 🧟) board.
gerritbot added a comment.Feb 11 2022, 10:55 PM

Change 762007 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/scap@master] build container image during scap sync-*

https://gerrit.wikimedia.org/r/762007

gerritbot added a project: Patch-For-Review.Feb 11 2022, 10:55 PM
gerritbot added a comment.Feb 14 2022, 10:05 PM

Change 762007 abandoned by Ahmon Dancy:

[mediawiki/tools/scap@master] Optionally build mw container image during scap sync-*

Reason:

look at https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/762546 instead

https://gerrit.wikimedia.org/r/762007

gerritbot added a comment.Feb 14 2022, 10:05 PM

Change 762546 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/scap@master] Optionally build mw container image during scap sync-*

https://gerrit.wikimedia.org/r/762546

gerritbot added a comment.Feb 15 2022, 6:25 PM

Change 762546 merged by jenkins-bot:

[mediawiki/tools/scap@master] Optionally build mw container image during scap sync-*

https://gerrit.wikimedia.org/r/762546

gerritbot added a comment.Feb 18 2022, 7:03 PM

Change 763808 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/release@master] Mods to webserver image build for scap

https://gerrit.wikimedia.org/r/763808

gerritbot added a comment.Feb 18 2022, 7:06 PM

Change 763808 merged by jenkins-bot:

[mediawiki/tools/release@master] Mods to webserver image build for scap

https://gerrit.wikimedia.org/r/763808

gerritbot added a comment.Feb 18 2022, 9:36 PM

Change 763846 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/train-dev@master] Kill off build-mw-image-loop.py

https://gerrit.wikimedia.org/r/763846

gerritbot added a comment.Feb 18 2022, 10:11 PM

Change 763609 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/scap@master] Add release_repo_build_and_push_images_cmd config parameter

https://gerrit.wikimedia.org/r/763609

Joe mentioned this in T302213: Give deployers access to build-mw-container-image Jenkins job.Feb 21 2022, 11:43 AM
dancy added a comment.Feb 23 2022, 7:40 PM

@Joe I tried running this today but it failed:

dancy@deploy1002$ sudo -u mwbuilder /usr/bin/make -C /srv/mwbuilder/release/make-container-image -f Makefile build-and-push-all-images \
GIT_BASE=https://gerrit.wikimedia.org/r/ BRANCH=master workdir_volume=/srv/mediawiki-staging mv_image_name=docker-registry.discovery.wmnet/restricted/mediawiki-multiversion webserver_image_name=docker-registry.discovery.wmnet/restricted/mediawiki-webserver in /srv/mwbuilder/release/make-container-image
make: Entering directory '/srv/mwbuilder/release/make-container-image'
/usr/bin/make -C multiversion-base
make[1]: Entering directory '/srv/mwbuilder/release/make-container-image/multiversion-base'
docker build \
        --pull \
        --build-arg "http_proxy=" \
        --build-arg "https_proxy=" \
        --build-arg "MV_BASE_PACKAGES=" \
        --build-arg "MV_EXTRA_CA_CERT=" \
        -t multiversion-base .
ERRO[0000] failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: permission denied
context canceled

It looks like the mwbuilder is not in the docker group.

dancy@deploy1002$ id mwbuilder
uid=493(mwbuilder) gid=495(mwbuilder) groups=495(mwbuilder)
gerritbot added a comment.Feb 23 2022, 7:44 PM

Change 763609 merged by jenkins-bot:

[mediawiki/tools/scap@master] Add release_repo_build_and_push_images_cmd config parameter

https://gerrit.wikimedia.org/r/763609

gerritbot added a comment.Feb 23 2022, 7:44 PM

Change 763846 merged by jenkins-bot:

[mediawiki/tools/train-dev@master] Kill off build-mw-image-loop.py

https://gerrit.wikimedia.org/r/763846

dancy changed the task status from Open to In Progress.Feb 23 2022, 10:04 PM
dancy raised the priority of this task from Medium to High.
dancy added a subtask: T302464: Deploy Scap version 4.4.1.Feb 23 2022, 11:47 PM
Joe added a comment.Feb 24 2022, 11:25 AM
In T297673#7733295, @dancy wrote:

@Joe I tried running this today but it failed:

dancy@deploy1002$ sudo -u mwbuilder /usr/bin/make -C /srv/mwbuilder/release/make-container-image -f Makefile build-and-push-all-images \
GIT_BASE=https://gerrit.wikimedia.org/r/ BRANCH=master workdir_volume=/srv/mediawiki-staging mv_image_name=docker-registry.discovery.wmnet/restricted/mediawiki-multiversion webserver_image_name=docker-registry.discovery.wmnet/restricted/mediawiki-webserver in /srv/mwbuilder/release/make-container-image
make: Entering directory '/srv/mwbuilder/release/make-container-image'
/usr/bin/make -C multiversion-base
make[1]: Entering directory '/srv/mwbuilder/release/make-container-image/multiversion-base'
docker build \
        --pull \
        --build-arg "http_proxy=" \
        --build-arg "https_proxy=" \
        --build-arg "MV_BASE_PACKAGES=" \
        --build-arg "MV_EXTRA_CA_CERT=" \
        -t multiversion-base .
ERRO[0000] failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: permission denied
context canceled

It looks like the mwbuilder is not in the docker group.

dancy@deploy1002$ id mwbuilder
uid=493(mwbuilder) gid=495(mwbuilder) groups=495(mwbuilder)

yes I made the mistake to think we just needed docker-pusher, which is not all we need. Let me amend that.

gerritbot added a comment.Feb 24 2022, 11:38 AM

Change 765494 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] deployment_server: re-add mwbuilder to the docker group

https://gerrit.wikimedia.org/r/765494

gerritbot added a comment.Feb 24 2022, 11:41 AM

Change 765494 merged by Giuseppe Lavagetto:

[operations/puppet@production] deployment_server: re-add mwbuilder to the docker group

https://gerrit.wikimedia.org/r/765494

Joe added a comment.Feb 24 2022, 11:44 AM

aaand should be fixed now.

dancy added a comment.Feb 24 2022, 3:31 PM
In T297673#7734748, @Joe wrote:

aaand should be fixed now.

Confirmed. Thanks @Joe!

gerritbot added a comment.Feb 24 2022, 9:57 PM

Change 765624 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap.cfg.erb: Add container image build settings

https://gerrit.wikimedia.org/r/765624

dancy updated the task description. (Show Details)Feb 24 2022, 10:52 PM
dancy updated the task description. (Show Details)Feb 24 2022, 11:02 PM
dancy changed the status of subtask T302464: Deploy Scap version 4.4.1 from Open to Stalled.Feb 28 2022, 6:02 PM
dancy changed the status of subtask T302464: Deploy Scap version 4.4.1 from Stalled to Open.Feb 28 2022, 7:25 PM
Joe changed the status of subtask T302464: Deploy Scap version 4.4.1 from Open to In Progress.Mar 1 2022, 6:46 AM
gerritbot added a comment.Mar 1 2022, 5:20 PM

Change 765624 merged by RLazarus:

[operations/puppet@production] scap.cfg.erb: Add container image build settings

https://gerrit.wikimedia.org/r/765624

Joe closed subtask T302464: Deploy Scap version 4.4.1 as Resolved.Mar 2 2022, 7:10 AM
dancy updated the task description. (Show Details)Mar 2 2022, 4:22 PM
dancy updated the task description. (Show Details)Mar 3 2022, 7:31 PM
dancy updated the task description. (Show Details)Mar 3 2022, 7:36 PM
gerritbot added a comment.Mar 8 2022, 7:21 PM

Change 769097 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/scap@master] Redirect image build output to a file

https://gerrit.wikimedia.org/r/769097

gerritbot added a comment.Mar 11 2022, 1:56 PM

Change 769097 merged by jenkins-bot:

[mediawiki/tools/scap@master] Redirect image build output to a file

https://gerrit.wikimedia.org/r/769097

taavi mentioned this in T303450: Add some users to the docker group on deployment servers.Mar 15 2022, 5:58 PM
dancy updated the task description. (Show Details)Mar 15 2022, 6:16 PM
thcipriani edited projects, added Release-Engineering-Team (🌱 Spring Cleaning — April 2022); removed Release-Engineering-Team (Done by Feb 23 🧟).Apr 4 2022, 8:53 PM
thcipriani moved this task from Backlog to In progress on the Release-Engineering-Team (🌱 Spring Cleaning — April 2022) board.Apr 4 2022, 8:54 PM
thcipriani edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team (🌱 Spring Cleaning — April 2022).Sep 7 2022, 3:40 PM
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:19 PM
dancy added a comment.Dec 16 2022, 7:31 PM

Today I disabled the build-mw-container-image and build-webserver-image jobs on https://releases-jenkins.wikimedia.org/.

thcipriani updated the task description. (Show Details)Dec 19 2022, 4:25 PM
thcipriani closed this task as Resolved.Mar 7 2023, 5:05 PM
thcipriani subscribed.

I believe this is happening now. Just noticed that this task was lingering.

Maintenance_bot removed a project: Patch-For-Review.Mar 7 2023, 5:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL