Had some time to play with this today.

I added the small blurb of documentation that I wish I had when attempting to resolve T203246.

I was able to generate the image docker-registry.wikimedia.org/wikimedia/mediawiki-services-zotero:20181019165254-production which runs, and listens on port 1969. Not sure how to exercise the image. Also, we'll need to add a .pipeline/helm.yaml chart to point it to the correct helm chart whenever that is finalized. For now, I've triggered a pipeline build that skips helm test to create the above image.

Adding @mmodell

I don't see a service_name in /srv/deployment/proton/deploy/.git/DEPLOY_HEAD. Currently service_name is commented out in /srv/deployment/proton/deploy/scap/scap.cfg.

Seems like this is something we could do in a few different ways. We could probably work out a way to do this in blubber using a variant. We could create an intermediate docker-pkg image.

In T206276#4670619, @GTirloni wrote:

Executable blubberoid in project blubber seems to have been built with Go 1.10.2 already.

tools.blubber@tools-bastion-03:~$ gdb ./blubberoid 
(gdb) p 'runtime.buildVersion'
$1 = 0x7e3716 "go1.10.2"

In T207018#4666216, @Addshore wrote:

The period of exceptions in logstash

https://logstash.wikimedia.org/goto/79a26df78d4623a8f5c1f373598fbde1
https://logstash.wikimedia.org/goto/4d6b1ec854cf50d1f3f89086a2f8b693

It looks like there was a short burt in DBPerformance logs for commonswiki.

Probably didn't relate to the patch

In T191183#4649245, @Dereckson wrote:

In T191183#4647075, @Krinkle wrote:

Gerrit wants 100x100px square thumbnails.

The 100x100 size isn't what currently happens in the repository:

$ identify *.png
default.png PNG 425x425 425x425+0+0 8-bit Grayscale sGray 13916B 0.000u 0:00.007
ladsgroup.png PNG 640x640 640x640+0+0 8-bit sRGB 228c 123552B 0.000u 0:00.000
niedzielski.png PNG 350x350 350x350+0+0 8-bit sRGB 150395B 0.000u 0:00.000
paladox.png PNG 400x400 400x400+0+0 8-bit sRGB 109c 2564B 0.000u 0:00.000

(the size are after optipng -o7 to give a fair representation of what we could expect)

The pthread_create failure sounds like we ran out of memory on that machine. @dduvall killed a whole bunch of left-over containers today that CI hadn't cleaned recently.

Can we use docker kill and docker run --name to fix this? That is, could we name each container after $JOB_NAME and $BUILD_NUMBER and then create a post-build job for all docker jobs that runs docker kill to stop the container for a particular job?

Added test .pipeline files for zotero.

Since there were no databases on the affected boxes that had any schema, I've backed up the ibdata1 file and restarted mysql as there were no hung transactions that were going to be able to finish without their underlying database.

Concept looks good to me. A few problems and questions inline.

In T201611#4611261, @akosiaris wrote:

In T201611#4606202, @thcipriani wrote:

In T201611#4606083, @akosiaris wrote:

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

We'll need npm to install the test dependencies and run the tests in the pipeline, so we'll need a -devel variant for that.

Done. nodejs10-devel image built and uploaded

In T205002#4611234, @Reedy wrote:

Those directories were still in /srv/mediawiki-staging on the deployment host, hence reporting the bug based on the investigation for T205000: Too many versions of MW core on deploy1001

That'll happen when you do a --delete, i.e. the branch directory will sit there empty until then next scap sync is run.

lgtm! Let's land try this in beta!

Looks good! I did some basic testing in beta and it seems like it'll work. Couple of nitpicks inline. Nice quick work on this :)

I've got a stretch instance called deployment-mwmaint01 running in beta with role::mediawiki_maintenance. I made a couple patches to make this happen: one because we don't have conftool in beta and another because we don't have the ldap-admins group in beta (and openldap::maintenance isn't probably needed on this machine).

In T201611#4606083, @akosiaris wrote:

@thcipriani, @dduvall, nodejs 10 image built and uploaded. It's the "slim" variant, I was wondering if it is enough or if we also require the "devel" variant

In T202100#4543309, @Ottomata wrote:

Done, worked everywhere except:

The following hosts were unreachable:
cloudservices1004.wikimedia.org

https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/4319/console got stuck in the same way I've observed in the period between when I first closed this task and when I deployed the second patch, but this time it aborted on its own.

I ran a no-op scap sync using the patch above to use php7.0

Takes a while on beta because of all the extensions (plus disks are slower than in production where it takes like 20 seconds). IIRC we haven't done much to parallelize any of this, it serially walks the extensions directory. There are probably some easy wins here, but, again, it's been a beta-only problem so it hasn't been a high priority.

In T192006#4579891, @Addshore wrote:

@thcipriani just checking, but this would result in an image of whatever is currently on master right?

Deployed this Thursday and actually did have a build timeout yesterday and nobody had to abort \o/ :
https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/2641/console

In T204077#4579698, @Stashbot wrote:

Mentioned in SAL (#wikimedia-releng) [2018-09-13T08:42:11Z] <hashar> aborted maintenance-disconnect-full-disks job | T204077

FYI, I've upstreamed my commit to git-fat: https://github.com/jedbrown/git-fat/commit/2fad58fd0631ed8dcb77358bb9b80e4cd091d3fe

In T203108#4560449, @faidon wrote:

I'd resolve this task, but I'm not sure what else needs to be done with regards to GitHub mirroring, CI etc.?

Will try to move to php7.0 per discussion on T191921

I've merged a patch to scap to allow setting php_version in scap.cfg that will be used in any call to mwscript.

Poking this for ETA.

Friday I removed the integration-slave-docker-1026 node since it was constantly running out of disk-space and then self-recovering (concurrent running containers eating into /).