If I go to a referrer and open the console in Chrome, I see the error, but only if I'm using the Minerva skin. Here's a referrer with &useskin=minerva appended.

In T337570#9578426, @dduvall wrote:

I went ahead and implemented/submitted a Phorge integration upstream

That include looks to be owned by the Security Team. From our experience with Kokkuri, my general advice from GitLab CI includes is either:

I went ahead and implemented/submitted a Phorge integration upstream as well as a feature request to allow the custom issue tracker ID delimiter to be configured. Let's see how they are received. If they won't take either, I will revisit the local gem setup.

In T337570#9569324, @bd808 wrote:

Thinking about other implementation possibilities here... do we have any way to inject custom javascript into the GitLab UI? If so the use case of turning mentions of T\d+ into links to Phab seems like something that could be done with a script that is the equivalent of a MediaWiki Gadget. We could also go in an interesting direction of creating a Greasemonkey script or even a Wikimedia GitLab browser extension that did this kind of client side enhancement.

In T337570#9453166, @Jelto wrote:

I'd like to see proper links to Phabricator in Gitlab, this would be a nice usability feature. Is the usage of a custom Gem still the preferred implementation for this? Currently the usage of local gems is disabled since Sep 2023 due to issues on upgrades. If we want to proceed with the custom gems we would need a solution to make sure the custom gem file survives a version upgrade or gets disabled and re-created after the upgrade.

We recently did some work on Blubber (and other pipeline tooling) documentation in T352259: Review critical-path deployment pipeline documentation, and part of the work entailed refactoring the reference documentation.

Thanks for tackling Shellbox, @bd808. It looks like MW-Vagrant's Blubberoid wrapper is in turn used by https://gerrit.wikimedia.org/g/mediawiki/vagrant/+/4b15d7aeb71e28b13a653f137dc18bec0376db8d/puppet/modules/role/templates/addimage/systemd/image-suggestion-api.epp

In T356684#9522131, @taavi wrote:

In T356684#9522127, @dduvall wrote:

Oh, you're using Blubberoid! It's deprecated and I'm planning to decommission it soon. Can you point me to your downstream project? Can we get that using the Blubber buildkit frontend perhaps (# syntax = line in the blubber.yaml?

Oops. I must've missed the announcement of that. Using the buildkit syntax seems to work, indeed.

Additionally for Blubber, I deprecated the Getting Started pages and Deployment Pipeline documentation and removed them from the portal. I renamed Usage to Examples and linked to the project examples folder which contains acceptance tests with descriptions for people. I added a Reference page and portal item and linked to the project's docs/configuration.md file.

Oh, you're using Blubberoid! It's deprecated and I'm planning to decommission it soon. Can you point me to your downstream project? Can we get that using the Blubber buildkit frontend perhaps (# syntax = line in the blubber.yaml?

What version of Blubber are you using? Someone else encountered this previously (see T346090: Blubber: Add workaround for Bookworm's Python pip restrictions) and the workaround we settled on was setting PIP_BREAK_SYSTEM_PACKAGES. Not the most elegant solution but these are container images so "breaking" system package installation seems... ok perhaps?

In T352259#9520369, @KBach wrote:

@dduvall do you think it makes sense for me to rename the User Guide (to Reference or similar) and review it slightly? Or should we wait for your mechanism of generating docs/configuration.md to be ready and deprecate the User Guide entirely when it's available?

Blubber:

• https://gitlab.wikimedia.org/repos/releng/blubber/-/blob/main/README.md?ref_type=heads - general review, especially the Usage section.

Sorry for the delay on my side. Assigning to myself as I think I'm the only one that still needs to perform review.

Just a note that I had the best luck with https://gitlab.com/gitlab-org/gitlab-compose-kit on a Debian system. It's unofficial but provides lighter weight options than GDK for simply running tests. There's also a lot less churn when tearing down single components or even the entire thing (due to cached images and its isolation of GitLab services via Docker Compose).

Thanks for the update, @Jdlrobson. I'll roll group0.

How shall we proceed with 1.42.0-wmf.12 train given the outstanding patch? Is this still a blocker?

Thanks, @Dzahn. After looking a bit more, I don't think the presence in scap_targets should affect train, so I'm deescalating this. Whether or not depooled hosts should still be present in scap_targets is up for debate.

This is a blocker until the host is removed from /etc/dsh/group/scap_targets.

Zuul is fully provisioned in devtools on zuul-1001.devtools.eqiad1.wikimedia.cloud. The Zuul services are currently run via systemd/docker.

In T350478#9305571, @dduvall wrote:

Still, we could refactor docker-gc to make a direct query to get usage info only for volumes and images, and whenever we upgrade Docker we'll start to benefit from omitting container filesystem usage calculations.

It seems that it's the calculation of running container filesystem usage that is taking so long, and docker-gc doesn't really need that usage figure since it only cleans up volumes and images.

I'd say this experiment was a success.

The development environment now includes a GitLab runner and is generally working.

Created a new repo at https://gitlab.wikimedia.org/repos/releng/gitlab-dev for GitLab/Zuul hacking. It needs a GitLab runner which I'll work on today.

In T347522#9204725, @matmarex wrote:

The issue only affects wikis that override MediaWiki:Difference-title, for example here on he.wp: https://he.wikipedia.org/w/index.php?title=מדיה_ויקי:Difference-title&action=edit

We need to see how common it is, and why it was done, and then either restore the ability to use HTML in that message, or propose an alternative solution. I'll have a look.

I've installed the gitlab-phorge gem on devtools by manually cloning the repo to /srv/gitlab-wmf-gems/gitlab-phorge and specifying the following project level hiera:

From my review comment:

This seems like an OK solution for now, as I don't think the risk of breaking the distro's Python installation necessarily applies within a container filesystem.

In T318289#8404147, @bd808 wrote:

Another way to attempt to break the cycle would be to implement a native LLB generation mode in Blubber in parallel with the Dockerfile generation mode. This would allow testing to see if T318866 is resolved by that change.

An experimental native-LLB build of Blubber's BuildKit frontend has been published as docker-registry.wikimedia.org/repos/releng/blubber/buildkit:experimental-native-llb.

In T345000#9128483, @xcollazo wrote:

Some testing from my side:

My pytest job ran successfully: https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/jobs/137133

My somewhat heavier publish_conda_env release job seems to have stalled though: https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/jobs/137134

It was already publishing the artifact at 11 minutes in, but then it got at warning, presumably from k8s?:

WARNING: Getting job pod status pods "runner-emdqpmey-project-1420-concurrent-02lhsm" is forbidden: User "system:serviceaccount:gitlab-runner:gitlab-runner-memopt" cannot get resource "pods" in API group "" in the namespace "gitlab-runner"

And even though these new pods have a timeout of 20m, the job is still running at 26m+ as of this comment, but seems stalled? This job typically takes ~10 minutes.

In T345178#9131602, @Jdforrester-WMF wrote:

Sure, e.g. https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/jobs/137599 which has two whole runs of npm test in it, one on ARM64 and another on AMD64. Sometimes one fails but not the other, such as when the packaged python is very slightly different between arches, and it's very challenging even with grep to work out which is which.

@xcollazo alright, we should be good now. Test out the memory-optimized tag and let us know the results!

Scratch that. I just realized that I overlooked the node tolerations in the runner config. I'll have to do a follow up. Sorry.

@xcollazo the memory optimized runner pool has been deployed. You can schedule jobs to this pool by using the tag memory-optimized, either at the pipeline level under defaults or at the job level.

thanks, @MoritzMuehlenhoff ! Closed

In T318382#8377321, @dduvall wrote:

In T318382#8317695, @dduvall wrote:

Thanks, Moritz. Can we do the same for buster, pulling in the newer 20.10.18~3~debian-buster packages as well? I think using the same exact version on contint and integration agents is probably best.

gentle ping, @MoritzMuehlenhoff

WMCS runners are no longer scheduling untagged jobs. Thanks for the review/merge, @Jelto !