| sbassett | |
| Oct 21 2021, 6:37 PM |
| F34762333: Screen Shot 2021-11-22 at 3.56.37 PM.png | |
| Nov 22 2021, 9:57 PM |
| F34705006: Screen Shot 2021-10-21 at 1.32.21 PM.png | |
| Oct 21 2021, 6:37 PM |
| F34705004: Screen Shot 2021-10-21 at 1.31.41 PM.png | |
| Oct 21 2021, 6:37 PM |
Hello Release-Engineering-Team -
It looks like runners have been completely disabled for gitlab.wikimedia.org at this time?
I assume this might be related to the ongoing discussion at T291978 etc? Is there a plan to enable any variety of runners soon? If not, I'd like to request that perhaps 2 to 3 runners be enabled for pipelines under the security group within Gitlab so that we have an easy means to continue testing and developing our security ci templates this quarter and next. Thanks.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| hiera: Add hostname/certname based lookup to secret hierarchy under labs | operations/puppet | production | +6 -2 |
This is more about T292094 (Limit GitLab shared runners to trusted contributors), though there's plenty of overlap.
@dduvall is currently rebuilding the runners to provision more space, but we can probably temporarily assign one of that pool to the security project group to unblock you all. On a very slightly longer timeline, I think we're likely to assign those runners all to a top-level group which we'll be moving all "officially hosted" projects to, including security.
Not quite. We have runners reconfigured at the moment, trying to work out how best to limit a single one to a group.
Another short-term option would be to create a new VPS project and provision your own dedicated runners for the /security group, but it's a fair amount of hoop-jumping.
Change 734703 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/puppet@production] hiera: Add hostname based lookup to secret hierarchy under labs
Hey @brennen @dduvall - we were hoping to get this resolved a bit sooner than later, as it's going to impact the work for our current appsec pipeline sprint. If this is stalled or lower priority, would it be possible to set up our own runner on wmcs, at least temporarily? I don't really want to do that, but I suppose we can if that would be the fastest and most feasible path forward. Thanks.
Mentioned in SAL (#wikimedia-releng) [2021-10-29T21:28:48Z] <brennen> manually registering runner-1008-security as a stopgap measure for T294050
As indicated by log entry above, I've manually registered runner-1008-security for the security project group. I don't think this will collide with anything.
Change 734703 merged by Andrew Bogott:
[operations/puppet@production] hiera: Add hostname/certname based lookup to secret hierarchy under labs
Mentioned in SAL (#wikimedia-releng) [2021-11-01T21:47:25Z] <brennen> gitlab runner-1008: re-registering runner for security group using host-specific config (T294050)
Hey @brennen et al-
Did this get reverted? gitlab.wikimedia.org is telling me that our projects don't have any active runners.
What project is this for? Everything under /repos/security should have access to all of the group runners allocated to /repos. We don't currently have shared runners for the instance as a whole.
Hey @brennen (and maybe @Jelto?) -
This seems to be happening again, sadly. After receiving the previous "This job is stuck..." notice, I looked at the CI config for the Secteam BoilerPlate Fork - PHP JS project here: https://gitlab.wikimedia.org/repos/security/secteam-boilerplate-fork/-/settings/ci_cd#js-runners-settings. It looks like there are available group runners, but that they're all currently offline? Not sure if this is perhaps due to the recent security upgrade T297183 or work related to T295481?
@sbassett I seems all runners were offline. During work on T295481 I produced a invalid GitLab runner configuration which propagated to the WMCS runners as well. I fixed the configuration and re-registered the runners (force puppet run). All runners are online again. https://gitlab.wikimedia.org/repos/security/secteam-boilerplate-fork/-/settings/ci_cd#js-runners-settings shows some runners as well.
@Jelto - ah, ok, thanks. Yes, things look good now. I guess it's also good to know that an invalid runner config has the potential to DoS available runners :)