In T332953#8769056, @thcipriani wrote:I see that translatewiki.net already supports at least one gitlab.com repo (and there's client code for GitLab already 🎉 ).
So, still needed:
- Create a translatewiki user able to push code on gitlab.wikimedia.org
- Ensure that user has needed permissions across repos (bonus points for a way to centrally admin that vs. ad hoc way in gerrit)
- Tricky part: recreate mediawiki-i18n-check, only run on changes from l10nbot/localization (may/likely to require changes to GitLab client code, too)
I'll get those conversations started.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | taavi | T320446 The following container images did not match any of the allowed registries ([['docker-registry.tools.wmflabs.org']]) | |||
| Resolved | taavi | T320476 do not use :latest for toolforge infrastructure components | |||
| Resolved | taavi | T320667 Cloud services enhancement proposal: Toolforge Kubernetes component workflow improvements | |||
| Resolved | dcaro | T336130 Automatically build Toolforge infrastructure container images in GitLab | |||
| Open | None | T334399 Move Toolforge PipelineLib repositories to GitLab | |||
| Open | None | T344765 Migrate striker to GitLab | |||
| Stalled | None | T343261 Migrate wikimedia/developer-portal to GitLab | |||
| Stalled | None | T344729 Migrate toolhub to GitLab | |||
| Open | None | T334419 Set up translatewiki.net exports to push (and merge) to Wikimedia GitLab |
Event Timeline
Comment Actions
I think the only thing missing from T318164: Add support for projects on Wikimedia GitLab that is mentioned in this task is the mediawiki-i18n-check.
Is that right @abi_?
If so, maybe we can change the scope of this ticket. Otherwise we can close this as a dupe :)
Comment Actions
It looks like https://gitlab.wikimedia.org/l10n-bot is pushing to the i18n branch, but it's not creating MRs or self-merging them.
Comment Actions
That is the correct assessment.
- mediawiki-i18n-check is pending.
- The code can create merge requests (See: https://gitlab.wikimedia.org/diegodlh/w2c-server/-/merge_requests/2) but it does not self merge them.
Comment Actions
A proposal for handling l10n-bot commits in Gitlab CI:
Create a translatewiki-CI repo in Gitlab which contains a pipeline that can be triggered from upstream MR pipelines in repos that want to use this functionality. The pipeline will only be triggered for MRs created by l10n-bot. The pipeline will have a job that performs a security check on the commit(s) in the MR. If the check passes, the job will merge the MR. If the check does not pass, the job will add a comment to the MR saying that human review is required. In either case the job completes successfully. The translatewiki-CI repo will have a variable holding an impersonation token for the l10n-bot account so that it can add the comments or perform the merge. (https://gitlab.wikimedia.org/admin/users/l10n-bot/impersonation_tokens). The l10n-bot user will be a member of this repo with Developer role so that the pipeline can be triggered.
Q: Why is it a triggered downstream job in another repo instead of a template that is included in a gitlab-ci.yml file?
A: The CI job needs to be able to 1) add comments on the MR saying that the commit requires review by a human or 2) merge a MR that satisfies the security check. Both of these operations required a token associated with the l10n-bot account. This token must not be easily accessible to upstream CI jobs, so it is stored as a secret in the downstream job.
Comment Actions
A user with Maintainer role examines the changes in the merge request and presses the Merge button in the UI if the changes look safe. (I tried to locate existing documentation on this procedure for Gerrit but didn't run across anything that says this explicitly).
Comment Actions
In gerrit we have the l10n-bot-watchers group, A member of that groups looks for open changes using this query https://gerrit.wikimedia.org/r/q/owner:L10n-bot+status:open and reviews and submits them manually.
Comment Actions
A proposal for handling l10n-bot commits in Gitlab CI
This all sounds pretty reasonable, if a touch convoluted, but I think that's inherent to the toolchain.
Comment Actions
I think the proposal for l10n-bot commits looks good.
For humans merging code that does not pass the test, that's still got a few details to be sorted.
I believe we have "l10n-bot-watchers" group because a +2 from the l10n-bot group triggers the same failing tests, so a +2 from an "l10n-bot-watcher" triggers the regular repo tests (which pass, since this is an l10n update, and the code merges).
In the case of GitLab and this proposal, I think an "l10n-bot-watcher" type group can just click the "merge" button. But this means (I think):
- We'd need to have that group (or person, or persons—TBD; maybe even the "l10n-bot" group?) added as a maintainer on the repos
2. We'd need to ensure we can merge with this failing test (I presume we can, but I've been wrong about GitLab before :)) (answered via T334419#8890677)
Comment Actions
Note that my proposal says that the security check job will either merge the MR or add a comment to the MR saying that a human must review, but in either case the job terminates successfully.
Comment Actions
I'm curious about the efficiency of the proposed manual review workflow. How many clicks does it take to review and merge one MR in GitLab?
For Gerrit:
- Ctrl+click to open in a new tab (can open all changes at once).
- One click to open the job output (if you manage to hit the box with arrow), else two clicks. Scroll down to find the relevant output from a wall of text.
- Close the tab (Ctrl+W or click).
- (Assuming no further review is needed at this point) One click to remove jenkins-bot V-1 vote. Page reloads.
- One click for V+2 vote. Page reloads.
- One click for Submit. Another click to confirm with "Continue". Page reloads.
- Close the tab (Ctrl+W or click).
That's 6 clicks and two keyboard shortcuts per change at minimum, while trying to hit small targets around the screen. Fortunately Gerrit loads quite fast here so no much time is spend waiting until the page loads. Ideally the new workflow would be at least as efficient, or we need to take a harder stance of eliminating false positives.
I don't usually use CR+2, because there is high chance that the tests fail for unrelated reasons, and handling for those would take even more human time.
I haven't calculated any stats, but it's not uncommon to have 10+ patches to review per export. We currently support 5 branches (master, 1.40, 1.39, 1.38 and 1.35). Master gets updates multiple timer per week, and older branches weekly.
Comment Actions
@Nikerabbit Using https://gitlab.wikimedia.org/search?scope=merge_requests&search=Localisation+updates+from+https%3A%2F%2Ftranslatewiki.net&state=opened as the example starting point,
- Ctrl-click each of the open merge requests.
- On an open MR, click on the Changes tab to see what changed.
- Click on the Overview tab when done reviewing.
- (Maybe): Click the Rebase button to rebase the change if needed.
- Click the Merge button.
Comment Actions
I would prefer to review the output from the CI which is usually only a few lines instead of the potentially hundreds of line of actual changes. If only those particular lines from the output can be posted as a comment that would be more fluent that the current workflow.
Comment Actions
A proposal for handling l10n-bot commits in Gitlab CI
Proposal looks good to me, but I have a few questions/concerns:
- The description mentions: "Ensure that user has needed permissions across repos (bonus points for a way to centrally admin that vs. ad hoc way in gerrit)" Have we landed on a way how to handle that? i.e. ensure l10n-bot has access everywhere it needs. I think maybe @thcipriani comment about a "l10n-bot-wather" group is trying to address this? But I'm not sure
- How is the beginning of the workflow triggered? In other words, what makes l10n-bot create an MR in the upstream repo in the first place?
- What happens in the case when the downstream pipeline needs to be re-run/re-tried? That can only be done by l10n-bot or some other user with at least developer access to the downstream repo translatewiki-CI
Comment Actions
That is handled by translatewiki.net. An export script is triggered either manually or by a systemd timer. This script updates translations in json files, creates a git commit if there are changes and force pushes that to a named branch in origin*1 and finally uses GitLab APIs to create a MR.
[1] We do not currently support forking and doing merge requests from forks.
More details:
Comment Actions
Giving l10n-bot Developer role in the repos that want translatewiki support will be a required step. This is already briefly mentioned in https://translatewiki.net/wiki/Setup_of_a_new_project#Access_for_delivering_translation_updates but could use a clearer description of how to do it. We'll set this up for mediawiki tree.
- What happens in the case when the downstream pipeline needs to be re-run/re-tried? That can only be done by l10n-bot or some other user with at least developer access to the downstream repo translatewiki-CI
That's a good question. I expect that the members of Gerrit's l10n-bot-watchers group will be members of the downstream repo so that will probably cover most rerun/retry cases should they come up.