Page MenuHomePhabricator

Patch testing in releases Jenkins: prevent false positives
Closed, ResolvedPublic

Description

Our releases Jenkins instance runs a nightly job to test that security patches apply cleanly on the master branches of MediaWiki core, extensions and plugins. For a few times in a row, this job has being failing due to the git repository with the MW code found in a dirty state. For example: https://releases-jenkins.wikimedia.org/job/Branch%20cut%20pretest%20patches/233/console:

02:02:01 Applying patch /srv/jenkins-agent/workspace/Branch cut pretest patches/work/patches/branch_cut_pretest/core/01-T355538.patch in /srv/jenkins-agent/workspace/Branch cut pretest patches/work/mediawiki/php-branch_cut_pretest
02:02:01 ERROR: git is not clean: /srv/jenkins-agent/workspace/Branch cut pretest patches/work/mediawiki/php-branch_cut_pretest/.
02:02:01 Applying patch /srv/jenkins-agent/workspace/Branch cut pretest patches/work/patches/branch_cut_pretest/extensions/ConfirmEdit/01-T355073.patch in /srv/jenkins-agent/workspace/Branch cut pretest patches/work/mediawiki/php-branch_cut_pretest
02:02:01 Applying patch /srv/jenkins-agent/workspace/Branch cut pretest patches/work/patches/branch_cut_pretest/extensions/WikibaseLexeme/01-T357101.patch in /srv/jenkins-agent/workspace/Branch cut pretest patches/work/mediawiki/php-branch_cut_pretest
02:02:01 [FAILED] /srv/jenkins-agent/workspace/Branch cut pretest patches/work/patches/branch_cut_pretest/core/01-T355538.patch
02:02:01 [APPLIED] /srv/jenkins-agent/workspace/Branch cut pretest patches/work/patches/branch_cut_pretest/extensions/ConfirmEdit/01-T355073.patch
02:02:01 [APPLIED] /srv/jenkins-agent/workspace/Branch cut pretest patches/work/patches/branch_cut_pretest/extensions/WikibaseLexeme/01-T357101.patch

The git repo was not clean because the Listings extension has been recently removed, which left an untracked dir in the job workspace:

jenkins-agent@releases1003:/srv/jenkins-agent/workspace/Branch cut pretest patches/work/mediawiki/php-branch_cut_pretest$ git status
On branch wmf/branch_cut_pretest
Your branch is up to date with 'origin/wmf/branch_cut_pretest'.
[...]
Untracked files:
  (use "git add <file>..." to include in what will be committed)
extensions/Listings/

The git error has a knock-on effect on whichever security patch is tested first. The patch gets reported as failing to apply, even though that is (most likely) not the case. Additionally, now that we are notifying patch failures, the problem also causes confusion downstream: https://phabricator.wikimedia.org/T355538#9615894

I think we should:

  1. Verify the way we are checking out submodules in the Jenkins job (as of a recent check, the prod repo looks healthy and the extension removed). Extensions/skins that have been removed should also be cleared in existing checkouts.
  2. Decouple the git check from the individual patches. The apply-patches command should fail if git is not clean, but individual patches should not be reported as failing.

Details

TitleReferenceAuthorSource BranchDest Branch
prep: remove deleted submodules from working treerepos/releng/scap!244jnucheT359809master
patches: do not notify security tasks about git failuresrepos/releng/scap!241jnucheT359809master
Customize query in GitLab

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Interesting problem. I'm very curious about how to properly deal w/ submodule removal. Ideally scap prep auto should handle this situation.

jnuche claimed this task.