Page MenuHomePhabricator

CI issues on fundraising/REL1_25
Closed, ResolvedPublic

Description

Couple new test failures I can't replicate locally:

"Wikibase requires MediaWiki 1.26 or above." (we shouldn't be testing wikibase against 1.25)

and

1) IPTCTest::testIPTCParseForcedUTFButInvalid
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 'ø'
+    0 => ''
)

Not sure about this one but the extended unicode char makes me think it's encoding related. I will try to get an idea of the nature of the breakage before bugging Releng, but any deployments in the mean time will have to be force merged.

Event Timeline

The current branching system used by Fundraising is incompatible with how Release Engineering has designed the Zuul layout for multi-repository involved jobs. They checkout one or more repos and for each repo it tries one of three possible git references: The unmerged change being tested, the target branch, or master.

So that when testing change A against branch "foo" in project "example/core" with example/one and example/two, it will end up checking out:

  • example/core: change A for its branch foo
  • example/one: current branch foo
  • example/two: current branch foo

Due to fundraising branches being on the same repository for regular releases (namely, mediawiki/core), this conflicts and means that changes for fundraising branches get the same jobs as for other branches by default. It also means that those jobs will checkout extensions and first look for a branch fundraising/REL1_25 and then fall back to master when no such branch is found in the extension repository.

This logic is part of the OpenStack Zuul spec and we shouldn't aim to change that as it'll mean having to maintain additional patches which can get tricky very fast.

Here's a radical idea: Create a fork of mediawiki in Gerrit under a different name (e.g. wikimedia/fundraising/mediawiki-core or some such) and create the REL1_25 branch in there. That way, Zuul will treat it as its own project with its own set of test jobs (could be re-used from core still of course, but it'll no longer inherit implicitly, only explicitly). It will also naturally make it compatible with how Zuul works and check out other projects' REL1_25 as well.

This should be trivial to do in Gerrit and won't require much change on the fundraising servers (simply update the address in git-remote origin set-url). It'll also make things more future proof. Since it happens quite often that we change or enable new jobs for MediaWiki core that currently all explicitly have to be disabled for fundraising branches. All that maintenance pressure would go away.

@Krinkle T124105 was specially to get tests to accommodate CI with fundraising/REL1_25 and I have suggested to create a specific job that would use Zuul cloner per project branch overriding (--project-branch mediawiki/core=fundraising/REL1_25 or just override the fallback branch via --branch=fundraising/REL1_25).

The job mwext-donationinterfacecore125-testextension-php53 has been introduced via https://gerrit.wikimedia.org/r/#/c/266653/7/jjb/wm-fundraising.yaml,cm

I am not sure why T124574 would suddenly surface, but if its fix is not included in the fundraising branch they should :)

@hashar T124574 is now fixed for them. @Legoktm suggested they backport it, which I did and it was merged. https://gerrit.wikimedia.org/r/#/c/275018/

@hashar also wikibase or wikidata or any of the extension dosent have that branch so that where the problem came.

Why does Wikibase/Wikidata repos are included in that job? That does not make sense. It is a specific/narrow job that only clones a limited subset of repositories.

If a generic job such as mw-testextensions-*, they should have a branch filter to prevent them from running on the fundraising/* branches.

@hashar it dosent run the on the extension specific job but ones on the gate extension mediawiki-extensions-php55

Which runs wikidata

Change 275050 had a related patch set uploaded (by Paladox):
Blacklist mediawiki-extensions-php55 test on branch fundraising/REL

https://gerrit.wikimedia.org/r/275050

Change 275050 merged by jenkins-bot:
Blacklist mediawiki-extensions-php55 test on branch fundraising/REL

https://gerrit.wikimedia.org/r/275050

Really appreciate everyone's input on this. FWIW having that test in place stopped us from deploying a syntax error this morning, as we mostly run current php or hhvm in development, and it's easy to let new features sneak in (in this case "function()[]" syntax) which only the 5.3 tests catch.

On that note, AFAIK there is no technical reason we can't run 5.5 and/or Mediawiki master in production, our extensions work fine (with perhaps some light refactoring). PCI rules have historically made the payments cluster difficult to update, but talking to @Ejegg today it sounds like recent security improvements may give us some wiggle room regarding what we deploy. I definitely think following the train (from a safe distance) would be really beneficial in the long run.

CC @Jgreen @awight in case I'm totally wrong about that.

Call this done as the described problems are fixed, but this sort of thing will probably reappear from time to time until we are running newer PHP

New, blocking failures:

https://integration.wikimedia.org/ci/job/mwext-donationinterfacecore-REL1_25-testextension-zend53/39/console

wikimedia/composer-merge-plugin: 1.0.0 installed, 1.3.1 required.
Error: your composer.lock file is not up to date, run "composer update" to install newer dependencies
Build step 'Execute shell' marked build as failure

https://integration.wikimedia.org/ci/job/npm-node-4.3/19018/console

Assertion error: node version v4.4.6 does not match '^v4[.]3[.]'
Build step 'Execute shell' marked build as failure

Are we going to just reopen this task with any new failures on fundraising CI? I sure hope not...

New, blocking failures:

https://integration.wikimedia.org/ci/job/mwext-donationinterfacecore-REL1_25-testextension-zend53/39/console

wikimedia/composer-merge-plugin: 1.0.0 installed, 1.3.1 required.
Error: your composer.lock file is not up to date, run "composer update" to install newer dependencies
Build step 'Execute shell' marked build as failure

https://integration.wikimedia.org/ci/job/npm-node-4.3/19018/console

Assertion error: node version v4.4.6 does not match '^v4[.]3[.]'
Build step 'Execute shell' marked build as failure

The last one is T139374 at least.

@greg
Sorry, I got frustrated and am jetlagged. Reopening with all these subscribers doesn't make sense, you're right.

We're having a bit of a CI crisis over here, nobody else's fault though. We need to get ourselves off of REL1_25 and closer to the WMF release branches. We came up with a strategy to accomplish this, but until then will be suffering regressions increasingly often...