Page MenuHomePhabricator

Depool precise jenkins instances
Closed, ResolvedPublic

Description

With the deprecation of precise, I suggest depooling precise jenkins instances

Bits from T143349: Deprecate precise instances in Labs by 2017-03-31


In T143349#2683793, @hashar wrote in October 2016:

For CI , we need to keep some Precise instances which are used to run Zend PHP 5.3 tests. That is until MediaWiki 1.23 LTS reaches end of life in May 2017 (lifecycle).

Instances are:

  • integration-slave-precise-1002
  • integration-slave-precise-1011
  • integration-slave-precise-1012

Summary by hashar in February 2017 at T143349#3024759 (see follow up comments there):

About Zend 5.3 / CI:

Context

Candidates were to use https://github.com/phpenv/phpenvhttps://github.com/CHH/phpenv but they are not packaged for Debian, barely maintained and that is a lot of cruft to add to CI. Instead we went splitting jobs on different distributions, the current shards are:

DistroPHP Flavor
Precise Zend 5.3
 TrustyZend 5.5
JessieZend 5.6, Zend 7.0 and HHVM

For Zend 7.0 we use the package from https://deb.sury.org which are co installable up from PHP 5.5 (iirc). So on Jessie we have: /usr/bin/php5 /usr/bin/php7 and /usr/bin/hhvm. We have a php script that picks whatever version is passed to the CI build via a PHP_BIN env variable.

Eventually we will coinstall Zend 5.5 on Jessie as well ( T144959 ) and phase out Trusty instances entirely, specially since Wikimedia production has been doing the same.

The PHP shell script and sharding has a huge advantage: it is ridiculously easy to maintain for us . Set the version you want in CI, invoke the wrapper of few lines of code: success. The oddity is we have to do some routing to map Zend 5.3 jobs to Precise and Zend 5.5 ones to Trusty.

I did try a few times to compile Zend 5.3 on Jessie stripping down some extensions, borrowing patches here and there. But I am illiterate in C or the PHP build chain so that did not go to far.

Plan as of now

CI envisioned porting Zend PHP 5.3 support on Trusty or Jessie back in June 2015 eventually after much discussion and the above work happening (php wrapper, package from sury.org) I have declined the idea. My assumption is that it is not worth the trouble just for the six months or so we would have used them (November 2016 - May 2017).

The end of life of MediaWiki 1.23 is May 2016 which would be during the European Hackathon. At that point:

  • the REL1_23 branches will be closed
  • all the CI related configuration for REL1_23 removed
  • the php53 Jenkins configuration removed
  • the Precise instances deleted

Other solutions?

I understand that Precise has to be dropped due to end of quarter on 03/31 and most importantly Ubuntu dropping support for it in April. It is surely annoying that MediaWiki 1.23 supports just a month later (in May). So from there I guess the choices are:

  • phase out precise / drop PHP5.3. Leaving us without Zend 5.3 CI from April 1st to the May EOL of 1.23. If a new 1.23 security release has to be cut out, figure out a manual strategy to test MediaWiki.
  • Keep the Precise instances as-is till EOL. They can use a stripped down set of puppet manifests so the rest can be purged from Precise references.
  • Figure out a way to forward port Zend 5.3 to Trusty/Jessie in a way that is co installable with the PHP flavors we have (eg /usr/bin/php53 /usr/share/php53 etc). Which might work or not depending on lib versions and whatever MediaWiki codes ends up relying on.

My preference goes toward keeping the Precise instance for an extra month and a half and phase them out the moment 1.23 is EOL.

Event Timeline

Pending outcome of T143349#3024759 . There is a lot of CI configuration work to happen before we can consider dropping the Precise instances.

Change 337203 had a related patch set uploaded (by Dzahn):
contint: drop npm settings for precise

https://gerrit.wikimedia.org/r/337203

There are still a bunch of changes made to REL1_23 branch, specially for the BlueSpice extensions ( https://gerrit.wikimedia.org/r/#/q/branch:REL1_23 ) most probably they are no more running PHP5.3.

I guess we can just move to trigger the Zend 5.5 lint/tests which would still catch most of the issues.

Change 339377 had a related patch set uploaded (by Hashar):
Delete php53lint jobs

https://gerrit.wikimedia.org/r/339377

Change 339388 had a related patch set uploaded (by Hashar):
Delete composer*php53

https://gerrit.wikimedia.org/r/339388

Change 339392 had a related patch set uploaded (by Hashar):
Delete integration-composer-check-php53

https://gerrit.wikimedia.org/r/339392

So. I think I will keep pilling up patches as above as axes through the CI config. Then eventually conclude with some tests to make sure jobs are properly triggered based on branch filter. I did some today https://gerrit.wikimedia.org/r/#/c/339377/2/tests/test_zuul_scheduler.py

Then I dont know. Either I deploy them one by one and make sure they are working or I just merge everything and hope for the best :]

One sure thing, updating the CI config is not rocket science, just grunt work and lot of carefully crafted patches. I am not too worried at this point.

Change 337203 merged by Dzahn:
contint: drop npm settings for precise

https://gerrit.wikimedia.org/r/337203

Will do that next week. I have announced it on wikitech-l:

Subject: CI dropping support for Zend PHP 5.3

Hello,

Next week I will remove Zend PHP 5.3 support from CI entirely. It is
only used for MediaWiki REL1_23 branch which is low traffic.

The reason is CI uses Ubuntu Precise instances that comes with PHP 5.3.
The distribution is end of life (as PHP) and it is being phased out of
the Wikimedia infrastructure.

I will replace the jobs by their equivalent running Zend PHP 5.5 which
would still provide test coverage and linting, albeit with a different
version.

Change 339392 merged by jenkins-bot:
[integration/config] Delete integration-composer-check-php53

https://gerrit.wikimedia.org/r/339392

Change 339377 merged by jenkins-bot:
[integration/config] Delete php53lint jobs

https://gerrit.wikimedia.org/r/339377

Change 339388 merged by jenkins-bot:
[integration/config] Delete composer*php53

https://gerrit.wikimedia.org/r/339388

Change 342446 had a related patch set uploaded (by Hashar):
[integration/config] Remove mediawiki-phpunit-php53* jobs

https://gerrit.wikimedia.org/r/342446

Change 342446 merged by jenkins-bot:
[integration/config] Remove mediawiki-phpunit-php53* jobs

https://gerrit.wikimedia.org/r/342446

Change 342450 had a related patch set uploaded (by Hashar):
[integration/config] Replace php53 jobs with php55 equivalents

https://gerrit.wikimedia.org/r/342450

Change 342450 merged by jenkins-bot:
[integration/config] Replace php53 jobs with php55 equivalents

https://gerrit.wikimedia.org/r/342450

Mentioned in SAL (#wikimedia-releng) [2017-03-13T13:19:20Z] <hashar> Depooled Precise instances from Jenkins T158652 leaving the instances up for now.

Change 343306 had a related patch set uploaded (by Hashar):
[operations/puppet] contint: remove Precise related switches

https://gerrit.wikimedia.org/r/343306

Mentioned in SAL (#wikimedia-releng) [2017-03-17T14:08:41Z] <hashar> salt -v '*precise*' cmd.run 'puppet agent --disable "Pending shutdown on March 20th - T158652"'

Change 343309 had a related patch set uploaded (by Hashar):
[operations/puppet] mediawiki: remove Precise class packages::legacy

https://gerrit.wikimedia.org/r/343309

A couple puppet clean up patches for contint and mediawiki modules.

Mentioned in SAL (#wikimedia-operations) [2017-03-20T08:47:34Z] <hashar> Jenkins: depooling / deleting Precise instances. T158652

Removed from Jenkins, puppet master and salt master. I have deleted the three instances via Horizon.

Pending puppet patches:

Change 343306 merged by Alexandros Kosiaris:
[operations/puppet] contint: remove Precise related switches

https://gerrit.wikimedia.org/r/343306

@hashar is still WIP or are we all done here?

hashar triaged this task as Medium priority.Mar 27 2017, 9:46 AM

The other puppet clean up patch https://gerrit.wikimedia.org/r/#/c/343309/ is on our radars. Will be merged once all Precises instances have been dropped from labs. So there is not much point in keeping this task open.

Change 343309 merged by Muehlenhoff:
[operations/puppet@production] mediawiki: remove Precise class packages::legacy

https://gerrit.wikimedia.org/r/343309

Change 356598 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove transient test

https://gerrit.wikimedia.org/r/356598

Change 356598 merged by jenkins-bot:
[integration/config@master] Remove transient test

https://gerrit.wikimedia.org/r/356598