Page MenuHomePhabricator

Upgrade php-ast to 1.0.1 in CI composer-package containers
Closed, ResolvedPublic

Description

CI currently runs with version 0.1.2 of php-ast, as can be seen in https://integration.wikimedia.org/ci/job/composer-package-php70-docker/5497/console. However, we need version 0.1.4 (at least) to test seccheck, and 0.1.2 is very old anyway.

Event Timeline

Daimona created this task.Mar 19 2019, 6:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2019, 6:58 PM

@Legoktm could you please take a look?

Will the current plugin continue to work properly with ast 0.1.4? Or do we also need to keep 0.1.2 for back/compat reasons?

@Legoktm I think the current version should work with 0.1.4. But actually, could you please add 0.1.5 as well? We'll need it for the next upgrade. Thanks!

@Legoktm I think the current version should work with 0.1.4.

Gotcha. I can do that then.

But actually, could you please add 0.1.5 as well? We'll need it for the next upgrade. Thanks!

Building newer versions of php-ast is trivial, the main problem is determining which version to use when running phan. If you look at https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/dockerfiles/mediawiki-phan/run.sh#36 you can see how I worked around this for standard phan.

I suppose we coud look at the phan version number being installed and do some conditional based on that.

@Legoktm Cool, thanks! I think for now you can use whatever hack comes more handy for seccheck, as at the end of the upgrade we should ideally have a single version installed.
A conditional checking the required phan version seems fine.

From a CI perspective, we need to keep supporting older branches of MediaWiki that we run tests against, so the "migration" of php-ast from my perspective only ends once the last branch using the older version is phased out.

@Legoktm Right. But anyway at the end of the migration we'll only have 0.1.4 for the current version, and ?[0] for the final version based on 1.2.x.

[0]: To be determined according to phan's requirement, I guess 1.0.x as in the phan job.

hashar renamed this task from Upgrade ext-ast to 0.1.4 to Upgrade php-ast to 0.1.4.Mar 20 2019, 8:27 AM
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.Mar 20 2019, 8:34 AM

Tentatively that has been solved already?

We have T174339 to upgrade to phan 0.8.5, which requires a newer version of php-ast than the one we had.

T174338: Provide php-ast 0.1.5 or later as a Debian package for CI was the task to get a newer one, I have forked the Debian repository back in January 2018 and rebuild it but the resulting Debian package never got published. I got distracted by an unrelated build failure.

Debian Stretch ships with 0.1.2, but testing has 0.1.6 so we can probably ask Operations to rebuild it for us and publish it to apt.wikimedia.org . As noted, the package is also available in sury.org, but that is not ideal (that might brings a lot more different packages as well).

Finally cscott found that Phan provides a parser to replace php-ast when it is not present phan --allow-polyfill-parser. And it seems to be deemed good enough.

From a CI perspective, we need to keep supporting older branches of MediaWiki that we run tests against, so the "migration" of php-ast from my perspective only ends once the last branch using the older version is phased out.

Potentially we could use per branches jobs that would be stalled to some arbirtary old version of php-ast/phan. However the current system is centrally managed which is a bit of a maintenance pain. For jobs that use php-ast we could only trigger them for the master/wmf branch, but most of the jobs should probably just loose the php-ast requirements via phan --allow-polyfill-parser.

We have T174339 to upgrade to phan 0.8.5, which requires a newer version of php-ast than the one we had.

IIUC that's for the phan job itself which yes, already has php-ast v.1.0.1 installed.

Debian Stretch ships with 0.1.2, but testing has 0.1.6 so we can probably ask Operations to rebuild it for us and publish it to apt.wikimedia.org . As noted, the package is also available in sury.org, but that is not ideal (that might brings a lot more different packages as well).

I think we should proceed with little steps to avoid breaking anything, and thus upgrade to 0.1.4 first, then 0.1.5 (and then ...?).

Finally cscott found that Phan provides a parser to replace php-ast when it is not present phan --allow-polyfill-parser. And it seems to be deemed good enough.

True, but it's been introduced in 0.8.13, which means that we still need ast until we get there.

@Legoktm I checked all the remaining versions up to 1.2.6 and I can now confirm that we only need 0.1.5 for php-ast. 0.1.2 should be kept for the current version only.

Daimona renamed this task from Upgrade php-ast to 0.1.4 to Upgrade php-ast to 1.0.1.Apr 28 2019, 8:48 AM

OK, so the final request is:

  • Have both an old version (0.1.2? 0.1.4?) and 1.0.1 for the seccheck job, just like it happens for the phan job
  • Have 1.0.1 enabled for jobs running in the seccheck repo, when seccheck 2.0 will be released.

Ideally, though, any non-phan-related job should have 1.0.1 installed, because 0.1.2 is very old anyway.

@Legoktm Per T216974#5143707, would it be possible to implement the same solution used for the phan job?

Ah, tests for the 2.x branch were just re-enabled, but now they fail due to ast 0.1.2 being installed. See for instance https://integration.wikimedia.org/ci/job/composer-package-php70-docker/6239/console.

At this point, we'd need php-ast 1.0.1 to be installed for the following jobs:

  • composer-package-php70-docker
  • composer-package-php71-docker
  • composer-package-php72-docker

For what concerns the 70 job, we would also need to keep ast 0.1.2 if the commit being analyzed is for the SecurityCheckPlugin repo, but NOT in the 2.x branch. If that's too complicated, I guess we can just upgrade to ast 1.0.1 everywhere and make tests on master temporarily fail, given that the release of 2.0 is close and I don't think we'll have to push anything to master until then.
@hashar Could you please take a look?

Legoktm claimed this task.Jun 26 2019, 7:13 PM

Sorry, this dropped off my radar. Will work on this today.

@Legoktm Thanks! Since we're here: seccheck72 jobs have been created lately as part of T226420. They'll also need ast 1.0.1 to work.
And before enabling them, libraryupgrader should upgrade seccheck everywhere; I'll leave a note in the dedicated task.

Change 519424 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Install php-ast in composer-package-* containers

https://gerrit.wikimedia.org/r/519424

Change 519424 merged by jenkins-bot:
[integration/config@master] Install php-ast in composer-package-* containers

https://gerrit.wikimedia.org/r/519424

At this point, we'd need php-ast 1.0.1 to be installed for the following jobs:

  • composer-package-php70-docker
  • composer-package-php71-docker
  • composer-package-php72-docker

I assume leaving off php73 was accidental?

For what concerns the 70 job, we would also need to keep ast 0.1.2 if the commit being analyzed is for the SecurityCheckPlugin repo, but NOT in the 2.x branch. If that's too complicated, I guess we can just upgrade to ast 1.0.1 everywhere and make tests on master temporarily fail, given that the release of 2.0 is close and I don't think we'll have to push anything to master until then.

For composer-package-*, I'm going to go the easy way out and just upgrade to 1.0.1 everywhere. For the seccheck job itself we'll need some kind of back-compat with 0.1.2 but I have an idea for that.

I assume leaving off php73 was accidental?

Yes, thanks.

For composer-package-*, I'm going to go the easy way out and just upgrade to 1.0.1 everywhere. For the seccheck job itself we'll need some kind of back-compat with 0.1.2 but I have an idea for that.

Nice, thanks! As for seccheck jobs, the back-compatibility will only be necessary for the PHP70 job (which should support both seccheck 1.5 and 2.0). The PHP72 job (which is, I guess, the one we'll want to settle on as soon as nothing uses seccheck 1.5) will only have to support seccheck 2.0 with ast 1.0.1.

OK, there's some PHP version difference that I still need to debug... :|

11:45:35 PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/ast.so' - /usr/lib/php/20151012/ast.so: undefined symbol: zend_string_init_interned in Unknown on line 0

Uhm, I just realized that ast 1.0.1 may not be available for PHP 7.0. Looking at phan's composer for 1.3.2 (source) I saw the PHP requirement of >=7.0.0 and ast 1.0.1+ recommended, and thought that any combination of the two was compatible.
So, to fix this, we should IMHO:

  • Temporarily downgrade ast requirement for seccheck, to keep PHP7.0 compatibility, *only* for the 2.0 release (since the next phan upgrade will drop php70 support anyway)
  • Run composer-package-php72 for the test pipeline in the seccheck repo (not necessary, but that was the goal anyway)
  • Make composer-package-php70 use an older version of ast.

As an alternative, all of the composer-package jobs could use ast 1.0.0 (which I see supports PHP70+), and be upgraded to 1.0.1 as soon as the PHP70 job is dropped, or seccheck >2.0 is released. In this case, the seccheck requirement should be lowered to 1.0.0, too.

Change 519599 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/tools/phan/SecurityCheckPlugin@2.x] Temporarily lower ast requirement

https://gerrit.wikimedia.org/r/519599

Change 519600 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[integration/config@master] Upgrade jobs for SecurityCheckPlugin

https://gerrit.wikimedia.org/r/519600

I changed the requirement for AST to 1.0.0.
@Legoktm Could you please try upgrading the PHP70 job to use AST 1.0.0? Everything should work at that point.

Change 519599 merged by SBassett:
[mediawiki/tools/phan/SecurityCheckPlugin@2.x] Temporarily lower ast requirement

https://gerrit.wikimedia.org/r/519599

sbassett triaged this task as Normal priority.Jun 28 2019, 9:48 PM

Change 519600 merged by jenkins-bot:
[integration/config@master] Upgrade jobs for SecurityCheckPlugin

https://gerrit.wikimedia.org/r/519600

Change 520054 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental

https://gerrit.wikimedia.org/r/520054

Change 520054 merged by jenkins-bot:
[integration/config@master] layout: [SecurityCheckPlugin] Revert blocking jobs to php70; move php71+ to experimental

https://gerrit.wikimedia.org/r/520054

Jdforrester-WMF renamed this task from Upgrade php-ast to 1.0.1 to Upgrade php-ast to 1.0.1 in CI containers.Jul 1 2019, 6:49 PM

@Jdforrester-WMF Thanks, but was there something wrong with the PHP71+ jobs?

@Jdforrester-WMF Thanks, but was there something wrong with the PHP71+ jobs?

Yes, all the PHP71+ jobs are poisoned by the AST upgrade that didn't work, sadly.

@Jdforrester-WMF Thanks, but was there something wrong with the PHP71+ jobs?

Yes, all the PHP71+ jobs are poisoned by the AST upgrade that didn't work, sadly.

Oh :( Is it the same error as T218719#5289432? I thought it was due to ast 1.0.1 being incompatible with PHP70 only, but if the same error appears for 71+ jobs, then there must be another reason...

@Jdforrester-WMF Thanks, but was there something wrong with the PHP71+ jobs?

Yes, all the PHP71+ jobs are poisoned by the AST upgrade that didn't work, sadly.

Oh :( Is it the same error as T218719#5289432? I thought it was due to ast 1.0.1 being incompatible with PHP70 only, but if the same error appears for 71+ jobs, then there must be another reason...

No, any access to ast is broken for PHP71+ jobs as part of the work on this task. :-(

@Jdforrester-WMF Thanks, but was there something wrong with the PHP71+ jobs?

Yes, all the PHP71+ jobs are poisoned by the AST upgrade that didn't work, sadly.

Oh :( Is it the same error as T218719#5289432? I thought it was due to ast 1.0.1 being incompatible with PHP70 only, but if the same error appears for 71+ jobs, then there must be another reason...

No, any access to ast is broken for PHP71+ jobs as part of the work on this task. :-(

Alas! I'm afraid I have no clue then.

Change 520793 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[integration/config@master] docker: Use php-ast 1.0.0 for PHP7.0

https://gerrit.wikimedia.org/r/520793

hashar added a comment.Jul 4 2019, 9:06 PM

OK, there's some PHP version difference that I still need to debug... :|
11:45:35 PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/ast.so' - /usr/lib/php/20151012/ast.so: undefined symbol: zend_string_init_interned in Unknown on line 0

Using docker-registry.wikimedia.org/releng/php-ast:1.0.1 and then running php --version with the ast module loaded, I get the same version for all php versions:

for minor in 0 1 2 3; do
  echo "===== php7.$minor =====";
  php7.$minor -dextension=/srv/modules/ast_7$minor.so --version;
  echo;
done;
php7.0

PHP Warning: PHP Startup: Unable to load dynamic library '/srv/modules/ast_70.so' - /srv/modules/ast_70.so: undefined symbol: zend_string_init_interned in Unknown on line 0
PHP 7.0.33-8+0~20190531121058.14+stretch~1.gbpe7d4ff (cli) (built: May 31 2019 12:10:58) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies

with Zend OPcache v7.0.33-8+0~20190531121058.14+stretch~1.gbpe7d4ff, Copyright (c) 1999-2017, by Zend Technologies
php7.1

PHP Warning: PHP Startup: Unable to load dynamic library '/srv/modules/ast_71.so' - /srv/modules/ast_71.so: undefined symbol: zend_string_init_interned in Unknown on line 0
PHP 7.1.30-1+0~20190531112602.19+stretch~1.gbpab9d28 (cli) (built: May 31 2019 11:26:03) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies

with Zend OPcache v7.1.30-1+0~20190531112602.19+stretch~1.gbpab9d28, Copyright (c) 1999-2018, by Zend Technologies
php7.2

PHP Warning: PHP Startup: Unable to load dynamic library '/srv/modules/ast_72.so' (tried: /srv/modules/ast_72.so (/srv/modules/ast_72.so: undefined symbol: zend_string_init_interned), /usr/lib/php/20170718srv/modules/ast_72.so.so (/usr/lib/php/20170718srv/modules/ast_72.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP 7.2.19-1+0~20190531112637.22+stretch~1.gbp75765b (cli) (built: May 31 2019 11:26:38) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

with Zend OPcache v7.2.19-1+0~20190531112637.22+stretch~1.gbp75765b, Copyright (c) 1999-2018, by Zend Technologies
php7.3

PHP 7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7 (cli) (built: May 31 2019 11:27:35) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.6, Copyright (c) 1998-2018 Zend Technologies

with Zend OPcache v7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7, Copyright (c) 1999-2018, by Zend Technologies

So I guess the php-ast is broken there?

Also, the php version in the phan containers might be slightly different than the one used to compile the extension in the php-ast container. We should probably use a multistage build instead so that when rebuilding the phan container, we get ast compiled with the same php version?

php7.0

PHP Warning: PHP Startup: Unable to load dynamic library '/srv/modules/ast_70.so' - /srv/modules/ast_70.so: undefined symbol: zend_string_init_interned in Unknown on line 0

This one is probably because php-ast 1.0.1 doesn't support PHP7.0 (patch above)

php7.1

PHP Warning: PHP Startup: Unable to load dynamic library '/srv/modules/ast_71.so' - /srv/modules/ast_71.so: undefined symbol: zend_string_init_interned in Unknown on line 0

This one is weird.

php7.2

PHP Warning: PHP Startup: Unable to load dynamic library '/srv/modules/ast_72.so' (tried: /srv/modules/ast_72.so (/srv/modules/ast_72.so: undefined symbol: zend_string_init_interned), /usr/lib/php/20170718srv/modules/ast_72.so.so (/usr/lib/php/20170718srv/modules/ast_72.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0

This one too.

php7.3

PHP 7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7 (cli) (built: May 31 2019 11:27:35) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.6, Copyright (c) 1998-2018 Zend Technologies

with Zend OPcache v7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7, Copyright (c) 1999-2018, by Zend Technologies

Is this actually working?


The only (probably stupid) things I can think of are:

  • It's trying to install a php-ast version incompatible with the used version of PHP. Specifically, is it possible that trying to compile ast 1.0.1 for PHP7.0 (which is incompatible) could result in "persisting" errors?
  • Maybe something screwed up in the php.ini file? Is ast being added correctly?
  • Maybe try to make install?
  • Maybe try to run phpdismod ast to remove the pre-existing version, like it's done in the mediawiki-phan container?
sbassett added a subscriber: sbassett.EditedJul 5 2019, 8:26 PM

Well, I spent too much time playing around with this today. It seems that php might have a problem building out the various versions of php-ast with all of the php7.*-dev deb packages installed at the same time :/ I eventually arrived at this set of Docker dance moves (see local test file of the php-ast dockerfile below), which no longer throws errors for me checking with php7.x --version and php7.x -m inside the container. Haven't tested any actual php-ast-dependent code just yet though. I'd imagine this could be cleaned up a bit, perhaps with a multistage build approach as @hashar mentioned above.

FROM docker-registry.wikimedia.org/releng/sury-php:latest

USER root
RUN apt-get update && apt-get install -y \
	build-essential \
	vim

RUN install --owner=nobody --group=nogroup --directory /srv/php-ast
RUN install --owner=nobody --group=nogroup --directory /srv/modules

USER nobody
RUN git clone https://github.com/nikic/php-ast /srv/php-ast

# php7.0 + ast 1.0.0
USER root
RUN apt-get install -y php7.0-dev
USER nobody

RUN cd /srv/php-ast && \
	git checkout 044436b0870005fcbf0f8eb9b6022d5afe6451bb && \
	git clean -fdx && \
	phpize7.0 && ./configure && make && \
	cp modules/ast.so /srv/modules/ast_70.so

USER root
RUN apt-get purge -y php7.0-dev
USER nobody

# php7.1 + ast 1.0.1
USER root
RUN apt-get install -y php7.1-dev
USER nobody

RUN cd /srv/php-ast && \
	git checkout e2953889168c7724e7a3dd385d279b77780967d9 && \
	git clean -fdx && \
	phpize7.1 && ./configure && make && \
	cp modules/ast.so /srv/modules/ast_71.so

USER root
RUN apt-get purge -y php7.1-dev
USER nobody

# php7.2 + ast 1.0.1
USER root
RUN apt-get install -y php7.2-dev
USER nobody

RUN cd /srv/php-ast && \
	git checkout e2953889168c7724e7a3dd385d279b77780967d9 && \
	git clean -fdx && \
	phpize7.2 && ./configure && make && \
	cp modules/ast.so /srv/modules/ast_72.so

USER root
RUN apt-get purge -y php7.2-dev
USER nobody

# php7.3 + ast 1.0.1
USER root
RUN apt-get install -y php7.3-dev
USER nobody

RUN cd /srv/php-ast && \
	git checkout e2953889168c7724e7a3dd385d279b77780967d9 && \
	git clean -fdx && \
	phpize7.3 && ./configure && make && \
	cp modules/ast.so /srv/modules/ast_73.so

# reinstall php's
USER root
RUN apt-get install -y php7.0-dev php7.1-dev php7.2-dev

# copy modules
RUN cp /srv/modules/ast_70.so /usr/lib/php/20151012/ast.so
RUN cp /srv/modules/ast_71.so /usr/lib/php/20160303/ast.so
RUN cp /srv/modules/ast_72.so /usr/lib/php/20170718/ast.so
RUN cp /srv/modules/ast_73.so /usr/lib/php/20180731/ast.so

Full local rebuild of my PHP images based on https://gerrit.wikimedia.org/r/520793, I get:

PHP70
~/D/c/integration-config   review/daimona_eaytoy/fixAST $±  dockerfiles  ./debug-image php

nobody@8b3f054bf489:/$ php7.0 -dextension=/srv/modules/ast_70.so --version

PHP Warning:  PHP Startup: Unable to load dynamic library '/srv/modules/ast_70.so' - /srv/modules/ast_70.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 7.0.33-0+deb9u3 (cli) (built: Mar  8 2019 10:01:24) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.33-0+deb9u3, Copyright (c) 1999-2017, by Zend Technologies

nobody@8b3f054bf489:/$ ls -la /srv
total 8
drwxr-xr-x 2 root root 4096 Jun 24 06:07 .
drwxr-xr-x 1 root root 4096 Jul  5 23:41 ..
PHP71
~/D/c/integration-config   review/daimona_eaytoy/fixAST $±  dockerfiles  ./debug-image php71

nobody@9242cd61267d:/$ php7.1 -dextension=/srv/modules/ast_71.so --version

PHP Warning:  PHP Startup: Unable to load dynamic library '/srv/modules/ast_71.so' - /srv/modules/ast_71.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 7.1.30-1+0~20190531112602.19+stretch~1.gbpab9d28 (cli) (built: May 31 2019 11:26:03) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.1.30-1+0~20190531112602.19+stretch~1.gbpab9d28, Copyright (c) 1999-2018, by Zend Technologies

nobody@9242cd61267d:/$ ls -la /srv
total 8
drwxr-xr-x 2 root root 4096 Jun 24 06:07 .
drwxr-xr-x 1 root root 4096 Jul  6 00:05 ..
PHP72
~/D/c/integration-config   review/daimona_eaytoy/fixAST $±  dockerfiles  ./debug-image php72

nobody@366a120151e7:/$ php7.2 -dextension=/srv/modules/ast_72.so --version

PHP Warning:  PHP Startup: Unable to load dynamic library '/srv/modules/ast_72.so' (tried: /srv/modules/ast_72.so (/srv/modules/ast_72.so: cannot open shared object file: No such file or directory), /usr/lib/php/20170718//srv/modules/ast_72.so.so (/usr/lib/php/20170718//srv/modules/ast_72.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP 7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1 (cli) (built: Mar 11 2019 11:23:18) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1, Copyright (c) 1999-2018, by Zend Technologies

nobody@366a120151e7:/$ ls -la /srv
total 8
drwxr-xr-x 2 root root 4096 Jun 24 06:07 .
drwxr-xr-x 1 root root 4096 Jul  5 23:36 ..
PHP73
~/D/c/integration-config   review/daimona_eaytoy/fixAST $±  dockerfiles  ./debug-image php73

nobody@d9a6095d56a6:/$ php7.3 -dextension=/srv/modules/ast_73.so --version

PHP Warning:  PHP Startup: Unable to load dynamic library '/srv/modules/ast_73.so' (tried: /srv/modules/ast_73.so (/srv/modules/ast_73.so: cannot open shared object file: No such file or directory), /usr/lib/php/20180731//srv/modules/ast_73.so.so (/usr/lib/php/20180731//srv/modules/ast_73.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP 7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7 (cli) (built: May 31 2019 11:27:35) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.6, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7, Copyright (c) 1999-2018, by Zend Technologies

nobody@d9a6095d56a6:/$ ls -la /srv
total 8
drwxr-xr-x 2 root root 4096 Jun 24 06:07 .
drwxr-xr-x 1 root root 4096 Jul  6 00:11 ..

So this is not working for our PHP versions, whether they are from sury or from SRE.

Change 520961 had a related patch set uploaded (by SBassett; owner: SBassett):
[integration/config@master] Make php-ast docker image build correctly

https://gerrit.wikimedia.org/r/520961

Daimona added a comment.EditedJul 6 2019, 9:25 AM

Great. So, apparently I managed to get this working manually. The web says that zend_string_init_interned is a symbol introduced in PHP7.2, so if the warning appears for PHP7.0 that's because it's being compiled for the wrong version of PHP. And in fact, it seems that running ./configure with a specified PHP config (--with-php-config=php-config7.x) solves the issue. I haven't tested it with any actual code, but it looks very promising. I'm going to update a patch in a few minutes.

Done in https://gerrit.wikimedia.org/r/#/c/integration/config/+/520793/, since that's necessary anyway. @Jdforrester-WMF could you please re-test?

Daimona renamed this task from Upgrade php-ast to 1.0.1 in CI containers to Upgrade php-ast to 1.0.1 in CI composer-package containers.Jul 8 2019, 3:40 PM

Change 520961 abandoned by SBassett:
Make php-ast docker image build correctly

Reason:
In favor of cleaner patch If19c5dd0ecda0074e236d6239300cbcb16011aeb

https://gerrit.wikimedia.org/r/520961

No. With the 386c3dc42d80b9bde35d86f6f91ca3b2e94ae051 rebuild locally, I still get:

~/D/c/integration-config   review/daimona_eaytoy/fixAST $+  dockerfiles  ./debug-image composer-package-php72

nobody@4bbdde6eb38d:/$ php7.2 --version
PHP 7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1 (cli) (built: Mar 11 2019 11:23:18) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1, Copyright (c) 1999-2018, by Zend Technologies

nobody@4bbdde6eb38d:/$ php7.2 -a
Interactive mode enabled

php > print_r( get_loaded_extensions() );
Array
(
    [0] => Core
    [1] => date
    [2] => libxml
    [3] => openssl
    [4] => pcre
    [5] => zlib
    [6] => filter
    [7] => hash
    [8] => pcntl
    [9] => Reflection
    [10] => SPL
    [11] => session
    [12] => sodium
    [13] => standard
    [14] => PDO
    [15] => xml
    [16] => calendar
    [17] => ctype
    [18] => curl
    [19] => dba
    [20] => dom
    [21] => mbstring
    [22] => fileinfo
    [23] => ftp
    [24] => gettext
    [25] => gmp
    [26] => iconv
    [27] => igbinary
    [28] => json
    [29] => exif
    [30] => pdo_sqlite
    [31] => Phar
    [32] => posix
    [33] => readline
    [34] => redis
    [35] => shmop
    [36] => SimpleXML
    [37] => sockets
    [38] => sqlite3
    [39] => sysvmsg
    [40] => sysvsem
    [41] => sysvshm
    [42] => tokenizer
    [43] => wddx
    [44] => xmlreader
    [45] => xmlwriter
    [46] => xsl
    [47] => zip
    [48] => Zend OPcache
)

@Jdforrester-WMF this looks different though, php-ast is compiled with no errors.

What if you invoke php with -dextension=path/to/ast?

Sorry, yes, the "correct" version of AST is installed and working if you manually point to it (at a crummy location):

~/D/c/integration-config   review/daimona_eaytoy/fixAST $±  dockerfiles  ./debug-image composer-package-php72

nobody@6c2f2ee8f2c1:/$ ls /usr/lib/php/201
20151012/ 20170718/

nobody@6c2f2ee8f2c1:/$ ls /usr/lib/php/20170718/ast.so
/usr/lib/php/20170718/ast.so

nobody@6c2f2ee8f2c1:/$ php -dextension=/usr/lib/php/20170718/ast.so --version
PHP 7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1 (cli) (built: Mar 11 2019 11:23:18) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.16-1+0~20190307202415.17+stretch~1.gbpa7be82+wmf1, Copyright (c) 1999-2018, by Zend Technologies

nobody@6c2f2ee8f2c1:/$ php -dextension=/usr/lib/php/20170718/ast.so -a
Interactive mode enabled

php > print_r( get_loaded_extensions() );
Array
(
    [0] => Core
    [1] => date
    [2] => libxml
    [3] => openssl
    [4] => pcre
    [5] => zlib
    [6] => filter
    [7] => hash
    [8] => pcntl
    [9] => Reflection
    [10] => SPL
    [11] => session
    [12] => sodium
    [13] => standard
    [14] => PDO
    [15] => xml
    [16] => calendar
    [17] => ctype
    [18] => curl
    [19] => dba
    [20] => dom
    [21] => mbstring
    [22] => fileinfo
    [23] => ftp
    [24] => gettext
    [25] => gmp
    [26] => iconv
    [27] => igbinary
    [28] => json
    [29] => exif
    [30] => pdo_sqlite
    [31] => Phar
    [32] => posix
    [33] => readline
    [34] => redis
    [35] => shmop
    [36] => SimpleXML
    [37] => sockets
    [38] => sqlite3
    [39] => sysvmsg
    [40] => sysvsem
    [41] => sysvshm
    [42] => tokenizer
    [43] => wddx
    [44] => xmlreader
    [45] => xmlwriter
    [46] => xsl
    [47] => zip
    [48] => ast
    [49] => Zend OPcache
)
php > exit

So… this is working?

Change 520793 merged by jenkins-bot:
[integration/config@master] docker: Use php-ast 1.0.0 for PHP7.0 and unbreak everything

https://gerrit.wikimedia.org/r/520793

Of course, the images that are working are the composer-package* images, but mediawiki-phan and mediawiki-phan-seccheck inherit from composer… Sigh.

Change 521422 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[integration/config@master] jjb: Bump composer-packate to latest

https://gerrit.wikimedia.org/r/521422

@Jdforrester-WMF Cool, thanks! Yes, for those I created T227385, although until we keep installing ast 0.1.2, we cannot fully benefit from the new container. Deploying the jjb patch above is enough to declare this task resolved.

Change 521422 merged by jenkins-bot:
[integration/config@master] jjb: Bump composer-package to latest

https://gerrit.wikimedia.org/r/521422

Jdforrester-WMF closed this task as Resolved.Jul 9 2019, 1:54 PM

Aye, thanks! https://integration.wikimedia.org/ci/job/composer-package-php70-docker/6421/console is our first working test.
And now I'm going to fix all that whitespace mess...