Page MenuHomePhabricator
Create Task
Maniphest T243590

Fix WikibaseDataModel CI for php 7.4
Closed, ResolvedPublic
Actions

Description

The travis CI build for php 7.4 breaks due to phpunit failures.

Apparently, all of the failing checks have to do with serialization / de-serialization:

  1. Wikibase\DataModel\Tests\ReferenceListTest::testGetValueHashIsTheSameForClone
  2. Wikibase\DataModel\Tests\ReferenceListTest::testSerializationStability
  3. Wikibase\DataModel\Tests\ReferenceListTest::testSerializeUnserializeRoundtrip
  4. Wikibase\DataModel\Tests\ReferenceListTest::testUnserializeCreatesNonIdenticalClones
  5. Wikibase\DataModel\Tests\Statement\StatementTest::testSerialize

In particular, assumptions such as:

	$clone = unserialize( serialize( $original ) );

	$this->assertTrue( $original->equals( $clone ) );
	$this->assertSame( $original->getValueHash(), $clone->getValueHash() );

do not seem to be valid any longer, in PHP 7.4

Possible cause: https://github.com/php/php-src/blob/ea1b8788773fe9d5fd517704da332f0725714b8b/UPGRADING#L148 (?)
Possible solution: https://github.com/php/php-src/blob/ea1b8788773fe9d5fd517704da332f0725714b8b/UPGRADING#L332 (?)

Example failures: https://travis-ci.org/wmde/WikibaseDataModel/jobs/641447598

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedhasharT252434 Test MW code in buster as well as stretch
 ResolvedJdforrester-WMFT250514 Create buster-based images for quibble
 ResolvedLadsgroupT279068 Wikibase data-client bridge selenium failure on buster but not stretch
 ResolvedMoritzMuehlenhoffT250515 Please provide our special component/php72 in buster-wikimedia
 ResolvedjijikiT264991 Upgrade the MediaWiki servers to ICU 63
 ResolvedMoritzMuehlenhoffT253377 WMF deployed EasyTimeline extension depends on Ploticus package which is not available in Debian Buster (but available again in Debian Bullseye)
 ResolvedTrizek-WMFT267145 CommRel support for ICU 63 upgrade
 ResolvedDzahnT267248 create mwdebug1003 - ganeti VM with buster and appserver role
 Resolved• GillesT268188 Release WikimediaDebug for mwdebug1003 addition
 ResolvedDzahnT267607 upgrade mwmaint servers to buster
ResolvedRequestPapaulT275928 decommission mwmaint2001.codfw.wmnet
 ResolvedjijikiT268524 Upgrade Parsoid servers to buster
DeclinedNoneT245888 Rename wtp* servers to parse* (Parsoid PHP servers)
 ResolvedDzahnT268248 upgrade scandium to buster
 ResolvedDzahnT265963 Replace production deployment servers and update them to Buster
Unknown Object (Task)
 ResolvedPapaulT266363 (Need By: TBD) rack/setup/install deploy2002
 Resolved• CmjohnsonT265653 (Need By: TBD) setup/install deploy1002
 ResolvedRequest• CmjohnsonT275831 decommission deploy1001.eqiad.wmnet
 ResolvedRequestPapaulT275832 decommission deploy2001.codfw.wmnet
 ResolvedAndrewT269004 Upgrade labweb servers to buster
 ResolvedArielGlennT269377 Upgrade snapshot hosts to Buster
 ResolvedDzahnT270517 Investigate opcache hit rate on Buster appserver
 ResolvedLegoktmT273312 Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade
 ResolvedPapaulT273803 mw2220 - broken IPMI / mgmt
ResolvedDzahnT274023 Convert mwdebug VMs to debian buster
 Declined• GillesT274026 Add 3 new canaries to WikimediaDebug
 ResolvedVolansT274689 sre.hosts.decomission -> generate_dns_snippets - > Cumin execution failed
 ResolvedDzahnT274403 mw1379 - down after reboot attempt and DRAC can't powercycle
 ResolvedLegoktmT275752 Jobrunner timeouts on cross-DC file uploads because of HTTP/2
 ResolvedPleaseStandT291127 MediaWiki session management does not work on split PHP 7.4 and < PHP 7.4 cluster
 ResolvedJoeT293450 Allow coexisting php version in our puppet code
 ResolvedtstarlingT295578 Test running php7.2 and php7.4 in parallel on the beta cluster
 ResolvedArielGlennT295580 Test php7.4 for dumps generation
 ResolvedJoeT293451 Allow sending traffic to php 7.2 or 7.4 selectively in the apache configuration for MediaWiki
 ResolvedtstarlingT263437 Allow easier ICU transitions in MediaWiki (change how sortkey collation is managed in the categorylinks table)
 ResolvedJdforrester-WMFT293924 Also run PHP 7.4 jobs on wmf branch patches
 ResolvedJdforrester-WMFT293851 Re-build all CI images of PHP 7.4 from sury's package to Wikimedia's one, to assure us that it will work
 ResolvedLegoktmT293449 Package php 7.4 for wikimedia production
 ResolvedJdforrester-WMFT295285 Upgrade primary branch of all Wikimedia-deployed repos to a version of mediawiki-tools-phan including T270553
 ResolvedDaimonaT295286 Release mediawiki-tools-phan 0.11.1
 ResolvedDaimonaT270553 Should mediawiki/mediawiki-phan-config ship a target php version to support running with newer php on the developer side?
 ResolvedJdforrester-WMFT314896 WikiEditor edit sampling code triggers 'PHP Deprecated: Invalid characters passed for attempted conversion, these have been ignored' on PHP7.4+
ResolvedJoeT293996 Build MediaWiki images for kubernetes based on php 7.4
 ResolvedJMeybohmT295725 Stop loading wddx PHP extension
 ResolvedJoeT306042 Serve beta cluster via PHP 7.4 by default
 ResolvedJoeT311386 Install php 7.4 in production
 ResolvedJoeT312634 Properly support php7.4 across the observability stack
 ResolvedJoeT313505 Remove prometheus-phpfpm-exporter from physical hosts
 ResolvedKrinkleT311388 Allow assigning each user to a specific php engine by setting a PHP_ENGINE cookie
 ResolvedJoeT311479 Split edge caches between php versions
 ResolvedClement_GoubertT312638 Parsoid migration to php 7.4
 ResolvedClement_GoubertT307219 Put parse parse10[01-24] in production
 ResolvedClement_GoubertT318946 Parsoid migration: Cleanup
 ResolvedMainframe98T312653 Add "php 7.4" option to the Wikimedia Debug extension
 ResolvedJoeT318894 Remove php 7.2 from production

Event Timeline

Addshore created this task.Jan 24 2020, 11:40 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptJan 24 2020, 11:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore triaged this task as Low priority.Jan 24 2020, 11:41 AM
Addshore moved this task from Incoming to Ready to pick up on the Wikidata-Campsite board.
Addshore moved this task from Backlog to Other on the PHP 7.4 support board.
Reedy moved this task from Other to Wikimedia production on the PHP 7.4 support board.Jan 24 2020, 11:54 AM
Reedy subscribed.Jan 24 2020, 12:04 PM

Lots of spam from php_codesniffer too, I guess T243296: Release a new version of WikibaseCodeSniffer has some baring on that

Addshore added a project: good first task.Jan 24 2020, 2:54 PM
Addshore moved this task from incoming to ready to go on the Wikidata board.Jan 24 2020, 3:24 PM
In T243590#5829279, @Reedy wrote:

Lots of spam from php_codesniffer too, I guess T243296: Release a new version of WikibaseCodeSniffer has some baring on that

Yup!

srishakatux subscribed.Jan 25 2020, 2:42 AM

@Addshore Hi! As you've listed this task as a good first task, more details in the task description are welcome. Maybe add where and how one can reproduce this error, link to the codebase, some hints/ideas for fixing the issue, etc. It might be easier that way for a newcomer to get started :)

If you think this task may not be too easy for a newcomer, consider removing the good first task tag! And, same goes for T243588

Addshore removed a project: good first task.Jan 25 2020, 10:11 AM
Addshore lowered the priority of this task from Low to Lowest.Jan 27 2020, 10:32 AM
Addshore moved this task from Ready to pick up to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Silvan_WMDE claimed this task.Jan 27 2020, 2:36 PM
Silvan_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Silvan_WMDE updated the task description. (Show Details)Jan 27 2020, 2:43 PM
Silvan_WMDE updated the task description. (Show Details)Jan 27 2020, 2:47 PM
Silvan_WMDE updated the task description. (Show Details)
Silvan_WMDE removed Silvan_WMDE as the assignee of this task.Jan 29 2020, 11:34 AM
Silvan_WMDE moved this task from Wikidata-Campsite-Iteration-∞ (On Hold) to Needs Tech Work on the Wikidata-Campsite board.
Silvan_WMDE edited projects, added Wikidata-Campsite; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).
Silvan_WMDE subscribed.
Silvan_WMDE unsubscribed.
Jdforrester-WMF subscribed.Mar 12 2020, 2:39 AM

Is this the same set of issues that emit errors in the CI run?

00:18:23.529 There were 3 errors:
00:18:23.529 
00:18:23.529 1) Wikibase\Client\Tests\Api\ApiPropsEntityUsageTest::testEntityUsage with data set "by title" (array('query', 'entityusage', 'Vienna11|Berlin22'), array(array(array(array(array('O', 'S')))), array(array(array(array('S')), array(array('S'))))))
00:18:23.529 === Logs generated by test case
00:18:23.529 [localisation] [debug] LocalisationCache using store LCStoreNull []
00:18:23.529 [localisation] [debug] LocalisationCache using store LCStoreNull []
00:18:23.529 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
00:18:23.529 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
00:18:23.529 ===
00:18:23.529 Trying to access array offset on value of type null
00:18:23.529 
00:18:23.529 /workspace/src/includes/api/ApiPageSet.php:127
00:18:23.529 /workspace/src/extensions/Wikibase/client/tests/phpunit/includes/Api/ApiPropsEntityUsageTest.php:117
00:18:23.529 /workspace/src/extensions/Wikibase/client/tests/phpunit/includes/Api/ApiPropsEntityUsageTest.php:164
00:18:23.529 /workspace/src/extensions/Wikibase/client/tests/phpunit/includes/Api/ApiPropsEntityUsageTest.php:226
00:18:23.529 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:416
00:18:23.529 /workspace/src/maintenance/doMaintenance.php:99
00:18:23.529 
00:18:23.529 2) Wikibase\Client\Tests\Api\ApiPropsEntityUsageTest::testEntityUsage with data set "by entity" (array('query', 'entityusage', 'Vienna11|Berlin22', 'Q3|Q4'), array(array(array(array(array('O', 'S')))), array(array(array(array('S')), array(array('S'))))))
00:18:23.529 === Logs generated by test case
00:18:23.529 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
00:18:23.529 [localisation] [debug] LocalisationCache using store LCStoreNull []
00:18:23.529 [localisation] [debug] LocalisationCache using store LCStoreNull []
00:18:23.529 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
00:18:23.529 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
00:18:23.529 ===
00:18:23.529 Trying to access array offset on value of type null
00:18:23.529 
00:18:23.529 /workspace/src/includes/api/ApiPageSet.php:127
00:18:23.529 /workspace/src/extensions/Wikibase/client/tests/phpunit/includes/Api/ApiPropsEntityUsageTest.php:117
00:18:23.529 /workspace/src/extensions/Wikibase/client/tests/phpunit/includes/Api/ApiPropsEntityUsageTest.php:164
00:18:23.529 /workspace/src/extensions/Wikibase/client/tests/phpunit/includes/Api/ApiPropsEntityUsageTest.php:226
00:18:23.529 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:416
00:18:23.529 /workspace/src/maintenance/doMaintenance.php:99
00:18:23.529 
00:18:23.529 3) Wikibase\Client\Tests\Changes\ChangeRunCoalescerTest::testCoalesceChanges with data set "local link breaks" (array(Wikibase\Lib\Changes\ItemChange Object (...), Wikibase\Lib\Changes\ItemChange Object (...)), array(Wikibase\Lib\Changes\ItemChange Object (...), Wikibase\Lib\Changes\ItemChange Object (...)))
00:18:23.529 === Logs generated by test case
00:18:23.529 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
00:18:23.529 ===
00:18:23.529 Wikibase\Client\Changes\ChangeRunCoalescer::coalesceRuns:array_key_exists(): Using array_key_exists() on objects is deprecated. Use isset() or property_exists() instead [Called from Wikibase\Client\Changes\ChangeRunCoalescer::coalesceRuns in /workspace/src/extensions/Wikibase/client/includes/Changes/ChangeRunCoalescer.php at line 287]
00:18:23.529 
00:18:23.529 /workspace/src/includes/debug/MWDebug.php:333
00:18:23.529 /workspace/src/includes/debug/MWDebug.php:188
00:18:23.529 /workspace/src/includes/GlobalFunctions.php:1068
00:18:23.529 /workspace/src/extensions/Wikibase/client/includes/Changes/ChangeRunCoalescer.php:287
00:18:23.529 /workspace/src/extensions/Wikibase/client/includes/Changes/ChangeRunCoalescer.php:80
00:18:23.529 /workspace/src/extensions/Wikibase/client/tests/phpunit/includes/Changes/ChangeRunCoalescerTest.php:386
00:18:23.529 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:416
00:18:23.529 /workspace/src/maintenance/doMaintenance.php:99
Daimona mentioned this in T247595: Make Wikibase compatible with PHP 7.4.Mar 13 2020, 11:23 AM
Daimona subscribed.Mar 13 2020, 11:28 AM
In T243590#5962809, @Jdforrester-WMF wrote:

Is this the same set of issues that emit errors in the CI run?

Apparently, no. Filed T247595 for that.

Reedy added a parent task: T247658: Make Wikimedia Production MediaWiki compatible with PHP 7.4.Mar 14 2020, 11:33 AM
WMDE-leszek subscribed.Mar 18 2020, 1:07 PM
Reedy added a comment.Apr 24 2020, 3:57 PM

PHP 7.4 is stable, and now in a version of Ubuntu (20.04) that is likely to pick up usage...

Does this make it worth bumping up the priority a bit?

Reedy updated the task description. (Show Details)Apr 24 2020, 3:59 PM
Addshore raised the priority of this task from Lowest to Medium.May 20 2020, 6:51 PM
• Pablo-WMDE subscribed.May 25 2020, 7:33 AM
WMDE-leszek edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.May 25 2020, 9:21 AM
• toan claimed this task.May 25 2020, 9:29 AM
• toan added a comment.May 26 2020, 1:06 PM

Added pull request here -> https://github.com/wmde/WikibaseDataModel/pull/824

• toan moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 26 2020, 1:07 PM
Addshore mentioned this in T253633: LowerCamelFunctionsNameSniff should allow __serialize and __unserialize magic methods for PHP 7.4.May 26 2020, 2:08 PM
Addshore closed this task as Resolved.May 26 2020, 2:33 PM
Addshore moved this task from Peer Review to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Will make the followup described in the PR now

Addshore mentioned this in T253637: Ensure WikibaseDataModel has hash stability tests for all hash methods.May 26 2020, 2:36 PM
Addshore mentioned this in T253737: Release and use wikibase/data-model 9.5.1 everywhere.May 27 2020, 9:42 AM
kostajh reopened this task as Open.Sep 2 2022, 8:58 AM
kostajh subscribed.

This seems to be an issue again now that CI is running PHP 7.4. See https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium-docker/56135/console

09:46:58 There was 1 failure:
09:46:58 
09:46:58 1) Wikibase\DataModel\Tests\ReferenceListTest::testSerializationStability
09:46:58 Failed asserting that two strings are identical.
09:46:58 --- Expected
09:46:58 +++ Actual
09:46:58 @@ @@
09:46:58 -'a:1:{i:0;O:28:"Wikibase\DataModel\Reference":1:{s:35:"Wikibase\DataModel\Referencesnaks";O:32:"Wikibase\DataModel\Snak\SnakList":2:{s:4:"data";a:1:{i:0;C:43:"Wikibase\DataModel\Snak\PropertyNoValueSnak":2:{P1}}s:5:"index";i:0;}}}'
09:46:58 +'a:1:{i:0;O:28:"Wikibase\DataModel\Reference":1:{s:35:"Wikibase\DataModel\Referencesnaks";C:32:"Wikibase\DataModel\Snak\SnakList":100:{a:2:{s:4:"data";a:1:{i:0;C:43:"Wikibase\DataModel\Snak\PropertyNoValueSnak":2:{P1}}s:5:"index";i:0;}}}}'
09:46:58 
09:46:58 /workspace/src/extensions/Wikibase/lib/packages/wikibase/data-model/tests/unit/ReferenceListTest.php:545
09:46:58 
09:46:58 FAILURES!
09:46:58 Tests: 24800, Assertions: 86154, Failures: 1, Skipped: 97.
Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptSep 2 2022, 8:58 AM
hashar added subscribers: Silvan_WMDE, Jakob_WMDE, hashar.Sep 2 2022, 9:09 AM

The Wikibase\DataModel\Tests\ReferenceListTest::testSerializationStability is:

/**
 * This test will change when the serialization format changes.
 * If it is being changed intentionally, the test should be updated.
 * It is just here to catch unintentional changes.
 */
public function testSerializationStability() {
        $list = new ReferenceList();
        $list->addNewReference( new PropertyNoValueSnak( 1 ) );

        /*
         * https://wiki.php.net/rfc/custom_object_serialization
         */
        if ( version_compare( phpversion(), '7.4', '>=' ) ) {
                $testString = "a:1:{i:0;O:28:\"Wikibase\\DataModel\\Reference\":1:{s:35:\"\x00Wikibase\\DataModel\\"
                        . "Reference\x00snaks\";O:32:\"Wikibase\\DataModel\\Snak\\SnakList\":2:{s:4:\""
                        . 'data";a:1:{i:0;C:43:"Wikibase\\DataModel\\Snak\\PropertyNoValueSnak":2:{P1}}s:5'
                        . ':"index";i:0;}}}';
        } else {
                $testString = "a:1:{i:0;O:28:\"Wikibase\\DataModel\\Reference\":1:{s:35:\"\x00Wikibase\\DataModel\\"
                        . "Reference\x00snaks\";C:32:\"Wikibase\\DataModel\\Snak\\SnakList\":100:{a:2:{s:4:\""
                        . 'data";a:1:{i:0;C:43:"Wikibase\\DataModel\\Snak\\PropertyNoValueSnak":2:{P1}}s:5'
                        . ':"index";i:0;}}}}';
        }

        $this->assertSame(
                $testString,
                $list->serialize()
        );
}

It does take in account serialization has changed with php 7.4 and the code is correct. The test fails on our CI because we have patched php7.4 to use the old serialization format, the switch got made as part of upgrading Quibble to 1.4.6 https://gerrit.wikimedia.org/r/c/integration/config/+/828611

The reason we had to patch php 7.4 is the Wikimedia cluster currently runs a mix of php 7.2 and 7.4. Thus when a process runs under php 7.4, the serialized object is stored in the cache with the new format. When a later process running with php 7.2 attempts to unserialize the new format, it fails leading to T316601.

The approach we took was to patch php 7.4 to use the old format as described at T316601#8201209. This way we can finish the migration on the Wikimedia cluster. For this failing test, it means that even if it varies its expectations based on phpversion() >= 7.4, when it runs on our patched php 7.4 the serialization is done with the old format. The test break.

@Jakob_WMDE and @Silvan_WMDE raised it on IRC in #wikimedia-releng. We concluded the easiest course of action right now is to mark the test skipped and restore it once Wikimedia has fully migrated to php 7.4 and that we move the CI images to the original php 7.4 version.

Michael subscribed.Sep 2 2022, 9:58 AM
hashar mentioned this in T316601: PHP Warning: Erroneous data format for unserializing 'Wikimedia\Rdbms\MySQLPrimaryPos'.Sep 2 2022, 10:00 AM
hoo mentioned this in T305785: Wikibase Snak hashes (and thus "mainsnak", "references" and "qualifiers" hashes) depend on legacy PHP serialization.Sep 16 2022, 10:27 PM
Lucas_Werkmeister_WMDE subscribed.Sep 26 2022, 1:39 PM
Lucas_Werkmeister_WMDE mentioned this in T318918: Undeploy patch to use old PHP serialization in PHP 7.4.Sep 29 2022, 10:54 AM
Lucas_Werkmeister_WMDE closed this task as Resolved.Oct 6 2022, 2:32 PM
Lucas_Werkmeister_WMDE closed subtask T316923: Restore skipped test in ReferenceListTest.php as Resolved.

I think this is done now.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL