SetClaimValueTest.php
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ArthurTaylor
	Apr 26 2024, 10:26 AM

Description

Problem

The Api/EditEntityTest class fails if run in a test suite with Api/SetClaimValueTest before it. 87 tests fail with the message:

Undefined index: %Q42%

Steps to reproduce
In a Mediawiki checkout with the Wikibase extension installed.

Copy phpunit.dist.xml to phpunit.xml
Add a test suite with the following three tests:

<testsuite name="failing_group">
  <file>./extensions/Wikibase/repo/tests/phpunit/includes/Api/SetClaimValueTest.php</file>
  <file>./extensions/Wikibase/repo/tests/phpunit/includes/Api/EditEntityTest.php</file>
</testsuite>

Run the named test suite:

mw docker mediawiki exec -- composer run phpunit:entrypoint -- --testsuite failing_group

Observed behaviour
The test run fails:

$ mw docker mediawiki exec -- composer run phpunit:entrypoint -- --testsuite failing_group
> phpunit '--testsuite' 'failing_group'
Running with MediaWiki settings because there might be integration tests
PHPUnit 9.6.16 by Sebastian Bergmann and contributors.

.........EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE 65 / 96 ( 67%)
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE                                   96 / 96 (100%)

Time: 00:01.157, Memory: 99.00 MB

There were 87 errors:
...

Expected Behaviour
The tests should pass.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Fix test failure in EditEntityTest when run in suite	mediawiki/extensions/Wikibase	master	+9 -10
	Create an extra property in the EditEntityTest	mediawiki/extensions/Wikibase	master	+4 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T361118 [EPIC][Infra] Reduce CI test runtime for Wikibase and related extensions
Resolved	kostajh	T361190 Split PHPUnit test suite for wmf-quibble-vendor-mysql-php74-docker
Resolved	ArthurTaylor	T363557 Api/EditEntityTest.php fails when run in a suite with Api/SetClaimValueTest.php

Event Timeline

ArthurTaylor created this task.Apr 26 2024, 10:26 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 26 2024, 10:26 AM

ArthurTaylor added a parent task: T361190: Split PHPUnit test suite for wmf-quibble-vendor-mysql-php74-docker.Apr 26 2024, 10:26 AM

ArthurTaylor moved this task from In Task Breakdown to In Development on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.

Maintenance_bot moved this task from Incoming to Ongoing on the wmde-wikidata-tech board.Apr 26 2024, 10:29 AM

Maintenance_bot added a project: Wikidata.

WMDE-leszek subscribed.Apr 26 2024, 10:55 AM

Change #1024693 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/Wikibase@master] Create an extra property in the EditEntityTest

https://gerrit.wikimedia.org/r/1024693

gerritbot added a project: Patch-For-Review.Apr 26 2024, 1:59 PM

I had a look at this today. The attached patch does fix the issue, but the reasons are not very satisfying.

The tests fail with Undefined index: %Q42% because the test setup is incomplete. Specifically, the call to $this->initTestEntities fails with a property lookup error - it can't find property P1. It needs property P1, because that is the value of %StringProp%, which is referenced in the Snaks that are created with the Berlin test element. Creating an extra property shifts %StringProp% to P2, and the test passes.

There is clearly some messy state between the two test classes. I suspect the cause is related to the caching of property lookups. P1 is also used in initTestEntities in SetClaimValueTest. But naïvely editing the CachingPropertyInfoLookup to return false all the time didn't immediately resolve the issue. I also tried creating a property in SetClaimValueTest and assigning its ID to %StringProp% in the $idMap for the call their to $this->initTestEntities. This had some effect, but didn't completely solve the issue.

I don't think it makes sense to merge the proposed patch, but having spent a day on the topic I also don't think it makes sense to spend too much longer looking at it. Any insights or ideas about what else we could try would be greatly appreciated.

Change #1025360 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/Wikibase@master] Fix test failure in EditEntityTest when run in suite

https://gerrit.wikimedia.org/r/1025360

Change #1024693 abandoned by Arthur taylor:

[mediawiki/extensions/Wikibase@master] Create an extra property in the EditEntityTest

Reason:

Found a better solution in 1025360

https://gerrit.wikimedia.org/r/1024693

Did some more investigating today. Per the commit message on the gerrit change, I found that the WikibaseApiTestCase::initTestEntities function called from addDBDataOnce in some of the Wikibase Api tests is
executed once per class during setup. Because addDBDataOnce has no corresponding tearDown method, the function was also taking care of deleting some state from previous runs. Unforunately, unless initTestEntities is the very first function called in addDBDataOnce, the tear down as implemented can also delete state from the current class's addDBDataOnce setup, causing it to fail when run with other addDBDataOnce suites.

In addition, the teardown was deleting data from the database, which is unnecessary since the tables are anyway cleared out between test suites. This change moves the teardown to the static tearDownAfterClass PHPUnit helper.

ArthurTaylor moved this task from In Development to Ready for Peer Review on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.Apr 29 2024, 1:59 PM

ArthurTaylor removed ArthurTaylor as the assignee of this task.Apr 30 2024, 12:35 PM

Lucas_Werkmeister_WMDE claimed this task.May 14 2024, 8:59 AM

Lucas_Werkmeister_WMDE moved this task from Ready for Peer Review to In Peer Review on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.

Change #1025360 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Fix test failure in EditEntityTest when run in suite

https://gerrit.wikimedia.org/r/1025360

Maintenance_bot removed a project: Patch-For-Review.May 14 2024, 1:31 PM

Lucas_Werkmeister_WMDE closed this task as Resolved.May 14 2024, 1:50 PM

Lucas_Werkmeister_WMDE reassigned this task from Lucas_Werkmeister_WMDE to ArthurTaylor.

Lucas_Werkmeister_WMDE moved this task from In Peer Review to Done on the [Archived]Wikidata Dev Team (Wikidata.org Slice) board.

Lucas_Werkmeister_WMDE subscribed.