Page MenuHomePhabricator

Wikidata vendor version updates (Feb 2019)
Closed, ResolvedPublic3 Estimated Story Points

Description

From composer outdated, 5 of the results are wikidata related

data-values/data-values               1.1.1   2.1.1    Defines the DataValue interface and some trivial implementations
data-values/geo                       3.0.1   4.1.0    Geographical value objects, parsers and formatters
data-values/serialization             1.2.2   1.2.3    Serializers and deserializers for DataValue implementations
diff/diff                             2.3.0   3.2.0    Small standalone library for representing differences between data structures, computing such differences, and applying th...
wikibase/data-model-services          3.12.0  3.13.0   Services around the Wikibase DataModel

https://packagist.org/packages/wikibase/data-model-services#3.12.0 is also blocking updating wikimedia/assert to 0.4.0 too

I see in Wikibase/composer.json we have:

  • "wikibase/data-model-services": "^3.10.0",, so we can do 3.13.0 matching the version constraints
  • "data-values/data-values": "^2.0.0|^1.0.0", so we can do 2.1.1 matching the version constraints
  • "data-values/serialization": "^1.2.1", so we can do 1.2.3 matching the version constraint

The two conflicting are

  • "diff/diff": "^2.3.0", - We can't bump to 3.0.0 until we drop HHVM on prod (due to php >= 7.0)
  • "data-values/geo": "^3.0.1|^2.1.2", - We can't bump to 3.0.0 until we drop HHVM on prod (due to php >= 7.0)

As such.. Are we able to upgrade these first three, and then do the other two after T176370: Migrate to PHP 7 in WMF production is complete (and diff/diff is bumped in Wikibase's composer.json)?

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I see in Wikibase/composer.json we have:

  • "wikibase/data-model-services": "^3.10.0",, so we can do 3.13.0 matching the version constraints
  • "data-values/data-values": "^2.0.0|^1.0.0", so we can do 2.1.1 matching the version constraints
  • "data-values/serialization": "^1.2.1", so we can do 1.2.3 matching the version constraint

This also needs to be checked with all other extensions.
I wrote a crappy script to do this some time ago, https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/460287/

The result right now is:

> composer-lock-diff --from ./../mw/composer.lock --to ./../current-composer.lock | grep -v "| REMOVED" | grep -v "| NEW" | sed 's/From/Master/g' | sed 's/To/Vendor/g'
+---------------------------------------+---------+---------+---------------------------------------------------------------------------+
| Production Changes                    | Master    | Vendor      | Compare                                                                   |
+---------------------------------------+---------+---------+---------------------------------------------------------------------------+
| data-values/data-values               | 2.1.1   | 1.1.1   | https://github.com/DataValues/DataValues/compare/2.1.1...1.1.1            |
| data-values/serialization             | 1.2.3   | 1.2.2   | https://github.com/DataValues/Serialization/compare/1.2.3...1.2.2         |
| guzzlehttp/psr7                       | 1.5.2   | 1.5.0   | https://github.com/guzzle/psr7/compare/1.5.2...1.5.0                      |
| wikibase/data-model-services          | 3.13.0  | 3.12.0  | https://github.com/wmde/WikibaseDataModelServices/compare/3.13.0...3.12.0 |
+---------------------------------------+---------+---------+---------------------------------------------------------------------------+

Tickets created for the ones we can action now.

"diff/diff": "^2.3.0", - We can't bump to 3.0.0 until we drop HHVM on prod (due to php >= 7.0)
"data-values/geo": "^3.0.1|^2.1.2", - We can't bump to 3.0.0 until we drop HHVM on prod (due to php >= 7.0)

Have been left for now and are blocked by T176370

Addshore triaged this task as Medium priority.Feb 19 2019, 10:23 AM

I see in Wikibase/composer.json we have:

  • "wikibase/data-model-services": "^3.10.0",, so we can do 3.13.0 matching the version constraints
  • "data-values/data-values": "^2.0.0|^1.0.0", so we can do 2.1.1 matching the version constraints
  • "data-values/serialization": "^1.2.1", so we can do 1.2.3 matching the version constraint

This also needs to be checked with all other extensions.
I wrote a crappy script to do this some time ago, https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/460287/

The result right now is:

> composer-lock-diff --from ./../mw/composer.lock --to ./../current-composer.lock | grep -v "| REMOVED" | grep -v "| NEW" | sed 's/From/Master/g' | sed 's/To/Vendor/g'
+---------------------------------------+---------+---------+---------------------------------------------------------------------------+
| Production Changes                    | Master    | Vendor      | Compare                                                                   |
+---------------------------------------+---------+---------+---------------------------------------------------------------------------+
| data-values/data-values               | 2.1.1   | 1.1.1   | https://github.com/DataValues/DataValues/compare/2.1.1...1.1.1            |
| data-values/serialization             | 1.2.3   | 1.2.2   | https://github.com/DataValues/Serialization/compare/1.2.3...1.2.2         |
| guzzlehttp/psr7                       | 1.5.2   | 1.5.0   | https://github.com/guzzle/psr7/compare/1.5.2...1.5.0                      |
| wikibase/data-model-services          | 3.13.0  | 3.12.0  | https://github.com/wmde/WikibaseDataModelServices/compare/3.13.0...3.12.0 |
+---------------------------------------+---------+---------+---------------------------------------------------------------------------+

So basically what I said :p

So basically what I said :p

If "composer outdated" https://getcomposer.org/doc/03-cli.md#outdated will just return the package updates.
The crappy script linked actually checks if the updates are compatible with the collection of extensions that we have installed that also define compatibility with composer packages.
If we blindly trust the outdated command then we will end up occasionally updating packages before all of our extensions are actually ready for the update, likely breaking things.

This time they happened to be the same, but I still wanted to document my process for future linkability and searchability etc (and to show you)

Reedy added a comment.EditedFeb 20 2019, 9:45 AM

So basically what I said :p

If "composer outdated" https://getcomposer.org/doc/03-cli.md#outdated will just return the package updates.
The crappy script linked actually checks if the updates are compatible with the collection of extensions that we have installed that also define compatibility with composer packages.
If we blindly trust the outdated command then we will end up occasionally updating packages before all of our extensions are actually ready for the update, likely breaking things.

This time they happened to be the same, but I still wanted to document my process for future linkability and searchability etc (and to show you)

I know. Which is why I looked more manually at the composer.json, and how/why I said not now for some. But the answer is still the same

All subtasks are resolved. Can this be closed?

Reedy changed the task status from Open to Stalled.Mar 3 2019, 2:11 PM

All subtasks are resolved. Can this be closed?

Nope. @Addshore only created tasks for the immediately actionable items. The other two are basically blocked on T176370. Setting to stalled for now

Addshore renamed this task from Wikidata vendor versions updates (Feb 2019) to Wikidata vendor version updates (Feb 2019).Jun 14 2019, 2:50 PM
Reedy added a comment.Oct 7 2019, 7:48 PM

Should be good to do the PHP 7 bumps now :)

Addshore changed the task status from Stalled to Open.Oct 10 2019, 4:40 PM
Addshore moved this task from blocked on others to ready to go on the Wikidata board.

From the description, the bits we clearly still need to do are:

"diff/diff": "^2.3.0", - We can't bump to 3.0.0 until we drop HHVM on prod (due to php >= 7.0)

"data-values/geo": "^3.0.1|^2.1.2", - We can't bump to 3.0.0 until we drop HHVM on prod (due to php >= 7.0)

There might be some other library upgrades that can happen too (we might want to check)

Reedy added a comment.EditedOct 11 2019, 1:27 PM

HHVM is definitely gone now :)

data-values/common                    0.4.2   0.4.3    Contains common implementations of the interfaces defined by DataValuesInterfaces
data-values/data-values               2.1.1   2.3.0    Defines the DataValue interface and some trivial implementations
data-values/geo                       3.0.1   4.2.0    Geographical value objects, parsers and formatters
diff/diff                             2.3.0   3.2.0    Small standalone library for representing differences between data structures, computing such differences, and applying th...

A quick look suggests the only other updates are a couple of data values

data-values/common is an open patch at https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/525594/ (just rebased and CR+2'd)

Addshore set the point value for this task to 3.Nov 5 2019, 1:12 PM
Addshore moved this task from Ready to estimate to Ready to pick up on the Wikidata-Campsite board.

Change 550665 had a related patch set uploaded (by Rosalie Perside (WMDE); owner: Rosalie Perside (WMDE)):
[mediawiki/vendor@master] Update datavalues/datavalues from 2.1.1 to 2.3.0

https://gerrit.wikimedia.org/r/550665

Change 550666 had a related patch set uploaded (by Rosalie Perside (WMDE); owner: Rosalie Perside (WMDE)):
[mediawiki/vendor@master] Update datavalues/geo from 3.0.1 to 4.2.0

https://gerrit.wikimedia.org/r/550666

Change 550665 merged by jenkins-bot:
[mediawiki/vendor@master] Update datavalues/datavalues from 2.1.1 to 2.3.0

https://gerrit.wikimedia.org/r/550665

It seems the datavalues/geo 4.2.0 release is broken. On trying to update the version in vendor/ it fails with Error: Class 'DataValues\Geo\PackagePrivate\LatLongPrecisionParser' not found .

Reedy added a comment.Nov 15 2019, 8:44 PM

It seems the datavalues/geo 4.2.0 release is broken. On trying to update the version in vendor/ it fails with Error: Class 'DataValues\Geo\PackagePrivate\LatLongPrecisionParser' not found .

That was from a lack of git add :)

The current failure is now:

15:41:57 There was 1 error:
15:41:57 
15:41:57 1) Wikibase\Lib\Tests\Formatters\GlobeCoordinateDetailsFormatterTest::testEscaping
15:41:57 TypeError: Return value of Mock_GlobeCoordinateValue_2ce92031::getPrecision() must be of the type float or null, string returned
15:41:57 
15:41:57 /workspace/src/vendor/data-values/geo/src/Formatters/GlobeCoordinateFormatter.php:50
15:41:57 /workspace/src/extensions/Wikibase/lib/includes/Formatters/GlobeCoordinateDetailsFormatter.php:70
15:41:57 /workspace/src/extensions/Wikibase/lib/tests/phpunit/Formatters/GlobeCoordinateDetailsFormatterTest.php:93
15:41:57 /workspace/src/maintenance/doMaintenance.php:99
15:41:57
Reedy added a comment.Nov 15 2019, 9:02 PM

To be able to use diff/diff ^3.2...

Problem 1
  - wikibase/data-model-services 3.15.0 requires diff/diff ~2.3 -> satisfiable by diff/diff[2.3.0, 2.x-dev] but these conflict with your requirements or minimum-stability.
  - wikibase/data-model-services 3.15.0 requires diff/diff ~2.3 -> satisfiable by diff/diff[2.3.0, 2.x-dev] but these conflict with your requirements or minimum-stability.
  - Installation request for wikibase/data-model-services 3.15.0 -> satisfiable by wikibase/data-model-services[3.15.0].

https://github.com/wmde/WikibaseDataModelServices/issues/222 filed as even master of wikibase/data-model-services does not allow a newer version

To be able to use diff/diff ^3.2...

Problem 1
  - wikibase/data-model-services 3.15.0 requires diff/diff ~2.3 -> satisfiable by diff/diff[2.3.0, 2.x-dev] but these conflict with your requirements or minimum-stability.
  - wikibase/data-model-services 3.15.0 requires diff/diff ~2.3 -> satisfiable by diff/diff[2.3.0, 2.x-dev] but these conflict with your requirements or minimum-stability.
  - Installation request for wikibase/data-model-services 3.15.0 -> satisfiable by wikibase/data-model-services[3.15.0].

https://github.com/wmde/WikibaseDataModelServices/issues/222 filed as even master of wikibase/data-model-services does not allow a newer version

This is done. Now waiting on a new version to be tagged in https://github.com/wmde/WikibaseDataModelServices/issues/225

Reedy added a comment.Nov 22 2019, 2:57 PM

The current failure is now:

15:41:57 There was 1 error:
15:41:57 
15:41:57 1) Wikibase\Lib\Tests\Formatters\GlobeCoordinateDetailsFormatterTest::testEscaping
15:41:57 TypeError: Return value of Mock_GlobeCoordinateValue_2ce92031::getPrecision() must be of the type float or null, string returned
15:41:57 
15:41:57 /workspace/src/vendor/data-values/geo/src/Formatters/GlobeCoordinateFormatter.php:50
15:41:57 /workspace/src/extensions/Wikibase/lib/includes/Formatters/GlobeCoordinateDetailsFormatter.php:70
15:41:57 /workspace/src/extensions/Wikibase/lib/tests/phpunit/Formatters/GlobeCoordinateDetailsFormatterTest.php:93
15:41:57 /workspace/src/maintenance/doMaintenance.php:99
15:41:57

Should I file a separate task about this?

Addshore removed Rosalie_WMDE as the assignee of this task.Dec 10 2019, 11:10 AM
Addshore added a subscriber: Rosalie_WMDE.
Restricted Application added a project: User-Addshore. ยท View Herald TranscriptDec 19 2019, 12:38 PM

I removed my +2 from the 4.2.1 geo patch, as Wikibase isn't yet using 4.2.1 of geo, so mw-vendor shouldn't use it.
Thus, this task is actually complete afaik (it only required the update to 3.

I'll file something else for another batch of updates now etc...

Change 550666 merged by jenkins-bot:
[mediawiki/vendor@master] Update datavalues/geo from 3.0.1 to 4.2.1

https://gerrit.wikimedia.org/r/550666

Reedy added a comment.Dec 24 2019, 1:00 AM

Just diff/diff to go...

Which requires an update from "diff/diff": "^2.3.0", in composer.json for Wikibase

I don't know if/how many changes are needed to Wikibase itself...

Let's create a patch to find out!

Reedy added a comment.Dec 24 2019, 1:04 AM

Aha, forgot about this

Problem 1
  - wikibase/data-model-services 3.15.0 requires diff/diff ~2.3 -> satisfiable by diff/diff[2.3.0, 2.x-dev] but these conflict with your requirements or minimum-stability.
  - wikibase/data-model-services 3.15.0 requires diff/diff ~2.3 -> satisfiable by diff/diff[2.3.0, 2.x-dev] but these conflict with your requirements or minimum-stability.
  - Installation request for wikibase/data-model-services 3.15.0 -> satisfiable by wikibase/data-model-services[3.15.0].

So my request at https://github.com/wmde/WikibaseDataModelServices/issues/225 to do a new release needs to be done, then Wikibase updates as appropriate

And maybe a couple of my other cleanup tasks be done soon... (composer.json and travis stuff)

Change 562569 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/extensions/Wikibase@master] Update diff/diff to 3.2.0 and wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562569

Change 562571 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/vendor@master] Update diff/diff to 3.2.0 and wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562571

Change 562634 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/Wikibase@master] Update use statement for DiffOpChange class

https://gerrit.wikimedia.org/r/562634

Change 562646 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikibaseLexeme@master] Stop using MapDiffer::setComparisonCallback()

https://gerrit.wikimedia.org/r/562646

Change 562634 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Update use statement for DiffOpChange class

https://gerrit.wikimedia.org/r/562634

Change 562649 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikibaseLexeme@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562649

Change 562646 abandoned by Reedy:
Stop using MapDiffer::setComparisonCallback()

https://gerrit.wikimedia.org/r/562646

Change 562657 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/Wikibase@master] Use DiffOpAdd and DiffOpRemove in Diff\DiffOp ns

https://gerrit.wikimedia.org/r/562657

Change 562657 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use DiffOpAdd and DiffOpRemove in Diff\DiffOp ns

https://gerrit.wikimedia.org/r/562657

Change 562571 merged by jenkins-bot:
[mediawiki/vendor@master] Update diff/diff to 3.2.0 and wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562571

Change 562649 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562649

Change 562761 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Update diff/diff to 3.2.0 and wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562761

Change 562762 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/WikibaseQualityConstraints@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562762

Change 562763 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/PropertySuggester@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562763

Change 562761 abandoned by Reedy:
Update diff/diff to 3.2.0 and wikibase/data-model-services to 4.0.0

Reason:
https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/ /562569/

https://gerrit.wikimedia.org/r/562761

Change 562569 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Update diff/diff to 3.2.0 and wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562569

Change 562807 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikibaseMediaInfo@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562807

Change 562808 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikibaseQualityConstraints@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562808

Change 562808 abandoned by Reedy:
Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562808

Change 562807 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562807

Change 562762 merged by Reedy:
[mediawiki/extensions/WikibaseQualityConstraints@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562762

Change 562763 merged by jenkins-bot:
[mediawiki/extensions/PropertySuggester@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/562763

Reedy closed this task as Resolved.Jan 8 2020, 7:21 PM

Yay! This is all done now

There might be a bit of followup/fallout needed to the Wikibase and related extensions

Change 563965 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/WikibaseQualityConstraints@master] Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/563965

Change 563965 abandoned by Addshore:
Update wikibase/data-model-services to 4.0.0

https://gerrit.wikimedia.org/r/563965