Page MenuHomePhabricator
Create Task
Maniphest T203528

Calendar PHP extension needed to run dumpRdf
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Report

Using the mediawiki:1.30 image, I attempted to run dumpRdf.php and got the following error when it reached claims containing dates. It's looking for cal_days_in_month() which I believe requires PHP to be compiled with --enable-calendar. I think this can be achieved in Docker by adding RUN docker-php-ext-install calendar in the dockerfile.

For the record, I was able to run dumpJson.php successfully.

sudo docker-compose exec wikibase php ./extensions/Wikibase/repo/maintenance/dumpRdf.php --output wikibase-dump-2018-09-04.ttl
Dumping shard 0/1
Processed 100 entities.
Processed 200 entities.
Processed 300 entities.
Processed 400 entities.
Processed 500 entities.
Processed 600 entities.
Processed 700 entities.
Processed 800 entities.
Processed 900 entities.
Processed 1000 entities.
Processed 1100 entities.
[09016aa90a6f713af43f7436] [no req]   Error from line 89 of /var/www/html/extensions/Wikibase/repo/includes/Rdf/DateTimeValueCleaner.php: Call to undefined function Wikibase\Rdf\cal_days_in_month()
Backtrace:
#0 /var/www/html/extensions/Wikibase/repo/includes/Rdf/DateTimeValueCleaner.php(46): Wikibase\Rdf\DateTimeValueCleaner->cleanupGregorianValue(string, integer)
#1 /var/www/html/extensions/Wikibase/repo/includes/Rdf/JulianDateTimeValueCleaner.php(43): Wikibase\Rdf\DateTimeValueCleaner->getStandardValue(DataValues\TimeValue)
#2 /var/www/html/extensions/Wikibase/repo/includes/Rdf/Values/TimeRdfBuilder.php(71): Wikibase\Rdf\JulianDateTimeValueCleaner->getStandardValue(DataValues\TimeValue)
#3 /var/www/html/extensions/Wikibase/repo/includes/Rdf/Values/TimeRdfBuilder.php(63): Wikibase\Rdf\Values\TimeRdfBuilder->sayDateLiteral(Wikimedia\Purtle\TurtleRdfWriter, DataValues\TimeValue)
#4 /var/www/html/extensions/Wikibase/repo/includes/Rdf/DispatchingValueSnakRdfBuilder.php(53): Wikibase\Rdf\Values\TimeRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#5 /var/www/html/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(128): Wikibase\Rdf\DispatchingValueSnakRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#6 /var/www/html/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(83): Wikibase\Rdf\SnakRdfBuilder->addSnakValue(Wikimedia\Purtle\TurtleRdfWriter, Wikibase\DataModel\Snak\PropertyValueSnak, string)
#7 /var/www/html/extensions/Wikibase/repo/includes/Rdf/TruthyStatementRdfBuilder.php(85): Wikibase\Rdf\SnakRdfBuilder->addSnak(Wikimedia\Purtle\TurtleRdfWriter, Wikibase\DataModel\Snak\PropertyValueSnak, string)
#8 /var/www/html/extensions/Wikibase/repo/includes/Rdf/TruthyStatementRdfBuilder.php(63): Wikibase\Rdf\TruthyStatementRdfBuilder->addMainSnak(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\Statement)
#9 /var/www/html/extensions/Wikibase/repo/includes/Rdf/TruthyStatementRdfBuilder.php(97): Wikibase\Rdf\TruthyStatementRdfBuilder->addStatements(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\StatementList)
#10 /var/www/html/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(381): Wikibase\Rdf\TruthyStatementRdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#11 /var/www/html/extensions/Wikibase/repo/includes/Dumpers/RdfDumpGenerator.php(151): Wikibase\Rdf\RdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#12 /var/www/html/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(281): Wikibase\Dumpers\RdfDumpGenerator->generateDumpForEntityId(Wikibase\DataModel\Entity\ItemId)
#13 /var/www/html/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(251): Wikibase\Dumpers\DumpGenerator->dumpEntities(array, integer)
#14 /var/www/html/extensions/Wikibase/repo/maintenance/dumpEntities.php(175): Wikibase\Dumpers\DumpGenerator->generateDump(Wikibase\Repo\Store\Sql\SqlEntityIdPager)
#15 /var/www/html/extensions/Wikibase/repo/maintenance/dumpRdf.php(128): Wikibase\DumpScript->execute()
#16 /var/www/html/maintenance/doMaintenance.php(92): Wikibase\DumpRdf->execute()
#17 /var/www/html/extensions/Wikibase/repo/maintenance/dumpRdf.php(177): require_once(string)
#18 {main}

Solution & Acceptance Criteria

Tech Pointers

The Docker files can be found in https://github.com/wmde/wikibase-docker/tree/master/wikibase on github
The mediawiki base image that we use already installs some extensions, you can find an example at https://github.com/wikimedia/mediawiki-docker/blob/master/stable/Dockerfile#L12-L13
The extension needs to be installed for both the 1.29 and 1.30 base images (and 1.31 if it is merged before this ticket is worked on)

Details

SubjectRepoBranchLines +/-
Add calendar extension to suggested packages - composer.jsonmediawiki/extensions/Wikibasemaster+3 -0
Customize query in gerrit

Related Objects

Event Timeline

Elfalem created this task.Sep 5 2018, 2:25 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptSep 5 2018, 2:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore renamed this task from Calendar extension needed to run dumpRdf to Calendar PHP extension needed to run dumpRdf.Sep 5 2018, 7:56 AM
Addshore subscribed.

Link to the method: https://secure.php.net/manual/en/function.cal-days-in-month.php

Addshore moved this task from incoming to consider for next sprint on the Wikidata board.Sep 5 2018, 7:58 AM
Addshore moved this task from Backlog to Bugs on the Wikibase-Docker-2017+ board.
Addshore added a project: Wikidata-Campsite.Sep 5 2018, 8:01 AM
Addshore updated the task description. (Show Details)
Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.

Thanks for the report @Elfalem , this should get picked up next week.

Addshore triaged this task as Medium priority.Sep 5 2018, 8:02 AM
Addshore updated the task description. (Show Details)Sep 5 2018, 8:05 AM
Addshore merged a task: T204538: Wikibase-docker php version missing cal_days_in_month().Sep 17 2018, 3:44 PM
Addshore added a subscriber: Tarrow.
Lucas_Werkmeister_WMDE added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).Sep 18 2018, 1:30 PM
Lucas_Werkmeister_WMDE set the point value for this task to 5.
Addshore moved this task from Ready to estimate to In Progress on the Wikidata-Campsite board.Sep 18 2018, 1:48 PM
Addshore moved this task from consider for next sprint to in progress on the Wikidata board.Sep 18 2018, 2:23 PM
Lucas_Werkmeister_WMDE subscribed.Sep 18 2018, 5:01 PM

Wikibase composer.json could suggest that the PHP extension be installed?

According to the documentation, you can specify extensions in require ("ext-calendar": "*").

Tarrow added a comment.Sep 19 2018, 3:01 PM

https://github.com/wmde/wikibase-docker/pull/49 is a 2 line change to the docker files

Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 19 2018, 3:02 PM
Addshore updated the task description. (Show Details)Sep 19 2018, 3:21 PM

I just merged the PR by @Tarrow completing half of the task, the other is for the extension requirement / suggestion in the composer.json

In T203528#4594689, @Lucas_Werkmeister_WMDE wrote:

Wikibase composer.json could suggest that the PHP extension be installed?

According to the documentation, you can specify extensions in require ("ext-calendar": "*").

I would use suggest instead of require, as some people might not even want to use this maint script.
https://getcomposer.org/doc/04-schema.md#suggest

Addshore moved this task from Doing to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 19 2018, 3:28 PM
Tarrow claimed this task.Sep 19 2018, 4:14 PM

I'll finish what I started but doing the composer.json too

Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 19 2018, 6:22 PM
gerritbot subscribed.Sep 28 2018, 8:35 AM

Change 463435 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[mediawiki/extensions/Wikibase@master] Add calendar extension to suggested packages - composer.json

https://gerrit.wikimedia.org/r/463435

gerritbot added a project: Patch-For-Review.Sep 28 2018, 8:35 AM
Tarrow moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 8 2018, 8:29 AM
Addshore moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 9 2018, 10:15 AM
gerritbot added a comment.Oct 9 2018, 10:56 AM

Change 463435 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add calendar extension to suggested packages - composer.json

https://gerrit.wikimedia.org/r/463435

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)).Oct 9 2018, 11:00 AM
Tarrow updated the task description. (Show Details)Oct 10 2018, 8:53 AM

@Elfalem This should all be fixed now.

Addshore closed this task as Resolved.Oct 12 2018, 8:26 AM
Addshore moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Addshore removed a project: Wikidata-Campsite.Nov 1 2018, 11:22 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits