Page MenuHomePhabricator

Calendar PHP extension needed to run dumpRdf
Closed, ResolvedPublic5 Estimated Story Points

Description

Report

Using the mediawiki:1.30 image, I attempted to run dumpRdf.php and got the following error when it reached claims containing dates. It's looking for cal_days_in_month() which I believe requires PHP to be compiled with --enable-calendar. I think this can be achieved in Docker by adding RUN docker-php-ext-install calendar in the dockerfile.

For the record, I was able to run dumpJson.php successfully.

sudo docker-compose exec wikibase php ./extensions/Wikibase/repo/maintenance/dumpRdf.php --output wikibase-dump-2018-09-04.ttl
Dumping shard 0/1
Processed 100 entities.
Processed 200 entities.
Processed 300 entities.
Processed 400 entities.
Processed 500 entities.
Processed 600 entities.
Processed 700 entities.
Processed 800 entities.
Processed 900 entities.
Processed 1000 entities.
Processed 1100 entities.
[09016aa90a6f713af43f7436] [no req]   Error from line 89 of /var/www/html/extensions/Wikibase/repo/includes/Rdf/DateTimeValueCleaner.php: Call to undefined function Wikibase\Rdf\cal_days_in_month()
Backtrace:
#0 /var/www/html/extensions/Wikibase/repo/includes/Rdf/DateTimeValueCleaner.php(46): Wikibase\Rdf\DateTimeValueCleaner->cleanupGregorianValue(string, integer)
#1 /var/www/html/extensions/Wikibase/repo/includes/Rdf/JulianDateTimeValueCleaner.php(43): Wikibase\Rdf\DateTimeValueCleaner->getStandardValue(DataValues\TimeValue)
#2 /var/www/html/extensions/Wikibase/repo/includes/Rdf/Values/TimeRdfBuilder.php(71): Wikibase\Rdf\JulianDateTimeValueCleaner->getStandardValue(DataValues\TimeValue)
#3 /var/www/html/extensions/Wikibase/repo/includes/Rdf/Values/TimeRdfBuilder.php(63): Wikibase\Rdf\Values\TimeRdfBuilder->sayDateLiteral(Wikimedia\Purtle\TurtleRdfWriter, DataValues\TimeValue)
#4 /var/www/html/extensions/Wikibase/repo/includes/Rdf/DispatchingValueSnakRdfBuilder.php(53): Wikibase\Rdf\Values\TimeRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#5 /var/www/html/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(128): Wikibase\Rdf\DispatchingValueSnakRdfBuilder->addValue(Wikimedia\Purtle\TurtleRdfWriter, string, string, string, Wikibase\DataModel\Snak\PropertyValueSnak)
#6 /var/www/html/extensions/Wikibase/repo/includes/Rdf/SnakRdfBuilder.php(83): Wikibase\Rdf\SnakRdfBuilder->addSnakValue(Wikimedia\Purtle\TurtleRdfWriter, Wikibase\DataModel\Snak\PropertyValueSnak, string)
#7 /var/www/html/extensions/Wikibase/repo/includes/Rdf/TruthyStatementRdfBuilder.php(85): Wikibase\Rdf\SnakRdfBuilder->addSnak(Wikimedia\Purtle\TurtleRdfWriter, Wikibase\DataModel\Snak\PropertyValueSnak, string)
#8 /var/www/html/extensions/Wikibase/repo/includes/Rdf/TruthyStatementRdfBuilder.php(63): Wikibase\Rdf\TruthyStatementRdfBuilder->addMainSnak(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\Statement)
#9 /var/www/html/extensions/Wikibase/repo/includes/Rdf/TruthyStatementRdfBuilder.php(97): Wikibase\Rdf\TruthyStatementRdfBuilder->addStatements(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Statement\StatementList)
#10 /var/www/html/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(381): Wikibase\Rdf\TruthyStatementRdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#11 /var/www/html/extensions/Wikibase/repo/includes/Dumpers/RdfDumpGenerator.php(151): Wikibase\Rdf\RdfBuilder->addEntity(Wikibase\DataModel\Entity\Item)
#12 /var/www/html/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(281): Wikibase\Dumpers\RdfDumpGenerator->generateDumpForEntityId(Wikibase\DataModel\Entity\ItemId)
#13 /var/www/html/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(251): Wikibase\Dumpers\DumpGenerator->dumpEntities(array, integer)
#14 /var/www/html/extensions/Wikibase/repo/maintenance/dumpEntities.php(175): Wikibase\Dumpers\DumpGenerator->generateDump(Wikibase\Repo\Store\Sql\SqlEntityIdPager)
#15 /var/www/html/extensions/Wikibase/repo/maintenance/dumpRdf.php(128): Wikibase\DumpScript->execute()
#16 /var/www/html/maintenance/doMaintenance.php(92): Wikibase\DumpRdf->execute()
#17 /var/www/html/extensions/Wikibase/repo/maintenance/dumpRdf.php(177): require_once(string)
#18 {main}

Solution & Acceptance Criteria

Tech Pointers

The Docker files can be found in https://github.com/wmde/wikibase-docker/tree/master/wikibase on github
The mediawiki base image that we use already installs some extensions, you can find an example at https://github.com/wikimedia/mediawiki-docker/blob/master/stable/Dockerfile#L12-L13
The extension needs to be installed for both the 1.29 and 1.30 base images (and 1.31 if it is merged before this ticket is worked on)

Event Timeline

Elfalem created this task.Sep 5 2018, 2:25 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptSep 5 2018, 2:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore renamed this task from Calendar extension needed to run dumpRdf to Calendar PHP extension needed to run dumpRdf.Sep 5 2018, 7:56 AM
Addshore added a subscriber: Addshore.
Addshore moved this task from Backlog to Bugs on the Wikibase-Containers board.
Addshore updated the task description. (Show Details)
Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.

Thanks for the report @Elfalem , this should get picked up next week.

Addshore triaged this task as Medium priority.Sep 5 2018, 8:02 AM
Addshore updated the task description. (Show Details)Sep 5 2018, 8:05 AM

Wikibase composer.json could suggest that the PHP extension be installed?

According to the documentation, you can specify extensions in require ("ext-calendar": "*").

Addshore updated the task description. (Show Details)Sep 19 2018, 3:21 PM

I just merged the PR by @Tarrow completing half of the task, the other is for the extension requirement / suggestion in the composer.json

Wikibase composer.json could suggest that the PHP extension be installed?

According to the documentation, you can specify extensions in require ("ext-calendar": "*").

I would use suggest instead of require, as some people might not even want to use this maint script.
https://getcomposer.org/doc/04-schema.md#suggest

Tarrow claimed this task.Sep 19 2018, 4:14 PM

I'll finish what I started but doing the composer.json too

Change 463435 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[mediawiki/extensions/Wikibase@master] Add calendar extension to suggested packages - composer.json

https://gerrit.wikimedia.org/r/463435

Change 463435 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add calendar extension to suggested packages - composer.json

https://gerrit.wikimedia.org/r/463435

Tarrow updated the task description. (Show Details)Oct 10 2018, 8:53 AM

@Elfalem This should all be fixed now.