Page MenuHomePhabricator

Wikisource: Reinstate OPDS export
Closed, ResolvedPublic3 Estimated Story Points

Description

We recently (probably when we moved to the VPS?) broke the support for OPDS export. This has only ever been for French Wikisource, and it takes the contents of Catégorie:Bon pour export and builds an OPDS XML file. This file is used by readers in programs such as Calibre and FBReader, and by websites such as https://www.noslivres.net/, to syndicate Wikisource content. The URL of the ODPS file is https://wsexport.wmflabs.org/wikisource-fr-good.atom

It shouldn't be too hard to extend support to other Wikisources (they just need to designate a category to pull from).

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I've manually regenerated the OPDS file and copied it to the above location, so at least things work again for now.

I just had a look at the logs. The OPDS update cron job started to fail in September because of out of memory errors.

The file was generated using a cron job. The cron job would be something like @weekly php /var/www/tool/bin/opds.php /var/www/tool/public/opds/fr.xml

However bin/opds.php is now broken because of the Symphony migration. A frist step would be to update it.

However bin/opds.php is now broken because of the Symphony migration. A frist step would be to update it.

I've brought it up to date (sorry it got forgotten before!): https://github.com/wsexport/tool/pull/284

This new command is ./bin/console app:opds --lang <lang> --category <cat> where lang is the code and cat is the category with no namespace. By default it will output to public/opds/<lang>/<cat>.xml — does that sound ok?

Once that's sorted I'll set up the cronjob, and we can ask other wikis what categories they'd like to export. I wondered about using all equivalents of Catégorie:Bon pour export (via Wikidata), but it might be nice to let wikis ask for whatever category exports they want.

Thank you! It looks like a great plan!

ifried renamed this task from Reinstate OPDS export to Wikisource: Reinstate OPDS export.Dec 10 2020, 5:46 PM
dom_walden subscribed.

I generated the OPDS for Cat%C3%A9gorie:Bon_pour_export on wsexport-test (here: https://wsexport-test.wmflabs.org/opds/fr/Bon_pour_export.xml).

I don't know much about the OPDS format, but it looks the same as the one on production from 23rd Nov. I loaded it into an Atom reader and it didn't seem to have a problem.

ifried subscribed.

I used the FB Reader mobile app on my Android phone to test this, and I was able to generate an OPDS catalog & download books from the catalog. For this reason, I'm marking this work as Done.