I understand the need but intuitively I don't see how that fits in the reconciliation service. Am I understanding correctly that this would be a new importer in OpenRefine? You would create a new OpenRefine project by supplying a category, and then you would get a new project with each row corresponding to a file in that category?

This is likely to be work for Joey on the OpenRefine side of things. If listing the files in a category can be done via SPARQL, then maybe this could be made possible by having a SPARQL importer (which is a long-standing feature request): https://github.com/OpenRefine/OpenRefine/issues/1212

Another possibility would be to let people use the "Create project by fetching URL" functionality, and teach users to use it with URLs such that
https://common-recon-service.toolforge.org/fetch_category?name=Category:My_Category
If we manage to document it at the right place then I guess it can be workable (but it's not a super nice integration).

In that case we can indeed add this feature to the recon service, but would just be another feature supported by the webservice outside of the reconciliation API specs.

Thanks for your suggestions @Pintoch 😄

IMHO It's absolutely not necessary for this to be a part of the reconciliation service. It can also very well be an importer indeed. I can update the task description accordingly (and then it's probably more appropriate to move it to GitHub).

In T290089#7330467, @Pintoch wrote:

Am I understanding correctly that this would be a new importer in OpenRefine? You would create a new OpenRefine project by supplying a category, and then you would get a new project with each row corresponding to a file in that category?

From the hypothetical 'typical end user's' point of view, that sounds like the smoothest scenario. It would indeed be most convenient if they can 'feed' a category (or a number of categories; including the option to drill down the category tree for n levels) to OpenRefine, which would then create a project with a file in each row.

If listing the files in a category can be done via SPARQL, then maybe this could be made possible by having a SPARQL importer (which is a long-standing feature request): https://github.com/OpenRefine/OpenRefine/issues/1212

I would start with the assumption that our typical end user doesn't know SPARQL. So this sounds like something nice to have, but probably only for advanced users. I expect regular users to show up with just a list of category names.

In T290089#7330598, @Pintoch wrote:

Another possibility would be to let people use the "Create project by fetching URL" functionality, and teach users to use it with URLs such that
https://common-recon-service.toolforge.org/fetch_category?name=Category:My_Category
If we manage to document it at the right place then I guess it can be workable (but it's not a super nice integration).

In that case we can indeed add this feature to the recon service, but would just be another feature supported by the webservice outside of the reconciliation API specs.

I agree that this could be doable, but also would be a bit convoluted, and probably hard to understand. The 'feed OpenRefine some category names' scenario is by far the easiest for our end users IMO 😎

In T290089#7330815, @Spinster wrote:

From the hypothetical 'typical end user's' point of view, that sounds like the smoothest scenario. It would indeed be most convenient if they can 'feed' a category (or a number of categories; including the option to drill down the category tree for n levels) to OpenRefine, which would then create a project with a file in each row.

I'm particularly fond of the pleasant way in which the ISA Tool does this. The user can specify many different categories there, and can indicate for each category how many levels deep the tool should 'dig up' files. Can be tried here https://isa.toolforge.org/campaigns/create (after logging in)

The 'feed OpenRefine some category names' scenario is by far the easiest for our end users IMO 😎

I do get that, but because this feature is really specific to Commons we need to be a bit cautious about not splashing the UI with something that is only understandable by a fraction of our user base. So far the Wikibase integration has had a relatively minimal impact on the UI for those who do not use it (you only see an extension button on the top right of project screens, the rest of the UI is only brought if you activate the extension).