Page MenuHomePhabricator

When image info query fails allimages generator fails irrecoverably leaving no continuation token
Closed, DuplicatePublic

Description

A request to the following endpoint fails:

https://commons.wikimedia.org/w/api.php?action=query&generator=allimages&prop=imageinfo&gailimit=40&gaisort=timestamp&gaistart=2019-11-01T00:00:00Z&gaiend=2019-12-01T00:00:00Z&iiprop=url|user|dimensions|extmetadata&iiurlwidth=300&format=json&gaicontinue=20191102113024|Saint-Lys_-_Place_Jean_Moulin_8.jpg

The error message is

{
  "error": {
    "code": "urlparamnormal",
    "info": "Could not normalize image parameters for CADAL08011455_清代学术丛书·第一集·颜氏学记:卷七至卷八.djvu.",
    "*": "See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes."
  },
  "servedby": "mw1231"
}

There was no way to skip the defective data and continue, given this response.

Expected behavior:
Ideally, if there is a problem with one of the pages returned in a generator query, that page would be skipped, allowing other queries to continue. Alternatively, the query could fail, but give a continue token that would allow the user to skip over the defective data themselves.

Event Timeline

Related issues:
T176894
T201205

I think this may actually be a more severe problem, however, since the image info 'subquery' in the generator query doesn't just fail itself, but rather stop the whole query (or string of queries) from succeeding..

@Anomie I respectfully disagree with the classification of this issue as a duplicate. My reasoning is that the issue T201205 is regarding an imageinfo request for a single file. The problem there (as I understand it) is that the apparatus cannot make a thumbnail (or at least this fails some validation) when an image is 0px by 0px. That results in none of the info parameters being retrieved for the image in that case.

In my case, I'm not interested in retrieving the other info parameters for a file if there is no way to create a thumbnail. I'm interested in the generator being able to continue, even if one of the 'subqueries' fails. Recall that a generator is a replacement (as I understand it) for the flow, e.g.,

  1. Retrieve list of images matching some date range
  2. Retrieve information about each image in the list

Using that flow, I could easily skip over any image if it returned the error mentioned in the issue. The problem is that that error is not just returned for a single imageinfo query in the generator function, but rather is the only thing returned at all.

Given that the stated purpose of the generator functionality is to save requests by replacing flows like my example, and that this manifestation of the bug renders the generator functionality unusable for such a use case, I think that this issue is a bit different.

This specific issue could be solved by fixing the bug from T201205, but I think a better solution might be to give the generator functionality the ability to skip over an error in one of the queries in the listed items, perhaps with some logging. In my specific case, I'd rather just see the error mentioned above in place of the image info for that one specific file , rather than as the entire response for the request, e.g., I'd prefer something like

{
  "batchcomplete": "",
  "continue": {
    "gaicontinue": "20191102113052|Zábřeh,_ulice_Kopečná.jpg",
    "continue": "gaicontinue||"
  },
  "query": {
    "pages": {
      "83592743": {
        "pageid": 83592743,
        "ns": 6,
        "title": "File:Portret van een onbekende jongen, RP-P-1931-422.jpg",
        "imagerepository": "local",
        "imageinfo": [ {...successful image info here...} ]
      },
      "83592737": {
        "pageid": 83592737,
        "ns": 6,
        "title": "File:Saint-Lys - Place Jean Moulin 8.jpg",
        "imagerepository": "local",
        "imageinfo": [
          {
            "error": {
              "code": "urlparamnormal",
               "info": "Could not normalize image parameters for CADAL08011455_清代学术丛书·第一集·颜氏学记:卷七至卷八.djvu.",
               "*": "See https://commons.wikimedia.org/w/api.php for API ... and breaking changes."
                },
            "servedby": "mw1231"
          }
        ]
      },
      "83592739": {
        "pageid": 83592739,
        "ns": 6,
        "title": "File:Zábřeh, ulice Komenského III.jpg",
        "imagerepository": "local",
        "imageinfo": [ {...successful image info here...} ]
      }
    }
  }
}

as the response body, since it allows me to simply discard the image whose info couldn't be retrieved. Instead, the error message which is related to the single image is the entire response.

The underlying cause is the same: ApiQueryImageInfo throws an error rather than reporting the failure inline. There's no way to "fix" the generator usage without also fixing that, so there's no point in having separate tasks open.

Changing the error reporting all around instead would complicate client logic to no real benefit.