Page MenuHomePhabricator

ORES 500 errors on a threshold lookup request
Closed, ResolvedPublic

Description

https://ores.wikimedia.org/v3/scores/frwiki/?models=damaging&model_info=statistics.thresholds.false.%22maximum+recall+@+precision+%3E=+0.995%22|statistics.thresholds.true.%22maximum+filter_rate+@+recall+%3E=+0.9%22|statistics.thresholds.true.%22maximum+recall+@+precision+%3E=+0.6%22|statistics.thresholds.true.%22maximum+recall+@+precision+%3E=+0.9%22&format=json

currently returns:

{
  "error": {
    "code": "internal server error",
    "message": "Traceback (most recent call last):\n  File \"./ores/wsgi/routes/v3/scores.py\", line 28, in process_score_request\n    score_response = scoring_system.score(score_request)\n  File \"./ores/scoring_systems/scoring_system.py\", line 42, in score\n    response = self._score(request)\n  File \"./ores/scoring_systems/celery_queue.py\", line 192, in _score\n    return super()._score(*args, **kwargs)\n  File \"./ores/scoring_systems/scoring_system.py\", line 61, in _score\n    self._build_model_info(request, response)\n  File \"./ores/scoring_systems/scoring_system.py\", line 122, in _build_model_info\n    context.format_model_info(model_name, request.model_info))\n  File \"./ores/scoring_context.py\", line 39, in format_model_info\n    return model_info.format(paths, formatting=\"json\")\n  File \"/srv/deployment/ores/venv/lib/python3.4/site-packages/revscoring/scoring/model_info.py\", line 106, in format\n    return self.format_json(path_tree, **kwargs)\n  File \"/srv/deployment/ores/venv/lib/python3.4/site-packages/revscoring/scoring/model_info.py\", line 130, in format_json\n    d[key] = key_val.format_json(sub_tree, **kwargs)\n  File \"/srv/deployment/ores/venv/lib/python3.4/site-packages/revscoring/scoring/statistics/classification/classification.py\", line 164, in format_json\n    doc[key] = self[key].format_json(sub_tree, **kwargs)\n  File \"/srv/deployment/ores/venv/lib/python3.4/site-packages/revscoring/scoring/model_info.py\", line 130, in format_json\n    d[key] = key_val.format_json(sub_tree, **kwargs)\n  File \"/srv/deployment/ores/venv/lib/python3.4/site-packages/revscoring/scoring/statistics/classification/scaled_threshold_statistics.py\", line 135, in format_json\n    threshold, pstats = opt.get_optimal(self)\nTypeError: 'NoneType' object is not iterable\n"
  }
}

Event Timeline

Halfak triaged this task as High priority.Nov 3 2017, 5:59 PM

With this change, nulls will appear instead of full-on errors when impossible thresholds are requested. E.g.

$ ./utility model_info ../editquality/models/frwiki.damaging.gradient_boosting.model 'statistics.thresholds.false."maximum recall @ precision >= 0.995"' 'statistics.thresholds.true."maximum recall @ precision >= 0.9"' --format json
{
  "statistics": {
    "thresholds": {
      "false": [
        {
          "threshold": 0.915,
          "match_rate": 0.675,
          "filter_rate": 0.325,
          "precision": 0.995,
          "!precision": 0.079,
          "recall": 0.692,
          "!recall": 0.887,
          "accuracy": 0.697,
          "fpr": 0.113,
          "f1": 0.816,
          "!f1": 0.144
        }
      ],
      "true": [
        null
      ]
    }
  }
}

Mentioned in SAL (#wikimedia-operations) [2017-11-20T21:37:48Z] <awight@tin> Started deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711

Mentioned in SAL (#wikimedia-operations) [2017-11-20T22:27:41Z] <awight@tin> Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 (duration: 49m 54s)

Mentioned in SAL (#wikimedia-operations) [2017-11-20T22:54:20Z] <awight@tin> Started deploy [ores/deploy@5084251]: Rollback ORES; T179711

Mentioned in SAL (#wikimedia-operations) [2017-11-20T22:55:26Z] <awight@tin> Finished deploy [ores/deploy@5084251]: Rollback ORES; T179711 (duration: 01m 05s)

Mentioned in SAL (#wikimedia-operations) [2017-11-27T21:33:09Z] <awight@tin> Started deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450

Mentioned in SAL (#wikimedia-operations) [2017-11-27T21:46:04Z] <awight@tin> Finished deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 12m 55s)

Mentioned in SAL (#wikimedia-operations) [2017-11-27T21:47:57Z] <awight@tin> Started deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450

Mentioned in SAL (#wikimedia-operations) [2017-11-27T21:53:03Z] <awight@tin> Finished deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 05m 06s)

Halfak claimed this task.