Page MenuHomePhabricator

ores returns error for draftquality models
Closed, ResolvedPublic

Description

Started from last night: http://ores-beta.wmflabs.org/v3/scores/enwiki/1237

{
  "enwiki": {
    "models": {
      "articlequality": {
        "version": "0.6.1"
      },
      "damaging": {
        "version": "0.4.0"
      },
      "draftquality": {
        "version": "0.1.0"
      },
      "drafttopic": {
        "version": "0.1.0"
      },
      "goodfaith": {
        "version": "0.4.0"
      },
      "wp10": {
        "version": "0.6.1"
      }
    },
    "scores": {
      "1237": {
        "articlequality": {
          "error": {
            "message": "ValueError: Failed to process datasource.english.sentiment.revision.polarity_score: not enough values to unpack (expected 3, got 2)\nTraceback (most recent call last):\n  File \"/srv/deployment/ores/deploy-cache/revs/25dfa4f5488af052237b79bd9f60631268c782f1/venv/lib/python3.5/site-packages/revscoring/dependencies/functions.py\", line 244, in _solve\n    value = dependent(*args)\n  File \"/srv/deployment/ores/deploy-cache/revs/25dfa4f5488af052237b79bd9f60631268c782f1/venv/lib/python3.5/site-packages/revscoring/dependencies/dependent.py\", line 54, in __call__\n    return self.process(*args, **kwargs)\n  File \"./draftquality/feature_lists/enwiki.py\", line 16, in get_polarity_score\n    synsets = list(swn.senti_synsets(t))\n  File \"/srv/deployment/ores/deploy-cache/revs/25dfa4f5488af052237b79bd9f60631268c782f1/venv/lib/python3.5/site-packages/nltk/corpus/reader/sentiwordnet.py\", line 88, in senti_synsets\n    synset_list = wn.synsets(string, pos)\n  File \"/srv/deployment/ores/deploy-cache/revs/25dfa4f5488af052237b79bd9f60631268c782f1/venv/lib/python3.5/site-packages/nltk/corpus/util.py\", line 99, in __getattr__\n    self.__load()\n  File \"/srv/deployment/ores/deploy-cache/revs/25dfa4f5488af052237b79bd9f60631268c782f1/venv/lib/python3.5/site-packages/nltk/corpus/util.py\", line 67, in __load\n    corpus = self.__reader_cls(root, *self.__args, **self.__kwargs)\n  File \"/srv/deployment/ores/deploy-cache/revs/25dfa4f5488af052237b79bd9f60631268c782f1/venv/lib/python3.5/site-packages/nltk/corpus/reader/wordnet.py\", line 1040, in __init__\n    index, lexname, _ = line.split()\nValueError: not enough values to unpack (expected 3, got 2)\n",
            "type": "CaughtDependencyError"
          }
        },

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I have the feeling that rebuilding the wheels with git lfs caused this (that was the only thing that was deployed yesterday)

Git lfs is broken:

ladsgroup@stat1007:~$ git clone ssh://ladsgroup@gerrit.wikimedia.org:29418/research/ores/wheels
.
.
.
ladsgroup@stat1007:~/wheels$ git lfs pull
Git LFS: (0 of 203 files) 0 B / 212.78 MB                                      EGit LFS: (0 of 203 files) 0 B / 212.78 MB

Doesn't download anything. @thcipriani @mmodell

I can download it on my laptop but my connection is super slow but it's not downloading anything on labs nodes or prod stat machines :/

Ladsgroup triaged this task as Unbreak Now! priority.Nov 8 2018, 3:09 PM

Apparently in scap deploy, git lfs pull didn't work on all of wheels assets (only on the *.whl files and not on the corpora text files):

ladsgroup@deployment-ores01:/srv/deployment/ores/deploy/submodules/wheels$ sudo git lfs pull
Git LFS: (55 of 55 files) 96.63 KB / 96.63 KB

That is broken on production now.

Mentioned in SAL (#wikimedia-operations) [2018-11-08T15:20:13Z] <akosiaris> 'cd /srv/deployment/ores/deploy/submodules/wheels && sudo -u deploy-service git lfs pull' on all ores1* and ores2* hosts T209060

Mentioned in SAL (#wikimedia-operations) [2018-11-08T15:25:44Z] <Amir1> rolling restart of celery on ores nodes (T209060)

Ladsgroup lowered the priority of this task from Unbreak Now! to High.Nov 8 2018, 3:30 PM

With the hotfix, it's back to normal but we can't deploy anything because it might break them again.

Change 472650 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[research/ores/wheels@master] Untrack non-wheel files from git lfs

https://gerrit.wikimedia.org/r/472650

Change 472650 merged by Ladsgroup:
[research/ores/wheels@master] Untrack non-wheel files from git lfs

https://gerrit.wikimedia.org/r/472650

Deployed the change on beta, it works. Will deploy it later next week.

Mentioned in SAL (#wikimedia-operations) [2018-11-09T14:48:40Z] <ladsgroup@deploy1001> deploy aborted: T191842 T209060 (duration: 09m 32s)

Mentioned in SAL (#wikimedia-operations) [2018-11-09T14:49:21Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@bb39f4b]: T191842 T209060, try II

Mentioned in SAL (#wikimedia-operations) [2018-11-09T15:04:04Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@bb39f4b]: T191842 T209060, try II (duration: 14m 43s)