| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Halfak | T179837 Deploy ORES early Nov 2017 | |||
| Resolved | Halfak | T179712 ORES 500s when model_info lookup fails due to a key error | |||
| Resolved | Halfak | T179711 ORES 500 errors on a threshold lookup request | |||
| Duplicate | None | T179845 Test to see what Extension:ORES will do when it gets this null threshold result. | |||
| Resolved | Halfak | T179838 Update ORES deploy wheels with revscoring 2.0.9 |
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-11-06T21:05:49Z] <ladsgroup@tin> Started deploy [ores/deploy@97a1d80]: Deploying early November (T179837)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-11-06T21:16:36Z] <ladsgroup@tin> Finished deploy [ores/deploy@97a1d80]: Deploying early November (T179837) (duration: 10m 47s)
Comment Actions
@Halfak hi! Was the fix for the 500s deployed yesterday? We are still seeing a ton of them in logstash..
Comment Actions
I am currently seeing the following on scb1002:
Nov 08 12:00:29 scb1002 systemd[1]: Unit celery-ores-worker.service entered failed state. Nov 08 12:30:19 scb1002 systemd[1]: Starting Celery workers... Nov 08 12:30:19 scb1002 systemd[1]: Started Celery workers. Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: Traceback (most recent call last): Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/bin/celery", line 11, in <module> Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: sys.exit(main()) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/__main__.py", lin Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: main() Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/celery.py", l Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: cmd.execute_from_commandline(argv) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/celery.py", l Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: super(CeleryCommand, self).execute_from_commandline(argv))) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: argv = self.setup_app_from_commandline(argv) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: self.app = self.find_app(app) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return find_app(app, symbol_by_name=self.symbol_by_name) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/app/utils.py", li Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: sym = symbol_by_name(app, imp=imp) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return symbol_by_name(name, imp=imp) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/kombu/utils/__init__.py" Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: module = imp(module_name, package=package, **kwargs) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/utils/imports.py" Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return imp(module, package=package) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/importlib/__init__.py", line 109, in i Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return _bootstrap._gcd_import(name[level:], package, level) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2254, in _gcd_import Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2237, in _find_and_load Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 1129, in _exec Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 1471, in exec_module Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154 Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: application = celery.build() Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154 Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: config, config['ores']['scoring_system']) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154 Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: config, name, section_key=section_key) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154 Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: config, name, section_key=section_key) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154 Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: for name in section['scoring_contexts']} Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154 Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: for name in section['scoring_contexts']} Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154 Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: scorer_model = Model.from_config(config, key) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/revscoring/scoring/model Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: Class = yamlconf.import_module(class_path) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/yamlconf/import_path.py" Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: module = importlib.import_module(module_path) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/importlib/__init__.py", line 109, in i Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return _bootstrap._gcd_import(name[level:], package, level) Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2254, in _gcd_import Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2237, in _find_and_load Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: ImportError: No module named 'revscoring.scorer_models' Nov 08 12:30:20 scb1002 systemd[1]: celery-ores-worker.service: main process exited, code=exited, status=1/FAILURE Nov 08 12:30:20 scb1002 systemd[1]: Unit celery-ores-worker.service entered failed state.
First occurrence of it seems around Nov 06 21:10:21, after the last Ores deployment (at least from what I can see in the SAL). All the other scb hosts are running fine.