Page MenuHomePhabricator

Deploy ORES early Nov 2017
Closed, ResolvedPublic

Event Timeline

Halfak created this task.Nov 6 2017, 5:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2017, 5:17 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-06T21:05:49Z] <ladsgroup@tin> Started deploy [ores/deploy@97a1d80]: Deploying early November (T179837)

Mentioned in SAL (#wikimedia-operations) [2017-11-06T21:16:36Z] <ladsgroup@tin> Finished deploy [ores/deploy@97a1d80]: Deploying early November (T179837) (duration: 10m 47s)

elukey added a subscriber: elukey.Nov 7 2017, 9:30 AM

@Halfak hi! Was the fix for the 500s deployed yesterday? We are still seeing a ton of them in logstash..

I am currently seeing the following on scb1002:

Nov 08 12:00:29 scb1002 systemd[1]: Unit celery-ores-worker.service entered failed state.
Nov 08 12:30:19 scb1002 systemd[1]: Starting Celery workers...
Nov 08 12:30:19 scb1002 systemd[1]: Started Celery workers.
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: Traceback (most recent call last):
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/bin/celery", line 11, in <module>
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: sys.exit(main())
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/__main__.py", lin
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: main()
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/celery.py", l
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: cmd.execute_from_commandline(argv)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/celery.py", l
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: super(CeleryCommand, self).execute_from_commandline(argv)))
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: argv = self.setup_app_from_commandline(argv)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: self.app = self.find_app(app)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return find_app(app, symbol_by_name=self.symbol_by_name)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/app/utils.py", li
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: sym = symbol_by_name(app, imp=imp)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/bin/base.py", lin
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return symbol_by_name(name, imp=imp)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/kombu/utils/__init__.py"
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: module = imp(module_name, package=package, **kwargs)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/celery/utils/imports.py"
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return imp(module, package=package)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/importlib/__init__.py", line 109, in i
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return _bootstrap._gcd_import(name[level:], package, level)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 1129, in _exec
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 1471, in exec_module
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: application = celery.build()
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: config, config['ores']['scoring_system'])
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: config, name, section_key=section_key)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: config, name, section_key=section_key)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: for name in section['scoring_contexts']}
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: for name in section['scoring_contexts']}
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/deploy-cache/revs/82a13ae1173ec570ea563f389ab22e1a69aa154
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: scorer_model = Model.from_config(config, key)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/revscoring/scoring/model
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: Class = yamlconf.import_module(class_path)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/site-packages/yamlconf/import_path.py"
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: module = importlib.import_module(module_path)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "/srv/deployment/ores/venv/lib/python3.4/importlib/__init__.py", line 109, in i
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: return _bootstrap._gcd_import(name[level:], package, level)
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked
Nov 08 12:30:20 scb1002 celery-ores-worker[10247]: ImportError: No module named 'revscoring.scorer_models'
Nov 08 12:30:20 scb1002 systemd[1]: celery-ores-worker.service: main process exited, code=exited, status=1/FAILURE
Nov 08 12:30:20 scb1002 systemd[1]: Unit celery-ores-worker.service entered failed state.

First occurrence of it seems around Nov 06 21:10:21, after the last Ores deployment (at least from what I can see in the SAL). All the other scb hosts are running fine.