Change default serializer of celery from pickle to json
Open, HighPublic

Description

It has been changed to default in celery 4 and has been recommended by celery experts both for security reasons and better debug (pickle is binary)

Ladsgroup created this task.Oct 5 2018, 3:07 PM
Ladsgroup triaged this task as High priority.
Restricted Application added a subscriber: jeblad. · View Herald TranscriptOct 5 2018, 3:07 PM

We need to do this but it got deprioritized over very much needed operational improvements.

Mentioned in SAL (#wikimedia-operations) [2018-11-28T10:11:25Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@9b9ba06]: T206333

Mentioned in SAL (#wikimedia-operations) [2018-11-28T10:26:13Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@9b9ba06]: T206333 (duration: 14m 48s)

The task tracker needs more work:

"Traceback (most recent call last):\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 50, in _reraise_errors\n    yield\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 221, in dumps\n    payload = encoder(data)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 69, in dumps\n    **dict(default_kwargs, **kwargs))\n  File \"/usr/lib/python3.5/json/__init__.py\", line 237, in dumps\n    **kw).encode(obj)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 198, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 256, in iterencode\n    return _iterencode(o, 0)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 59, in default\n    return super(JSONEncoder, self).default(o)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 179, in default\n    raise TypeError(repr(o) + \" is not JSON serializable\")\nTypeError: {<datasource.revision.parent.text>, <datasource.revision.page.namespace.id>, <datasource.revision.user.info.groups>, <datasource.revision.comment>, <datasource.revision.timestamp>, <datasource.revision.user.id>, <datasource.revision.user.info.registration>, <datasource.revision.text>} is not JSON serializable\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"./ores/wsgi/routes/v3/util.py\", line 101, in process_score_request\n    score_response = scoring_system.score(score_request)\n  File \"./ores/scoring_systems/scoring_system.py\", line 59, in score\n    response = self._score(request)\n  File \"./ores/scoring_systems/celery_queue.py\", line 195, in _score\n    return super()._score(*args, **kwargs)\n  File \"./ores/scoring_systems/scoring_system.py\", line 118, in _score\n    inprogress_results=inprogress_results)\n  File \"./ores/scoring_systems/celery_queue.py\", line 100, in _process_missing_scores\n    request.to_json(), list(missing_models), rev_id, root_cache)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/task.py\", line 412, in delay\n    return self.apply_async(args, kwargs)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/task.py\", line 535, in apply_async\n    **options\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/base.py\", line 728, in send_task\n    amqp.send_task_message(P, name, message, **options)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/amqp.py\", line 552, in send_task_message\n    **properties\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/messaging.py\", line 169, in publish\n    compression, headers)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/messaging.py\", line 252, in _prepare\n    body) = dumps(body, serializer=serializer)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 221, in dumps\n    payload = encoder(data)\n  File \"/usr/lib/python3.5/contextlib.py\", line 77, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 54, in _reraise_errors\n    reraise(wrapper, wrapper(exc), sys.exc_info()[2])\n  File \"/srv/ores/venv/lib/python3.5/site-packages/vine/five.py\", line 178, in reraise\n    raise value.with_traceback(tb)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 50, in _reraise_errors\n    yield\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 221, in dumps\n    payload = encoder(data)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 69, in dumps\n    **dict(default_kwargs, **kwargs))\n  File \"/usr/lib/python3.5/json/__init__.py\", line 237, in dumps\n    **kw).encode(obj)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 198, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 256, in iterencode\n    return _iterencode(o, 0)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 59, in default\n    return super(JSONEncoder, self).default(o)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 179, in default\n    raise TypeError(repr(o) + \" is not JSON serializable\")\nkombu.exceptions.EncodeError: {<datasource.revision.parent.text>, <datasource.revision.page.namespace.id>, <datasource.revision.user.info.groups>, <datasource.revision.comment>, <datasource.revision.timestamp>, <datasource.revision.user.id>, <datasource.revision.user.info.registration>, <datasource.revision.text>} is not JSON serializable\n"

but result backend definitely can go json and it actually smaller.

Change 477289 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Use json for result serializer

https://gerrit.wikimedia.org/r/477289

Change 477302 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change result serializer to json

https://gerrit.wikimedia.org/r/477302

Change 477289 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Accept json for result serializer

https://gerrit.wikimedia.org/r/477289

Change 477302 abandoned by Ladsgroup:
ores: Change result serializer to json

Reason:
It's being done through deployment.

https://gerrit.wikimedia.org/r/477302

Mentioned in SAL (#wikimedia-operations) [2018-12-17T11:48:20Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@18d3657]: T206333 T211267

Mentioned in SAL (#wikimedia-operations) [2018-12-17T12:02:34Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@18d3657]: T206333 T211267 (duration: 14m 14s)

the last bit is root_caches which is a dictionary on its own:

{'datasource.revision.text': "Lots of text",
'datasource.revision.user.id': 66,
'datasource.revision.user.info.doc': {'groups': ['extendedconfirmed', '*', 'user', 'autoconfirmed'], 'editcount': 95329, 'registration': '2001-08-21T08:34:39Z', 'name': 'Bryan Derksen', 'gender': 'male', 'userid': 66},
'datasource.revision.comment': 'homo sapiens link',
'datasource.revision.id': 34568,
'datasource.extractor.dependents': {<datasource.revision.text>, <datasource.revision.parent.text>, <datasource.revision.user.id>, <datasource.revision.comment>, <datasource.revision.user.info.registration>, <datasource.revision.timestamp>, <datasource.revision.page.namespace.id>, <datasource.revision.user.info.groups>},
'datasource.revision.doc': {'revid': 34568, 'page': {'ns': 0, 'title': 'Neanderthal', 'pageid': 27298083},'slots': {'main': {'contentformat': 'text/x-wiki', '*': "Lots of text",
'contentmodel': 'wikitext'}},
'user': 'Bryan Derksen', 'timestamp': '2002-03-21T18:03:09Z', 'comment': 'homo sapiens link', 'parentid': 34566, 'size': 2087, 'userid': 66},
'datasource.revision.user.info.registration': Timestamp('2001-08-21T08:34:39Z'),
'datasource.revision.timestamp': Timestamp('2002-03-21T18:03:09Z'),
'datasource.revision.page.namespace.id': 0,
'datasource.revision.parent.doc': {'revid': 34566, 'page': {'ns': 0, 'title': 'Neanderthal', 'pageid': 27298083}, 'slots': {'main': {'contentformat': 'text/x-wiki', '*': "Lots of text"}

Three things here are not json-able: datasource.revision.user.info.registration, datasource.revision.user.info.registration and datasource.extractor.dependents I will give it a try.