Page MenuHomePhabricator
Create Task
Maniphest T206333

Change default serializer of celery from pickle to json
Closed, ResolvedPublic
Actions

Description

It has been changed to default in celery 4 and has been recommended by celery experts both for security reasons and better debug (pickle is binary)

Details

SubjectRepoBranchLines +/-
Drop accepting pickle altogethermediawiki/services/ores/deploymaster+1 -1
Change default task serializer of celery from pickle to jsonmediawiki/services/ores/deploymaster+1 -1
ores: Change result serializer to jsonoperations/puppetproduction+1 -1
ores: Accept json for result serializeroperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ladsgroup triaged this task as High priority.Oct 5 2018, 3:07 PM
Ladsgroup created this task.
Restricted Application added a subscriber: jeblad. · View Herald TranscriptOct 5 2018, 3:07 PM
Ladsgroup closed subtask T206334: Make ScoreReqeust json seriablizable as Resolved.Nov 1 2018, 8:00 PM
Ladsgroup mentioned this in rORES4cc7589bce4d: Add bc layer for ScoreRequest argument in score process task in celery.Nov 2 2018, 2:09 PM
awight mentioned this in rORESe219c807a4c0: Add bc layer for ScoreRequest argument in score process task in celery (#282).Nov 2 2018, 7:40 PM
Ladsgroup mentioned this in rORES636df5ba4724: Send ScoreRequest as JSON over the wire.Nov 5 2018, 1:55 PM
Ladsgroup mentioned this in rORES75ced9b9ceff: Send ScoreRequest as JSON over the wire.Nov 7 2018, 4:00 PM
awight mentioned this in rORES7d9734eacead: Fix ScoreRequest json deserialization in the task (#287).Nov 15 2018, 6:12 PM
RoySmith subscribed.Nov 16 2018, 8:41 PM
Ladsgroup edited projects, added Machine-Learning-Team; removed Machine-Learning-Team (Active Tasks).Nov 19 2018, 12:49 PM

We need to do this but it got deprioritized over very much needed operational improvements.

Ladsgroup mentioned this in rORESde0ea9705015: Send ScoreRequest as json object over the celery wire.Nov 26 2018, 2:50 PM
Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.Nov 26 2018, 5:42 PM
Ladsgroup mentioned this in rORES85e71d781b02: Make several other task arguments json-serializable.Nov 26 2018, 6:53 PM
Ladsgroup mentioned this in rORES50ef503b3394: Make several other task arguments json-serializable.Nov 26 2018, 6:58 PM
awight mentioned this in rORES4ff11a4ecc0c: Make several other task arguments json-serializable (#294).Nov 26 2018, 7:04 PM
Stashbot added a comment.Nov 28 2018, 10:11 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-28T10:11:25Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@9b9ba06]: T206333

Stashbot added a comment.Nov 28 2018, 10:26 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-28T10:26:13Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@9b9ba06]: T206333 (duration: 14m 48s)

Ladsgroup mentioned this in rORES00109199507d: Send missing_models as jsonable object over celery wire.Nov 28 2018, 11:15 AM
Ladsgroup added a comment.Nov 28 2018, 1:25 PM

The task tracker needs more work:

"Traceback (most recent call last):\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 50, in _reraise_errors\n    yield\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 221, in dumps\n    payload = encoder(data)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 69, in dumps\n    **dict(default_kwargs, **kwargs))\n  File \"/usr/lib/python3.5/json/__init__.py\", line 237, in dumps\n    **kw).encode(obj)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 198, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 256, in iterencode\n    return _iterencode(o, 0)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 59, in default\n    return super(JSONEncoder, self).default(o)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 179, in default\n    raise TypeError(repr(o) + \" is not JSON serializable\")\nTypeError: {<datasource.revision.parent.text>, <datasource.revision.page.namespace.id>, <datasource.revision.user.info.groups>, <datasource.revision.comment>, <datasource.revision.timestamp>, <datasource.revision.user.id>, <datasource.revision.user.info.registration>, <datasource.revision.text>} is not JSON serializable\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"./ores/wsgi/routes/v3/util.py\", line 101, in process_score_request\n    score_response = scoring_system.score(score_request)\n  File \"./ores/scoring_systems/scoring_system.py\", line 59, in score\n    response = self._score(request)\n  File \"./ores/scoring_systems/celery_queue.py\", line 195, in _score\n    return super()._score(*args, **kwargs)\n  File \"./ores/scoring_systems/scoring_system.py\", line 118, in _score\n    inprogress_results=inprogress_results)\n  File \"./ores/scoring_systems/celery_queue.py\", line 100, in _process_missing_scores\n    request.to_json(), list(missing_models), rev_id, root_cache)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/task.py\", line 412, in delay\n    return self.apply_async(args, kwargs)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/task.py\", line 535, in apply_async\n    **options\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/base.py\", line 728, in send_task\n    amqp.send_task_message(P, name, message, **options)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/celery/app/amqp.py\", line 552, in send_task_message\n    **properties\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/messaging.py\", line 169, in publish\n    compression, headers)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/messaging.py\", line 252, in _prepare\n    body) = dumps(body, serializer=serializer)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 221, in dumps\n    payload = encoder(data)\n  File \"/usr/lib/python3.5/contextlib.py\", line 77, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 54, in _reraise_errors\n    reraise(wrapper, wrapper(exc), sys.exc_info()[2])\n  File \"/srv/ores/venv/lib/python3.5/site-packages/vine/five.py\", line 178, in reraise\n    raise value.with_traceback(tb)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 50, in _reraise_errors\n    yield\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py\", line 221, in dumps\n    payload = encoder(data)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 69, in dumps\n    **dict(default_kwargs, **kwargs))\n  File \"/usr/lib/python3.5/json/__init__.py\", line 237, in dumps\n    **kw).encode(obj)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 198, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 256, in iterencode\n    return _iterencode(o, 0)\n  File \"/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py\", line 59, in default\n    return super(JSONEncoder, self).default(o)\n  File \"/usr/lib/python3.5/json/encoder.py\", line 179, in default\n    raise TypeError(repr(o) + \" is not JSON serializable\")\nkombu.exceptions.EncodeError: {<datasource.revision.parent.text>, <datasource.revision.page.namespace.id>, <datasource.revision.user.info.groups>, <datasource.revision.comment>, <datasource.revision.timestamp>, <datasource.revision.user.id>, <datasource.revision.user.info.registration>, <datasource.revision.text>} is not JSON serializable\n"

but result backend definitely can go json and it actually smaller.

Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Nov 28 2018, 1:32 PM

https://github.com/wikimedia/ores/pull/297
https://github.com/wikimedia/ores/pull/296

Ladsgroup mentioned this in rORES817915caf495: Change default celery result backend serializer to json.Nov 28 2018, 1:32 PM
awight moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Nov 28 2018, 11:16 PM
gerritbot added a comment.Dec 3 2018, 3:21 PM

Change 477289 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Use json for result serializer

https://gerrit.wikimedia.org/r/477289

gerritbot added a project: Patch-For-Review.Dec 3 2018, 3:21 PM
gerritbot added a comment.Dec 3 2018, 4:44 PM

Change 477302 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] ores: Change result serializer to json

https://gerrit.wikimedia.org/r/477302

gerritbot added a comment.Dec 3 2018, 4:48 PM

Change 477289 merged by Alexandros Kosiaris:
[operations/puppet@production] ores: Accept json for result serializer

https://gerrit.wikimedia.org/r/477289

gerritbot added a comment.Dec 15 2018, 4:41 PM

Change 477302 abandoned by Ladsgroup:
ores: Change result serializer to json

Reason:
It's being done through deployment.

https://gerrit.wikimedia.org/r/477302

Stashbot added a comment.Dec 17 2018, 11:48 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T11:48:20Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@18d3657]: T206333 T211267

Stashbot mentioned this in T211267: ORES precaching seem to not understand its own config.Dec 17 2018, 11:48 AM
Stashbot added a comment.Dec 17 2018, 12:02 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T12:02:34Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@18d3657]: T206333 T211267 (duration: 14m 14s)

Ladsgroup moved this task from Pending deployment to Parked on the Machine-Learning-Team (Active Tasks) board.Dec 17 2018, 12:45 PM
Ladsgroup added a comment.Dec 17 2018, 4:06 PM

the last bit is root_caches which is a dictionary on its own:

{'datasource.revision.text': "Lots of text",
'datasource.revision.user.id': 66,
'datasource.revision.user.info.doc': {'groups': ['extendedconfirmed', '*', 'user', 'autoconfirmed'], 'editcount': 95329, 'registration': '2001-08-21T08:34:39Z', 'name': 'Bryan Derksen', 'gender': 'male', 'userid': 66},
'datasource.revision.comment': 'homo sapiens link',
'datasource.revision.id': 34568,
'datasource.extractor.dependents': {<datasource.revision.text>, <datasource.revision.parent.text>, <datasource.revision.user.id>, <datasource.revision.comment>, <datasource.revision.user.info.registration>, <datasource.revision.timestamp>, <datasource.revision.page.namespace.id>, <datasource.revision.user.info.groups>},
'datasource.revision.doc': {'revid': 34568, 'page': {'ns': 0, 'title': 'Neanderthal', 'pageid': 27298083},'slots': {'main': {'contentformat': 'text/x-wiki', '*': "Lots of text",
'contentmodel': 'wikitext'}},
'user': 'Bryan Derksen', 'timestamp': '2002-03-21T18:03:09Z', 'comment': 'homo sapiens link', 'parentid': 34566, 'size': 2087, 'userid': 66},
'datasource.revision.user.info.registration': Timestamp('2001-08-21T08:34:39Z'),
'datasource.revision.timestamp': Timestamp('2002-03-21T18:03:09Z'),
'datasource.revision.page.namespace.id': 0,
'datasource.revision.parent.doc': {'revid': 34566, 'page': {'ns': 0, 'title': 'Neanderthal', 'pageid': 27298083}, 'slots': {'main': {'contentformat': 'text/x-wiki', '*': "Lots of text"}

Three things here are not json-able: datasource.revision.user.info.registration, datasource.revision.user.info.registration and datasource.extractor.dependents I will give it a try.

Ladsgroup mentioned this in rORESdd96453c102a: [POC] What needs to be done to make tasks json serializeable.Dec 17 2018, 4:44 PM
Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Dec 21 2018, 2:01 PM

https://github.com/wikimedia/revscoring/pull/419

Halfak moved this task from Review to Parked on the Machine-Learning-Team (Active Tasks) board.Jan 7 2019, 5:49 PM

@Ladsgroup looks like this is waiting on you for some work.

Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Jan 18 2019, 2:46 PM

done now

Halfak moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Jan 23 2019, 9:17 PM
Halfak moved this task from Pending deployment to Parked on the Machine-Learning-Team (Active Tasks) board.

Merged the revscoring change, but I moved this back to "Active" because I guess we'll need to test in ORES.

Ladsgroup added a comment.Jan 24 2019, 3:23 PM

I'm rebuilding models of articlequality and put it in another branch, will deploy it on ores-staging and change the config to see how things would work.

Ladsgroup mentioned this in rOEQ7ae65fe0426a: Rebuild models with json-compatible datasources.Jan 28 2019, 1:00 PM
Ladsgroup added a comment.Jan 28 2019, 1:13 PM

I rebuilt all the models, uploaded them on ores-staging and changed the serialization to json, it's still not working. There's something missing here. I'm debugging.

Ladsgroup mentioned this in rOEQ3bcb6377e719: Rebuild bswiki models with revscoring 2.3.1.Jan 30 2019, 10:47 PM
Ladsgroup mentioned this in rOEQ8b49c849fefa: Rebuild bswiki models with revscoring 2.3.1.Jan 30 2019, 10:53 PM
Ladsgroup mentioned this in rOEQ5da5dbd7152f: Rebuild bswiki models with revscoring 1.3.1.Jan 30 2019, 10:57 PM
Ladsgroup added a comment.Jan 30 2019, 11:16 PM

Okay, It seems I made some progress:

Traceback (most recent call last):
  File "./ores/wsgi/routes/v3/util.py", line 101, in process_score_request
    score_response = scoring_system.score(score_request)
  File "./ores/scoring_systems/scoring_system.py", line 59, in score
    response = self._score(request)
  File "./ores/scoring_systems/celery_queue.py", line 196, in _score
    return super()._score(*args, **kwargs)
  File "./ores/scoring_systems/scoring_system.py", line 104, in _score
    request, missing_model_set_revs)
  File "./ores/scoring_systems/scoring_system.py", line 151, in _extract_root_caches
    model_set, rev_ids, injection_caches=request.injection_caches)
  File "./ores/scoring_context.py", line 173, in extract_root_dependency_caches
    for rev_id, (error, _) in zip(rev_ids, error_root_vals):
  File "/srv/ores/venv/lib/python3.5/site-packages/revscoring/extractors/api/extractor.py", line 110, in _extract_many
    if self.revision & all_dependents:
  File "/srv/ores/venv/lib/python3.5/site-packages/revscoring/dependencies/dependent.py", line 150, in __and__
    return self._dependents.union(*self._dependent_sets) & other
TypeError: unsupported operand type(s) for &: 'set' and 'list'
Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Jan 30 2019, 11:25 PM

https://github.com/wikimedia/revscoring/pull/427

Halfak added a comment.Feb 4 2019, 4:39 PM

Notes left on the PR

Ladsgroup mentioned this in rOEQ49d6100601b4: Rebuild bswiki models with revscoring 2.3.2.Feb 4 2019, 8:03 PM
Ladsgroup added a comment.Feb 4 2019, 8:14 PM

After merging the #429 PR on revscoring and releasing 2.3.2 Now I get this on ores-staging:

Traceback (most recent call last):
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py", line 69, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <datasource.revision.comment> is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./ores/wsgi/routes/v3/util.py", line 101, in process_score_request
    score_response = scoring_system.score(score_request)
  File "./ores/scoring_systems/scoring_system.py", line 59, in score
    response = self._score(request)
  File "./ores/scoring_systems/celery_queue.py", line 196, in _score
    return super()._score(*args, **kwargs)
  File "./ores/scoring_systems/scoring_system.py", line 118, in _score
    inprogress_results=inprogress_results)
  File "./ores/scoring_systems/celery_queue.py", line 101, in _process_missing_scores
    request.to_json(), list(missing_models), rev_id, root_cache)
  File "/srv/ores/venv/lib/python3.5/site-packages/celery/app/task.py", line 412, in delay
    return self.apply_async(args, kwargs)
  File "/srv/ores/venv/lib/python3.5/site-packages/celery/app/task.py", line 535, in apply_async
    **options
  File "/srv/ores/venv/lib/python3.5/site-packages/celery/app/base.py", line 728, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/srv/ores/venv/lib/python3.5/site-packages/celery/app/amqp.py", line 552, in send_task_message
    **properties
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/messaging.py", line 169, in publish
    compression, headers)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/messaging.py", line 252, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/srv/ores/venv/lib/python3.5/site-packages/vine/five.py", line 178, in reraise
    raise value.with_traceback(tb)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py", line 69, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/srv/ores/venv/lib/python3.5/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
kombu.exceptions.EncodeError: <datasource.revision.comment> is not JSON serializable

And this is the root_cache datasource:

{'datasource.revision.comment': 'Wiktionary', 'datasource.revision.text': "Like for hr:, I tried to put interwiki between the bosnian version and the others versions, in both direction for de:, en: and fr:. 'Hope it helps and I don't built to much bullshit... [[User:Traroth|Traroth]] 09:58, 12 Sep 2003 (CEST)\r\n\r\nMy other Wikipedia pages :\r\n\r\n[http://meta.wikipedia.org/wiki/User:Traroth Meta]\r\n[http://fr.wikipedia.org/wiki/Utilisateur:Traroth French]\r\n[http://en.wikipedia.org/wiki/User:Traroth English]\r\n[http://de.wikipedia.org/wiki/Benutzer:Traroth German]\r\n[http://hr.wikipedia.org/wiki/User:Traroth Croatian]\r\n[http://cz.wikipedia.org/wiki/Wikipedista:Traroth Czech]\r\n[http://wiktionary.org/wiki/User:Traroth Wiktionary]", 'datasource.revision.doc': {'parentid': 182, 'size': 653, 'page': {'pageid': 120, 'ns': 2, 'title': 'Korisnik:Traroth'}, 'userid': 10, 'comment': 'Wiktionary', 'slots': {'main': {'contentmodel': 'wikitext', 'contentformat': 'text/x-wiki', '*': "Like for hr:, I tried to put interwiki between the bosnian version and the others versions, in both direction for de:, en: and fr:. 'Hope it helps and I don't built to much bullshit... [[User:Traroth|Traroth]] 09:58, 12 Sep 2003 (CEST)\r\n\r\nMy other Wikipedia pages :\r\n\r\n[http://meta.wikipedia.org/wiki/User:Traroth Meta]\r\n[http://fr.wikipedia.org/wiki/Utilisateur:Traroth French]\r\n[http://en.wikipedia.org/wiki/User:Traroth English]\r\n[http://de.wikipedia.org/wiki/Benutzer:Traroth German]\r\n[http://hr.wikipedia.org/wiki/User:Traroth Croatian]\r\n[http://cz.wikipedia.org/wiki/Wikipedista:Traroth Czech]\r\n[http://wiktionary.org/wiki/User:Traroth Wiktionary]"}}, 'timestamp': '2003-09-16T10:57:52Z', 'revid': 212, 'user': 'Traroth'}, 'datasource.extractor.dependents': [<datasource.revision.comment>, <datasource.revision.text>, <datasource.revision.parent.text>, <datasource.revision.user.info.registration_str>, <datasource.revision.page.namespace.id>, <datasource.revision.user.info.groups>, <datasource.revision.timestamp_str>, <datasource.revision.user.id>], 'datasource.revision.parent.text': "Like for hr:, I tried to put interwiki between the bosnian version and the others versions, in both direction for de:, en: and fr:. 'Hope it helps and I don't built to much bullshit... [[User:Traroth|Traroth]] 09:58, 12 Sep 2003 (CEST)\r\n\r\nMy other Wikipedia pages :\r\n\r\n[http://meta.wikipedia.org/wiki/User:Traroth Meta]\r\n[http://fr.wikipedia.org/wiki/Utilisateur:Traroth French]\r\n[http://en.wikipedia.org/wiki/User:Traroth English]\r\n[http://de.wikipedia.org/wiki/Benutzer:Traroth German]\r\n[http://hr.wikipedia.org/wiki/User:Traroth Croatian]\r\n[http://cz.wikipedia.org/wiki/Wikipedista:Traroth Czech]", 'datasource.revision.user.info.registration_str': None, 'datasource.revision.page.namespace.id': 2, 'datasource.revision.parent.doc': {'parentid': 181, 'size': 599, 'page': {'pageid': 120, 'ns': 2, 'title': 'Korisnik:Traroth'}, 'userid': 10, 'comment': '', 'slots': {'main': {'contentmodel': 'wikitext', 'contentformat': 'text/x-wiki', '*': "Like for hr:, I tried to put interwiki between the bosnian version and the others versions, in both direction for de:, en: and fr:. 'Hope it helps and I don't built to much bullshit... [[User:Traroth|Traroth]] 09:58, 12 Sep 2003 (CEST)\r\n\r\nMy other Wikipedia pages :\r\n\r\n[http://meta.wikipedia.org/wiki/User:Traroth Meta]\r\n[http://fr.wikipedia.org/wiki/Utilisateur:Traroth French]\r\n[http://en.wikipedia.org/wiki/User:Traroth English]\r\n[http://de.wikipedia.org/wiki/Benutzer:Traroth German]\r\n[http://hr.wikipedia.org/wiki/User:Traroth Croatian]\r\n[http://cz.wikipedia.org/wiki/Wikipedista:Traroth Czech]"}}, 'timestamp': '2003-09-16T10:21:02Z', 'revid': 182, 'user': 'Traroth'}, 'datasource.revision.user.info.groups': ['*', 'user', 'autoconfirmed'], 'datasource.revision.user.info.doc': {'name': 'Traroth', 'userid': 10, 'gender': 'unknown', 'registration': None, 'editcount': 43, 'groups': ['*', 'user', 'autoconfirmed']}, 'datasource.revision.user.id': 10, 'datasource.revision.timestamp_str': '2003-09-16T10:57:52Z', 'datasource.revision.id': 212}
Ladsgroup added a comment.Feb 4 2019, 8:21 PM

https://github.com/wikimedia/revscoring/pull/430 Should fix it ^ @Halfak I'm not sure if it's correct, please take a look.

RoySmith unsubscribed.Feb 5 2019, 2:45 AM
Ladsgroup mentioned this in rOEQ08e3a726a273: Rebuild bswiki models with revscoring 2.3.2.Feb 6 2019, 11:33 AM
Ladsgroup mentioned this in rOEQ864d01b0abe0: Rebuild bswiki models with revscoring 2.3.2.Feb 6 2019, 11:51 AM
Ladsgroup moved this task from Review to Parked on the Machine-Learning-Team (Active Tasks) board.Feb 6 2019, 12:21 PM

Okay, after rebuilding bswiki models with revscoring 2.3.3, It works. Now I rebuild all models with that.

gerritbot added a comment.Feb 21 2019, 6:16 PM

Change 492013 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/services/ores/deploy@master] Change default task serializer of celery from pickle to json

https://gerrit.wikimedia.org/r/492013

Ladsgroup mentioned this in rORESDEPLOY2d8470984635: Change default task serializer of celery from pickle to json.Feb 21 2019, 6:21 PM
gerritbot added a comment.Feb 21 2019, 6:21 PM

Change 492013 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Change default task serializer of celery from pickle to json

https://gerrit.wikimedia.org/r/492013

Stashbot added a comment.Feb 21 2019, 6:42 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T18:42:36Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@2d84709]: Change default task serializer of celery from pickle to json (T206333)

Stashbot added a comment.Feb 21 2019, 6:59 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T18:59:29Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@2d84709]: Change default task serializer of celery from pickle to json (T206333) (duration: 16m 54s)

Ladsgroup closed subtask T215406: Rebuild all models with revscoring 2.3.3 as Resolved.Feb 21 2019, 7:18 PM
Ladsgroup moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.Feb 21 2019, 7:19 PM
gerritbot added a comment.Feb 21 2019, 7:40 PM

Change 492033 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/services/ores/deploy@master] Drop accepting pickle altogether

https://gerrit.wikimedia.org/r/492033

gerritbot added a comment.Feb 21 2019, 7:44 PM

Change 492033 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Drop accepting pickle altogether

https://gerrit.wikimedia.org/r/492033

Ladsgroup mentioned this in rORESDEPLOYe9c772b439a8: Drop accepting pickle altogether.Feb 21 2019, 7:46 PM
Ladsgroup mentioned this in rORESDEPLOY1ac647a04c3f: Drop accepting pickle altogether.
Stashbot added a comment.Feb 21 2019, 7:52 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T19:52:43Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@5d937b1]: Drop accepting pickle altogether (T206333)

Stashbot added a comment.Feb 21 2019, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-21T20:06:01Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@5d937b1]: Drop accepting pickle altogether (T206333) (duration: 13m 17s)

Ladsgroup closed this task as Resolved.Feb 21 2019, 8:07 PM

It's done \o/

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits