Consistent TimeoutErrors when using Celery 4
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	awight
	Nov 1 2017, 6:17 PM

Description

These errors only occur with "?features=", which disables caching and includes features in the response. I haven't been able to reproduce the issue without "features", even when scoring uncached revisions. The strangest thing about these timeouts is that we see the errors, but no scores:

P6240 (An Untitled Masterwork)

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 class="paste-embed-body" style="max-height: 27.6em;">

1	{ "enwiki": { "models": { "damaging": { "version": "0.4.0" }, "draftquality": { "version": "0.1.0" }, "goodfaith": { "version": "0.4.0" }, "wp10": { "version": "0.6.1" } }, "scores": { "123456": { "damaging": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" }, "features": { "feature.english.badwords.revision.diff.match_delta_decrease": 0, "feature.english.badwords.revision.diff.match_delta_increase": 0, "feature.english.badwords.revision.diff.match_delta_sum": 0, "feature.english.badwords.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_increase": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_sum": 0.0, "feature.english.dictionary.revision.diff.dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.dict_word_delta_increase": 1, "feature.english.dictionary.revision.diff.dict_word_delta_sum": 1, "feature.english.dictionary.revision.diff.dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_increase": 1.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_sum": 1.0, "feature.english.dictionary.revision.diff.non_dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.non_dict_word_delta_increase": 2, "feature.english.dictionary.revision.diff.non_dict_word_delta_sum": 2, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_increase": 2.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_sum": 2.0, "feature.english.informals.revision.diff.match_delta_decrease": 0, "feature.english.informals.revision.diff.match_delta_increase": 0, "feature.english.informals.revision.diff.match_delta_sum": 0, "feature.english.informals.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.informals.revision.diff.match_prop_delta_increase": 0.0, "feature.english.informals.revision.diff.match_prop_delta_sum": 0.0, "feature.len(<datasource.tokenized(datasource.revision.parent.text)>)": 0, "feature.len(<datasource.tokenized(datasource.revision.text)>)": 11, "feature.len(<datasource.wikitext.revision.markups>)": 2, "feature.len(<datasource.wikitext.revision.parent.markups>)": 0, "feature.len(<datasource.wikitext.revision.parent.uppercase_words>)": 0, "feature.len(<datasource.wikitext.revision.parent.words>)": 0, "feature.len(<datasource.wikitext.revision.words>)": 3, "feature.revision.comment.has_link": false, "feature.revision.comment.suggests_section_edit": true, "feature.revision.diff.longest_new_repeated_char": 2, "feature.revision.diff.longest_new_token": 16, "feature.revision.page.is_articleish": false, "feature.revision.page.is_draftspace": false, "feature.revision.page.is_mainspace": false, "feature.revision.user.has_advanced_rights": false, "feature.revision.user.is_admin": false, "feature.revision.user.is_anon": true, "feature.revision.user.is_bot": false, "feature.revision.user.is_curator": false, "feature.revision.user.is_patroller": false, "feature.revision.user.is_trusted": false, "feature.temporal.revision.user.seconds_since_registration": 0, "feature.wikitext.revision.chars": 36, "feature.wikitext.revision.diff.markup_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_delta_sum": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_prop_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_sum": 2.0, "feature.wikitext.revision.diff.number_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_delta_increase": 0.0, "feature.wikitext.revision.diff.number_delta_sum": 0.0, "feature.wikitext.revision.diff.number_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.number_prop_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_sum": 0.0, "feature.wikitext.revision.external_links": 0, "feature.wikitext.revision.headings": 0, "feature.wikitext.revision.parent.chars": 0, "feature.wikitext.revision.parent.external_links": 0, "feature.wikitext.revision.parent.headings": 0, "feature.wikitext.revision.parent.ref_tags": 0, "feature.wikitext.revision.parent.templates": 0, "feature.wikitext.revision.parent.wikilinks": 0, "feature.wikitext.revision.ref_tags": 0, "feature.wikitext.revision.templates": 0, "feature.wikitext.revision.wikilinks": 0 } }, "draftquality": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" } }, "goodfaith": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" }, "features": { "feature.english.badwords.revision.diff.match_delta_decrease": 0, "feature.english.badwords.revision.diff.match_delta_increase": 0, "feature.english.badwords.revision.diff.match_delta_sum": 0, "feature.english.badwords.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_increase": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_sum": 0.0, "feature.english.dictionary.revision.diff.dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.dict_word_delta_increase": 1, "feature.english.dictionary.revision.diff.dict_word_delta_sum": 1, "feature.english.dictionary.revision.diff.dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_increase": 1.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_sum": 1.0, "feature.english.dictionary.revision.diff.non_dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.non_dict_word_delta_increase": 2, "feature.english.dictionary.revision.diff.non_dict_word_delta_sum": 2, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_increase": 2.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_sum": 2.0, "feature.english.informals.revision.diff.match_delta_decrease": 0, "feature.english.informals.revision.diff.match_delta_increase": 0, "feature.english.informals.revision.diff.match_delta_sum": 0, "feature.english.informals.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.informals.revision.diff.match_prop_delta_increase": 0.0, "feature.english.informals.revision.diff.match_prop_delta_sum": 0.0, "feature.len(<datasource.tokenized(datasource.revision.parent.text)>)": 0, "feature.len(<datasource.tokenized(datasource.revision.text)>)": 11, "feature.len(<datasource.wikitext.revision.markups>)": 2, "feature.len(<datasource.wikitext.revision.parent.markups>)": 0, "feature.len(<datasource.wikitext.revision.parent.uppercase_words>)": 0, "feature.len(<datasource.wikitext.revision.parent.words>)": 0, "feature.len(<datasource.wikitext.revision.words>)": 3, "feature.revision.comment.has_link": false, "feature.revision.comment.suggests_section_edit": true, "feature.revision.diff.longest_new_repeated_char": 2, "feature.revision.diff.longest_new_token": 16, "feature.revision.page.is_articleish": false, "feature.revision.page.is_draftspace": false, "feature.revision.page.is_mainspace": false, "feature.revision.user.has_advanced_rights": false, "feature.revision.user.is_admin": false, "feature.revision.user.is_anon": true, "feature.revision.user.is_bot": false, "feature.revision.user.is_curator": false, "feature.revision.user.is_patroller": false, "feature.revision.user.is_trusted": false, "feature.temporal.revision.user.seconds_since_registration": 0, "feature.wikitext.revision.chars": 36, "feature.wikitext.revision.diff.markup_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_delta_sum": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_prop_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_sum": 2.0, "feature.wikitext.revision.diff.number_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_delta_increase": 0.0, "feature.wikitext.revision.diff.number_delta_sum": 0.0, "feature.wikitext.revision.diff.number_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.number_prop_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_sum": 0.0, "feature.wikitext.revision.external_links": 0, "feature.wikitext.revision.headings": 0, "feature.wikitext.revision.parent.chars": 0, "feature.wikitext.revision.parent.external_links": 0, "feature.wikitext.revision.parent.headings": 0, "feature.wikitext.revision.parent.ref_tags": 0, "feature.wikitext.revision.parent.templates": 0, "feature.wikitext.revision.parent.wikilinks": 0, "feature.wikitext.revision.ref_tags": 0, "feature.wikitext.revision.templates": 0, "feature.wikitext.revision.wikilinks": 0 } }, "wp10": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" }, "features": { "feature.english.stemmed.revision.stems_length": 28, "feature.enwiki.main_article_templates": 0, "feature.enwiki.revision.category_links": 0, "feature.enwiki.revision.cite_templates": 0, "feature.enwiki.revision.cn_templates": 0, "feature.enwiki.revision.image_links": 0, "feature.enwiki.revision.infobox_templates": 0, "feature.enwiki.revision.who_templates": 0, "feature.wikitext.revision.chars": 36, "feature.wikitext.revision.content_chars": 36, "feature.wikitext.revision.external_links": 0, "feature.wikitext.revision.headings_by_level(2)": 0, "feature.wikitext.revision.headings_by_level(3)": 0, "feature.wikitext.revision.ref_tags": 0, "feature.wikitext.revision.templates": 0, "feature.wikitext.revision.wikilinks": 0 } } } } } }

{ "enwiki": { "models": { "damaging": { "version": "0.4.0" }, "draftquality": { "version": "0.1.0" }, "goodfaith": { "version": "0.4.0" }, "wp10": { "version": "0.6.1" } }, "scores": { "123456": { "damaging": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" }, "features": { "feature.english.badwords.revision.diff.match_delta_decrease": 0, "feature.english.badwords.revision.diff.match_delta_increase": 0, "feature.english.badwords.revision.diff.match_delta_sum": 0, "feature.english.badwords.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_increase": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_sum": 0.0, "feature.english.dictionary.revision.diff.dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.dict_word_delta_increase": 1, "feature.english.dictionary.revision.diff.dict_word_delta_sum": 1, "feature.english.dictionary.revision.diff.dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_increase": 1.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_sum": 1.0, "feature.english.dictionary.revision.diff.non_dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.non_dict_word_delta_increase": 2, "feature.english.dictionary.revision.diff.non_dict_word_delta_sum": 2, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_increase": 2.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_sum": 2.0, "feature.english.informals.revision.diff.match_delta_decrease": 0, "feature.english.informals.revision.diff.match_delta_increase": 0, "feature.english.informals.revision.diff.match_delta_sum": 0, "feature.english.informals.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.informals.revision.diff.match_prop_delta_increase": 0.0, "feature.english.informals.revision.diff.match_prop_delta_sum": 0.0, "feature.len(<datasource.tokenized(datasource.revision.parent.text)>)": 0, "feature.len(<datasource.tokenized(datasource.revision.text)>)": 11, "feature.len(<datasource.wikitext.revision.markups>)": 2, "feature.len(<datasource.wikitext.revision.parent.markups>)": 0, "feature.len(<datasource.wikitext.revision.parent.uppercase_words>)": 0, "feature.len(<datasource.wikitext.revision.parent.words>)": 0, "feature.len(<datasource.wikitext.revision.words>)": 3, "feature.revision.comment.has_link": false, "feature.revision.comment.suggests_section_edit": true, "feature.revision.diff.longest_new_repeated_char": 2, "feature.revision.diff.longest_new_token": 16, "feature.revision.page.is_articleish": false, "feature.revision.page.is_draftspace": false, "feature.revision.page.is_mainspace": false, "feature.revision.user.has_advanced_rights": false, "feature.revision.user.is_admin": false, "feature.revision.user.is_anon": true, "feature.revision.user.is_bot": false, "feature.revision.user.is_curator": false, "feature.revision.user.is_patroller": false, "feature.revision.user.is_trusted": false, "feature.temporal.revision.user.seconds_since_registration": 0, "feature.wikitext.revision.chars": 36, "feature.wikitext.revision.diff.markup_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_delta_sum": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_prop_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_sum": 2.0, "feature.wikitext.revision.diff.number_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_delta_increase": 0.0, "feature.wikitext.revision.diff.number_delta_sum": 0.0, "feature.wikitext.revision.diff.number_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.number_prop_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_sum": 0.0, "feature.wikitext.revision.external_links": 0, "feature.wikitext.revision.headings": 0, "feature.wikitext.revision.parent.chars": 0, "feature.wikitext.revision.parent.external_links": 0, "feature.wikitext.revision.parent.headings": 0, "feature.wikitext.revision.parent.ref_tags": 0, "feature.wikitext.revision.parent.templates": 0, "feature.wikitext.revision.parent.wikilinks": 0, "feature.wikitext.revision.ref_tags": 0, "feature.wikitext.revision.templates": 0, "feature.wikitext.revision.wikilinks": 0 } }, "draftquality": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" } }, "goodfaith": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" }, "features": { "feature.english.badwords.revision.diff.match_delta_decrease": 0, "feature.english.badwords.revision.diff.match_delta_increase": 0, "feature.english.badwords.revision.diff.match_delta_sum": 0, "feature.english.badwords.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_increase": 0.0, "feature.english.badwords.revision.diff.match_prop_delta_sum": 0.0, "feature.english.dictionary.revision.diff.dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.dict_word_delta_increase": 1, "feature.english.dictionary.revision.diff.dict_word_delta_sum": 1, "feature.english.dictionary.revision.diff.dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_increase": 1.0, "feature.english.dictionary.revision.diff.dict_word_prop_delta_sum": 1.0, "feature.english.dictionary.revision.diff.non_dict_word_delta_decrease": 0, "feature.english.dictionary.revision.diff.non_dict_word_delta_increase": 2, "feature.english.dictionary.revision.diff.non_dict_word_delta_sum": 2, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_decrease": 0.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_increase": 2.0, "feature.english.dictionary.revision.diff.non_dict_word_prop_delta_sum": 2.0, "feature.english.informals.revision.diff.match_delta_decrease": 0, "feature.english.informals.revision.diff.match_delta_increase": 0, "feature.english.informals.revision.diff.match_delta_sum": 0, "feature.english.informals.revision.diff.match_prop_delta_decrease": 0.0, "feature.english.informals.revision.diff.match_prop_delta_increase": 0.0, "feature.english.informals.revision.diff.match_prop_delta_sum": 0.0, "feature.len(<datasource.tokenized(datasource.revision.parent.text)>)": 0, "feature.len(<datasource.tokenized(datasource.revision.text)>)": 11, "feature.len(<datasource.wikitext.revision.markups>)": 2, "feature.len(<datasource.wikitext.revision.parent.markups>)": 0, "feature.len(<datasource.wikitext.revision.parent.uppercase_words>)": 0, "feature.len(<datasource.wikitext.revision.parent.words>)": 0, "feature.len(<datasource.wikitext.revision.words>)": 3, "feature.revision.comment.has_link": false, "feature.revision.comment.suggests_section_edit": true, "feature.revision.diff.longest_new_repeated_char": 2, "feature.revision.diff.longest_new_token": 16, "feature.revision.page.is_articleish": false, "feature.revision.page.is_draftspace": false, "feature.revision.page.is_mainspace": false, "feature.revision.user.has_advanced_rights": false, "feature.revision.user.is_admin": false, "feature.revision.user.is_anon": true, "feature.revision.user.is_bot": false, "feature.revision.user.is_curator": false, "feature.revision.user.is_patroller": false, "feature.revision.user.is_trusted": false, "feature.temporal.revision.user.seconds_since_registration": 0, "feature.wikitext.revision.chars": 36, "feature.wikitext.revision.diff.markup_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_delta_sum": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.markup_prop_delta_increase": 2.0, "feature.wikitext.revision.diff.markup_prop_delta_sum": 2.0, "feature.wikitext.revision.diff.number_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_delta_increase": 0.0, "feature.wikitext.revision.diff.number_delta_sum": 0.0, "feature.wikitext.revision.diff.number_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.number_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.number_prop_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_delta_sum": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_decrease": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_increase": 0.0, "feature.wikitext.revision.diff.uppercase_word_prop_delta_sum": 0.0, "feature.wikitext.revision.external_links": 0, "feature.wikitext.revision.headings": 0, "feature.wikitext.revision.parent.chars": 0, "feature.wikitext.revision.parent.external_links": 0, "feature.wikitext.revision.parent.headings": 0, "feature.wikitext.revision.parent.ref_tags": 0, "feature.wikitext.revision.parent.templates": 0, "feature.wikitext.revision.parent.wikilinks": 0, "feature.wikitext.revision.ref_tags": 0, "feature.wikitext.revision.templates": 0, "feature.wikitext.revision.wikilinks": 0 } }, "wp10": { "error": { "message": "Timed out after 15 seconds.", "type": "TimeoutError" }, "features": { "feature.english.stemmed.revision.stems_length": 28, "feature.enwiki.main_article_templates": 0, "feature.enwiki.revision.category_links": 0, "feature.enwiki.revision.cite_templates": 0, "feature.enwiki.revision.cn_templates": 0, "feature.enwiki.revision.image_links": 0, "feature.enwiki.revision.infobox_templates": 0, "feature.enwiki.revision.who_templates": 0, "feature.wikitext.revision.chars": 36, "feature.wikitext.revision.content_chars": 36, "feature.wikitext.revision.external_links": 0, "feature.wikitext.revision.headings_by_level(2)": 0, "feature.wikitext.revision.headings_by_level(3)": 0, "feature.wikitext.revision.ref_tags": 0, "feature.wikitext.revision.templates": 0, "feature.wikitext.revision.wikilinks": 0 } } } } } }

Reading debug logs and looking in the score cache, the scores are indeed generated and stored.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Bump ores to get new subtask design	mediawiki/services/ores/deploy	CELERY_4	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	awight	T175180 Deploy ORES (revscoring 2.0)
Resolved	Ladsgroup	T178441 Upgrade celery to 4.1.0 for ORES
Resolved	Ladsgroup	T179524 Consistent TimeoutErrors when using Celery 4

Event Timeline

awight created this task.Nov 1 2017, 6:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 1 2017, 6:17 PM

awight added a parent task: T175180: Deploy ORES (revscoring 2.0).Nov 1 2017, 6:18 PM

awight moved this task from Unsorted to Active Tasks on the Machine-Learning-Team board.Nov 1 2017, 6:20 PM

awight edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

awight updated the task description. (Show Details)Nov 1 2017, 6:23 PM

Here are debug logs of failed and succeeded requests, I'm combing through the wdiff now.

failed47 KBDownload

succeeded47 KBDownload

wdiff succeeded failed > wd

wd59 KBDownload

I instrumented the four places we can raise this error, and we're hitting this one, a Celery exception from score_result.get, caught by _process_missing_scores.
https://github.com/wiki-ai/ores/blob/master/ores/scoring_systems/celery_queue.py#L143

Simplifying the query to just one model, https://ores-beta.wmflabs.org/v3/scores/enwiki/123456?features=&models=damaging

failed26 KBDownload

succeeded26 KBDownload

There's a good chance the problem is a race condition. In the successful run, looking at the differing lines we see:

Setting state to SENT (process_score_map)
Received task: ores.scoring_systems.celery_queue._process_score_map
Task accepted: ores.scoring_systems.celery_queue._process_score_map
Setting state to SENT (lookup_score_in_map)
Received task: ores.scoring_systems.celery_queue._lookup_score_in_map
TaskPool: Apply (lookup_score_in_map)
Looking up damaging
Task accepted: ores.scoring_systems.celery_queue._lookup_score_in_map

In the run that times out, the order is:

Setting state to SENT (process_score_map)
Received task: ores.scoring_systems.celery_queue._process_score_map
Setting state to SENT (lookup_score_in_map)
Received task: ores.scoring_systems.celery_queue._lookup_score_in_map
TaskPool: Apply (lookup_score_in_map)
Task accepted: ores.scoring_systems.celery_queue._process_score_map
Looking up damaging
Task accepted: ores.scoring_systems.celery_queue._lookup_score_in_map

awight mentioned this in T178441: Upgrade celery to 4.1.0 for ORES.Nov 1 2017, 10:30 PM

awight renamed this task from Consistent TimeoutErrors affecting ores-beta.wmflabs.org to Consistent TimeoutErrors when using Celery 4.Nov 2 2017, 6:16 PM

awight added a parent task: T178441: Upgrade celery to 4.1.0 for ORES.

The approach I want to try is to coerce the "inprogress" and "missing" child tasks into a Celery native subtask "group".

awight mentioned this in rORES01e7c3d4a5d1: Higher granularity tasks, one for each model.Nov 4 2017, 7:20 PM

I ended up doing something simpler but wasteful. The synchronous subtask step came from sharing feature extraction between multiple models in a scoring request. Now we'll be doing feature extraction twice, but scoring each model in its own task. This burns 2x the CPU by duplicating feature extraction work. We'll want to win back that optimization, but I'm postponing because this smaller code change allows us to move to Celery 4, massively increase the number of workers, and get on the new cluster.

awight mentioned this in rORES516cf868781b: Catch up other scoring systems, fix tests.Nov 4 2017, 9:47 PM

Change 389287 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@CELERY_4] Bump ores to get new subtask design

https://gerrit.wikimedia.org/r/389287

Change 389287 merged by Awight:
[mediawiki/services/ores/deploy@CELERY_4] Bump ores to get new subtask design

https://gerrit.wikimedia.org/r/389287

awight mentioned this in rORESDEPLOY29905e509984: Bump ores to get new subtask design.Nov 4 2017, 11:03 PM

awight mentioned this in rORESbe82121df1a1: Higher granularity tasks, one for each model.Nov 13 2017, 9:10 PM

awight mentioned this in rORES86f94a1bcb68: Catch up other scoring systems, fix tests.

awight moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Nov 13 2017, 10:54 PM

Halfak moved this task from Review to Parked on the Machine-Learning-Team (Active Tasks) board.Nov 20 2017, 5:51 PM

awight edited projects, added Machine-Learning-Team; removed Machine-Learning-Team (Active Tasks).Nov 28 2017, 5:10 PM

awight moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Jun 21 2018, 11:39 AM

Harej triaged this task as High priority.Aug 3 2018, 7:04 PM

There's a Celery bug which seems to map to our issue, but I can't find it in the task history.

When we do an experimental upgrade, we must have monitoring in place to detect these TimeoutErrors.

awight lowered the priority of this task from High to Low.Sep 26 2018, 4:43 PM

awight removed awight as the assignee of this task.Sep 26 2018, 6:28 PM

I'm running celery 4.2 on ores-staging.wmflabs.org and run a hammer test with ?features= on and it worked just fine.

@Ladsgroup That's great news! Note that in T179524#3735543 I disclose that I may have worked around this problem with a barbaric hack.

In T179524#4667923, @awight wrote:

@Ladsgroup That's great news! Note that in T179524#3735543 I disclose that I may have worked around this problem with a barbaric hack.

Well, when we have task tracker in place this should not be needed anymore. (ores-staging is working with the task tracker on)

Ladsgroup claimed this task.Nov 1 2018, 9:35 AM

Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptNov 1 2018, 9:35 AM

Ladsgroup moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.Nov 1 2018, 9:35 AM

Ladsgroup closed this task as Resolved.Nov 1 2018, 8:01 PM