Page MenuHomePhabricator

Failed executing job: AssembleUploadChunks "Disallowed by AbuseFilter"
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error

Request ID: W6KknQrAEHIAAHs3YOAAAADC

message
Failed executing job: AssembleUploadChunks File:###.##.webm filekey=###.##.webm filename=###.webm requestId=W6I1yQrAAC4AAAPF@1QAAAAD session={"headers":"array(...)","ip":"#.#.#.#","sessionId":"###","userId":#}
Sample 1
{| style="width:100%; padding:1px 4px; border:1px solid #bb7070; background:#ffdbdb;"
| style="text-align:center; width:60px;" |[[File:Dialog-error.svg|50px|link=|Your action has triggered the Abuse Filter]]
| An automated filter has identified this edit as potentially unconstructive, and it has been disallowed. If this edit is constructive, please [[Commons:Abuse filter#False positives|report this error]].
|}
Sample 2
'''This upload has been automatically identified as harmful and has been disabled.'''

Notes

Logstash has recorded this job as failing 112 times in the last 30 days, since at least 1.32.0-wmf.16, all of which cary the wiki: commonswiki tag.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

This seems to be due to a flaky AbuseFilter on commons. However, I don't know what this job is supposed to do and why such filter is triggered.

Happened 11 times in the last 30 days, and now we have the filter ID. Given that it is public, I'm gonna say that this happen due to filter 192. Shares the cause - and possibly the solution - with T212082.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM

Recent occurrences are indeed all caused by filter 192, according to the filter's purpose.
So... given that these failures are actually valid and intentional, do we want to keep them, or treat AbuseFilter rejections as "not an error" (even though the actual goal was not achieved)?

@matthiasmullie AbuseFilters should not apply during job invocations, however bypassing must be done with care to ensure the filters are accomodated earlier in the process. Otherwise it would indeed allow undesired actions to be taken.

When a job fails, it is considered a system failure in our software. For some jobs a retry is allowed. But after that it is generally irrecovable. There is no feedback back to the user as far as I know. When an abuse filter error happens during the Edit or Upload API actions from UploadWizard, or on a regular wiki action page, this feedback is handled by the GUI in question and turned into something useful that the user can understand and act on accordingly. For example, by choosing a different title, or differnent license etc whatever it might be that the filter flagged.

The filtereable criteria for a user upload are hopefully applied during intake. Then the job shouldd run on that same authoritity to save regardless.

Perhaps a next step here could be to understand why these same filters did not trigger earlier in the GUI. Or if they did, why it was still queued as a job? Or perhaps some of the information is missing in the initial run? Or perhaps there is someting generic/faulty in the job context that wrongly causes the job to be caught by a filter?

These jobs are only triggered for async chucked API-based uploads.
The abuse filters can't be executed earlier in the process - running them on individual chunks makes no sense, and this job is where they are recombined into the actual file.
The result status does feed back to the API, though (stored in session - next API poll request will be informed about success/failure status)
So yeah, the more I think about it, the more I'm convinced that the entire existing process is just fine, except that the job shouldn't be considered a (software) failure in this case; it was a rejection, and those details are already being passed along to the API clients - i.e. the job has completed successfully.

Change 608886 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/core@master] concatenateChunks failures are not job failures

https://gerrit.wikimedia.org/r/c/mediawiki/core/ /608886

Change 608886 merged by jenkins-bot:
[mediawiki/core@master] concatenateChunks failures are not job failures

https://gerrit.wikimedia.org/r/608886

Cparle claimed this task.
Cparle added a subscriber: Cparle.

No instances of this error in kibana since the week this went out in the train, so closing