Track errors encountered when processing a crawler URL and show them to users
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Mar 4 2021, 4:55 AM

Description

Currently the crawler history screen will show an "invalid" label for any URL that had one or more errors during the crawl. There is not currently however any way to actually see what the error was (HTTP problem, invalid JSON, missing required fields, etc) which means that as the submitter of a URL I really can't take any actions to fix the problem.

Details

	Subject	Repo	Branch	Lines +/-
	crawler: Capture logs from url processing	wikimedia/toolhub	main	+266 -95

Customize query in gerrit

Related Objects

Mentioned In: rWTHUade10bf086e5: crawler: Capture logs from url processing

Event Timeline

bd808 created this task.Mar 4 2021, 4:55 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2021, 4:55 AM

bd808 claimed this task.Mar 29 2021, 10:38 PM

bd808 moved this task from Backlog to Review on the Toolhub board.

bd808 added a project: Developer-Advocacy (Jan-Mar 2021).

Restricted Application added a project: User-bd808. · View Herald TranscriptMar 29 2021, 10:38 PM

Change 675596 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[wikimedia/toolhub@main] crawler: Capture logs from url processing

https://gerrit.wikimedia.org/r/675596

gerritbot added a project: Patch-For-Review.Mar 29 2021, 10:45 PM

Change 675596 merged by jenkins-bot:

[wikimedia/toolhub@main] crawler: Capture logs from url processing

https://gerrit.wikimedia.org/r/675596

jenkins-bot mentioned this in rWTHUade10bf086e5: crawler: Capture logs from url processing.Apr 2 2021, 4:39 PM

bd808 closed this task as Resolved.Apr 2 2021, 4:45 PM

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2021, 5:10 PM