Epic: Create a centralized logging interface for tracking and reporting dead link fixes
Closed, ResolvedPublic

Description

We should create a centralized logging interface (probably on Tool Labs) that keeps track of which pages have had their dead links fixed, when, and by what agent/bot. This will facilitate 3 things:

  • If a bot dies, it can pick up where it left off
  • It will help prevent bots from doing redundant work
  • It will provide a centralized (and hopefully comprehensive) reporting interface for the Internet Archive and other archive providers

This tool should provide 2 APIs and a web interface:

  • The first API is for recording fix attempts. It should include the following information: wiki, page name, possibly page id, timestamp, possibly revision id, number of links fixed, agent/bot, archive service used
  • The second API (optional) should return the last page (and its wiki) processed by a given agent/bot and the timestamp it was processed at. It's input should be the agent/bot name.
  • The web interface should show a chart of total number of dead link pages fixed or processed per day (on a given wiki). You should also be able to filter by agent.
  • The web interface could optionally provide a paginated log of all dead link pages that have been fixed/processed.

Each of these will need to broken out into separate tasks.

kaldari created this task.Feb 2 2016, 11:49 PM
kaldari updated the task description. (Show Details)
kaldari raised the priority of this task from to Needs Triage.
kaldari added a project: Community-Tech.
kaldari added a subscriber: kaldari.
Restricted Application added a project: Internet-Archive. · View Herald TranscriptFeb 2 2016, 11:49 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
kaldari set Security to None.Feb 2 2016, 11:54 PM
kaldari added subscribers: Cyberpower678, DannyH.
Cyberpower678 triaged this task as Low priority.Feb 2 2016, 11:58 PM
Cyberpower678 claimed this task.

Will bots need to be whitelisted in order to use the interface?

Will bots need to be whitelisted in order to use the interface?

I don't think it would be needed. Do you see any reason it would be?

Will bots need to be whitelisted in order to use the interface?

I don't think it would be needed. Do you see any reason it would be?

Preventing spam?

Cyberpower678 closed this task as Resolved.Mar 25 2016, 11:13 PM
DannyH moved this task from Epic/Tracking to Archive on the Community-Tech board.Mar 29 2016, 5:42 PM