Page MenuHomePhabricator
Create Task
Maniphest T264810

Toolinfo.json crawler
Closed, ResolvedPublic
Actions

Description

  • Crawler process that loads toolinfo.json source URLs, validates their content, and populates the Toolhub database
  • API endpoint(s) to view crawler status information (run date, errors from run)
    • GET /api/v1/crawler/runs - paginated, filterable list of known runs
    • GET /api/v1/crawler/runs/{id} - run details

Details

SubjectRepoBranchLines +/-
crawler: Move create/update logic to custom managerwikimedia/toolhubmain+98 -44
api: Endpoints for crawler run status datawikimedia/toolhubmain+204 -35
Add a basic crawlerwikimedia/toolhubmain+653 -116
Customize query in gerrit

Related Objects
Search...

Event Timeline

bd808 created this task.Oct 6 2020, 10:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 6 2020, 10:06 PM
bd808 triaged this task as Medium priority.Oct 6 2020, 10:33 PM
gerritbot added a comment.Nov 6 2020, 1:29 AM

Change 639666 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] Toolinfo storage model and basic API

https://gerrit.wikimedia.org/r/639666

gerritbot added a project: Patch-For-Review.Nov 6 2020, 1:29 AM
bd808 claimed this task.Nov 6 2020, 5:42 PM
bd808 moved this task from Backlog to In Progress on the Toolhub board.
Restricted Application added a project: User-bd808. · View Herald TranscriptNov 6 2020, 5:42 PM
bd808 removed a project: Patch-For-Review.Nov 6 2020, 6:06 PM
bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.
In T264810#6607947, @gerritbot wrote:

Change 639666 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] Toolinfo storage model and basic API

https://gerrit.wikimedia.org/r/639666

This patch is really T264811: Toolinfo read API, but also necessary for this task.

gerritbot added a comment.Nov 17 2020, 10:50 PM

Change 641559 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] Add a basic crawler

https://gerrit.wikimedia.org/r/641559

gerritbot added a project: Patch-For-Review.Nov 17 2020, 10:50 PM
gerritbot added a comment.Nov 19 2020, 1:30 AM

Change 641559 merged by jenkins-bot:
[wikimedia/toolhub@main] Add a basic crawler

https://gerrit.wikimedia.org/r/641559

gerritbot added a comment.Nov 19 2020, 8:47 PM

Change 642131 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] api: Endpoints for crawler run status data

https://gerrit.wikimedia.org/r/642131

gerritbot added a comment.Nov 20 2020, 5:40 PM

Change 642131 merged by jenkins-bot:
[wikimedia/toolhub@main] api: Endpoints for crawler run status data

https://gerrit.wikimedia.org/r/642131

Aklapper awarded a token.Nov 20 2020, 5:41 PM
srishakatux closed subtask T264969: User interface to view crawler status information as Resolved.Dec 16 2020, 7:39 PM
gerritbot added a comment.Dec 19 2020, 1:20 AM

Change 650639 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[wikimedia/toolhub@main] crawler: Move create/update logic to custom manager

https://gerrit.wikimedia.org/r/650639

gerritbot added a comment.Dec 20 2020, 7:23 PM

Change 650639 merged by jenkins-bot:
[wikimedia/toolhub@main] crawler: Move create/update logic to custom manager

https://gerrit.wikimedia.org/r/650639

bd808 closed this task as Resolved.Dec 21 2020, 8:15 PM

We will need some follow up work in the future to schedule crawler runs, but the core work is done.

bd808 added a project: Developer-Advocacy (Oct-Dec 2020).Jan 14 2021, 5:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits